From steve at pearwood.info Fri Feb 1 00:23:50 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 01 Feb 2013 10:23:50 +1100 Subject: [Python-ideas] Definition Symmetry In-Reply-To: References: Message-ID: <510AFD06.1090305@pearwood.info> On 01/02/13 03:35, Jason Keene wrote: > Why do function definitions require parens? > >>>> class MyClass: > ... pass > ... >>>> def my_func: > File "", line 1 > def my_func: > ^ > SyntaxError: invalid syntax > > This seems to me to break a symmetry with class definitions. Why do you think that class definitions and function definitions should be symmetrical? -- Steven From steve at pearwood.info Fri Feb 1 00:26:33 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 01 Feb 2013 10:26:33 +1100 Subject: [Python-ideas] Extend module objects to support properties In-Reply-To: References: <51087225.3040801@hastings.org> <51094D8D.606@hastings.org> <5109BEC4.4050604@pearwood.info> <5109CE95.7060104@hastings.org> Message-ID: <510AFDA9.9080507@pearwood.info> On 01/02/13 09:00, Chris Angelico wrote: > On Thu, Jan 31, 2013 at 12:53 PM, Larry Hastings wrote: >> But if the user assigns a different value to math.__dict__['pi'], math.pi >> will diverge, which again could break code. (Who might try to assign a >> different value to pi? The 1897 House Of Representatives of Indiana for >> one!) >> >> >> More generally, it's often useful to monkeypatch "constants" at runtime, for >> testing purposes (and for less justifiable purposes). Why prevent that? I >> cite the Consenting Adults rule. > > I've never actually been in the situation of doing it, but wouldn't it > be reasonable to switch out math.pi to be (say) a decimal.Decimal > rather than a float? To what purpose? Monkey-patching math.pi doesn't affect any of the other math functions. All it will do is break Python code that uses math.pi and expects it to be a float. -- Steven From zuo at chopin.edu.pl Fri Feb 1 00:27:07 2013 From: zuo at chopin.edu.pl (=?utf-8?B?SmFuIEthbGlzemV3c2tp?=) Date: Fri, 01 Feb 2013 00:27:07 +0100 Subject: [Python-ideas] =?utf-8?q?Definition_Symmetry?= Message-ID: <20130131232712.70B9BD60F5@filifionka.chopin.edu.pl> First of all, the class definition concept and the function definition concept should *not* be considered 'symmetric'. They are fundamentally different in several aspects - especially the body of a class definition is executed once, at the *definition* time (and *before* the actual class is made) and the body of a function definition can be executed multiple times, at the *call* time (and obviously *after* the function is made). Regards. *j -- [sent from phone] -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri Feb 1 00:36:51 2013 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 1 Feb 2013 10:36:51 +1100 Subject: [Python-ideas] Extend module objects to support properties In-Reply-To: <510AFDA9.9080507@pearwood.info> References: <51087225.3040801@hastings.org> <51094D8D.606@hastings.org> <5109BEC4.4050604@pearwood.info> <5109CE95.7060104@hastings.org> <510AFDA9.9080507@pearwood.info> Message-ID: On Fri, Feb 1, 2013 at 10:26 AM, Steven D'Aprano wrote: > On 01/02/13 09:00, Chris Angelico wrote: >> I've never actually been in the situation of doing it, but wouldn't it >> be reasonable to switch out math.pi to be (say) a decimal.Decimal >> rather than a float? > > To what purpose? Monkey-patching math.pi doesn't affect any of the other > math > functions. All it will do is break Python code that uses math.pi and expects > it to be a float. True. I guess you'd just have your own local pi for that. (Your own local bakery maybe?) Consider the notion withdrawn. ChrisA From ncoghlan at gmail.com Fri Feb 1 01:50:03 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 1 Feb 2013 10:50:03 +1000 Subject: [Python-ideas] Extend module objects to support properties In-Reply-To: References: <51087225.3040801@hastings.org> <51094D8D.606@hastings.org> Message-ID: On Fri, Feb 1, 2013 at 5:56 AM, Eric Snow wrote: > That said, Larry's original proposal relates to sys, a built-in module > written in C (in CPython of course). In that case the solution is not > quite the same, since module initialization interacts with sys.modules > differently. [1][2] Accommodating the original request would require > more work, whether to muck with the import C-API or making sys an > instance of another type, as someone suggested. sys is already special cased so heavily in the interpreter initialization, making it a little more special really wouldn't bother me much :) As far as customising other extension modules goes, the same "replace yourself in sys.modules" trick should still work, you'd just do it via the C API from your module init function. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From tjreedy at udel.edu Fri Feb 1 02:07:27 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 31 Jan 2013 20:07:27 -0500 Subject: [Python-ideas] Definition Symmetry In-Reply-To: <20130131160657.4620f918@anarchist.wooz.org> References: <510AC6F1.1060503@ei-grad.ru> <20130131160657.4620f918@anarchist.wooz.org> Message-ID: On 1/31/2013 4:06 PM, Barry Warsaw wrote: > On Jan 31, 2013, at 11:33 PM, Andrew Grigorev wrote: > >> Other strange thing is that the `raise` statement doesn't require to >> instantiate an Exception object, allowing to pass an Exception class to it. >> >> raise NotImplementedError >> raise NotImplementedError() >> >> Is there any difference between this two lines of code? Questions like this below on python-list, not here. > The main difference (I *think* this is still true) is that in the first > example, if the exception is caught in C it can avoid instantiation. The the second form allows addition of a message, which is usually a good idea. -- Terry Jan Reedy From tjreedy at udel.edu Fri Feb 1 02:34:20 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 31 Jan 2013 20:34:20 -0500 Subject: [Python-ideas] Extend module objects to support properties In-Reply-To: References: <51087225.3040801@hastings.org> <51094D8D.606@hastings.org> <5109BEC4.4050604@pearwood.info> <5109CE95.7060104@hastings.org> Message-ID: On 1/31/2013 5:00 PM, Chris Angelico wrote: > I've never actually been in the situation of doing it, but wouldn't it > be reasonable to switch out math.pi to be (say) a decimal.Decimal > rather than a float? Mixed float-decimal arithmetic does not work. If the float functions in math used 'math.pi', that would not work, but they do not. On the other hand, the stdlib and test suite do have a few expressions like 2.0 * math.pi and math.pi / 180.0 that would have to be changed to 2 * math.pi and math.pi / 180. (This might be a reasonable change anyway.) Non-float inputs to math functions like sin are apparently converted to float when possible (I tested int, Decimal, and Fraction with sin). So if you can otherwise avoid float +-*/ decimal, the answer might be yes. -- Terry Jan Reedy From guido at python.org Fri Feb 1 02:58:32 2013 From: guido at python.org (Guido van Rossum) Date: Thu, 31 Jan 2013 17:58:32 -0800 Subject: [Python-ideas] PEP 3156: getting the socket or peer name from the transport In-Reply-To: References: <20130127122121.6b779ada@pitrou.net> <1359288997.3488.2.camel@localhost.localdomain> Message-ID: On Thu, Jan 31, 2013 at 12:51 AM, Yuval Greenfield wrote: > On Mon, Jan 28, 2013 at 5:45 PM, Guido van Rossum wrote: >> >> Hm. I'm not keen on precomputing all of that, since most protocols >> won't need it, and the cost add up. This is not WSGI. The protocol has >> the transport object and can ask it specific questions -- if through a >> general API, like get_extra_info(key, [default]). >> > > I forgot to ask before, but why is get_extra_info better than normal > attributes and methods? > > val = transport.get_extra_info(key, None) > if val is None: > pass > > # vs > > if hasattr(transport, key): > val = transport.key > else: > pass hasattr() smells bad. It also has namespace issues (hasattr(transport, "write") returns true) and if people forget to use it (perhaps because the transport they normally use always has a certain attribute) their code is brittle. Defining a new API with a string key signals clearly that the value may or may not exist, and reminds people to test the result for None. (Of course they can forget that too. But it still feels different to me.) -- --Guido van Rossum (python.org/~guido) From guido at python.org Fri Feb 1 03:00:13 2013 From: guido at python.org (Guido van Rossum) Date: Thu, 31 Jan 2013 18:00:13 -0800 Subject: [Python-ideas] PEP 3156: getting the socket or peer name from the transport In-Reply-To: References: <20130127122121.6b779ada@pitrou.net> <1359288997.3488.2.camel@localhost.localdomain> Message-ID: On Thu, Jan 31, 2013 at 12:52 AM, Yuval Greenfield wrote: >> val = getattr(transport, key) You probably meant val = getattr(transport, "key", None) Your original example also forgot the quotes around "key" in the hasattr() call.) The number of attempts needed to get your example right tells me this is not a good proposal. :-) -- --Guido van Rossum (python.org/~guido) From jasonkeene at gmail.com Fri Feb 1 03:13:32 2013 From: jasonkeene at gmail.com (Jason Keene) Date: Thu, 31 Jan 2013 21:13:32 -0500 Subject: [Python-ideas] Definition Symmetry In-Reply-To: <510AFD06.1090305@pearwood.info> References: <510AFD06.1090305@pearwood.info> Message-ID: In a way they do the same thing, they both create an object (function/class) from a suite and assign it to the name given after the keyword (def/class). Sure they do totally different things with the suite in creating the object, but in essence it's a name assignment. On Thursday, January 31, 2013, Steven D'Aprano wrote: > On 01/02/13 03:35, Jason Keene wrote: > >> Why do function definitions require parens? >> >> class MyClass: >>>>> >>>> ... pass >> ... >> >>> def my_func: >>>>> >>>> File "", line 1 >> def my_func: >> ^ >> SyntaxError: invalid syntax >> >> This seems to me to break a symmetry with class definitions. >> > > Why do you think that class definitions and function definitions > should be symmetrical? > > > > -- > Steven > ______________________________**_________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/**mailman/listinfo/python-ideas > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericsnowcurrently at gmail.com Fri Feb 1 03:58:42 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Thu, 31 Jan 2013 19:58:42 -0700 Subject: [Python-ideas] Extend module objects to support properties In-Reply-To: References: <51087225.3040801@hastings.org> <51094D8D.606@hastings.org> Message-ID: On Jan 31, 2013 5:50 PM, "Nick Coghlan" wrote: > > On Fri, Feb 1, 2013 at 5:56 AM, Eric Snow wrote: > > That said, Larry's original proposal relates to sys, a built-in module > > written in C (in CPython of course). In that case the solution is not > > quite the same, since module initialization interacts with sys.modules > > differently. [1][2] Accommodating the original request would require > > more work, whether to muck with the import C-API or making sys an > > instance of another type, as someone suggested. > > sys is already special cased so heavily in the interpreter > initialization, making it a little more special really wouldn't bother > me much :) > > As far as customising other extension modules goes, the same "replace > yourself in sys.modules" trick should still work, you'd just do it via > the C API from your module init function. Don't extension modules get added to sys.modules only after their init function returns? -eric -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Feb 1 04:07:37 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 1 Feb 2013 13:07:37 +1000 Subject: [Python-ideas] Extend module objects to support properties In-Reply-To: References: <51087225.3040801@hastings.org> <51094D8D.606@hastings.org> Message-ID: On Fri, Feb 1, 2013 at 12:58 PM, Eric Snow wrote: > > On Jan 31, 2013 5:50 PM, "Nick Coghlan" wrote: >> >> On Fri, Feb 1, 2013 at 5:56 AM, Eric Snow >> wrote: >> > That said, Larry's original proposal relates to sys, a built-in module >> > written in C (in CPython of course). In that case the solution is not >> > quite the same, since module initialization interacts with sys.modules >> > differently. [1][2] Accommodating the original request would require >> > more work, whether to muck with the import C-API or making sys an >> > instance of another type, as someone suggested. >> >> sys is already special cased so heavily in the interpreter >> initialization, making it a little more special really wouldn't bother >> me much :) >> >> As far as customising other extension modules goes, the same "replace >> yourself in sys.modules" trick should still work, you'd just do it via >> the C API from your module init function. > > Don't extension modules get added to sys.modules only after their init > function returns? Oh, you're right, they're currently responsible for creating their own module object (I was thinking of the way I *wished* extension module imports worked). Returning the module object from the init function still works though - we just need to make sure the extension loader can cope with the init function returning a non-module object :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From timothy.c.delaney at gmail.com Fri Feb 1 01:19:14 2013 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Fri, 1 Feb 2013 11:19:14 +1100 Subject: [Python-ideas] constant/enum type in stdlib In-Reply-To: References: <51085AAB.6090303@canterbury.ac.nz> <5108A87D.9000207@canterbury.ac.nz> <20130130082639.0b28d7eb@pitrou.net> Message-ID: Wow - this ended up being more difficult that I'd anticipated. Ensuring that decorators work in Michael Foord-inspired enums, etc mandated a fairly big redesign. As it was you couldn't do something like: class A(): a = 1 class B(Enum): B = A.a because when the name 'A' was resolved in the definition of class B, it was returning an int (the next enum value). The end result may well still have some holes, but it's looking pretty good to me. Enums are constructed like: class Color(Enum): RED, GREEN, BLUE CYAN = 10 MAGENTA YELLOW BLACK where the assigned enum starts a new count i.e. the above is 0, 1, 2, 10, 11, 12, 13. Arbitrary attributes may be assigned and not contribute to the enumeration so long as the object being assigned does not implement __index__ (if it does, it creates a discontiguous enumeration). #!/usr/bin/env python3 import builtins import collections import operator class EnumValue(int): def __new__(cls, key, value): e = super().__new__(cls, value) super().__setattr__(e, 'key', key) super().__setattr__(e, 'owner', None) return e def __setattr__(self, key, value): raise TypeError("Cannot set attribute of type %r" % (type(self),)) def __str__(self): return "%s.%s" % (self.owner.__name__, self.key) def __repr__(self): if self.owner is not None: return "<%s '%s.%s': %d>" % (self.__qualname__, self.owner.__qualname__, self.key, int(self)) return "<%s '%s': %d>" % (self.__qualname__, self.key, int(self)) class EnumProxy(object): def __init__(self, key, value=None): self.key = key self.values = [value] self.used = False def __repr__(self): return "<%s '%s': %s>" % (self.__qualname__, self.key, self.values) def _get(self, used=None): if used: self.used = True try: return locals()[self.key] except KeyError: try: return globals()[self.key] except KeyError: try: return getattr(builtins, self.key) except KeyError: raise NameError(self.key, self.values) def __call__(self, *p, **kw): return self._get(True)(*p, **kw) def __getattr__(self, name): return getattr(self._get(True), name) class EnumValues(collections.OrderedDict): def __init__(self): super().__init__() self.sealed = False def __getitem__(self, key): try: obj = super().__getitem__(key) if not self.sealed and isinstance(obj, EnumProxy): obj.values.append(None) return obj except KeyError: # Don't do anything with __dunder__ attributes if key[:2] == '__' and key[-2:] == '__': raise proxy = EnumProxy(key, None) super().__setitem__(key, proxy) return proxy def __setitem__(self, key, value): if key[:2] == '__' and key[-2:] == '__': return super().__setitem__(key, value) try: if isinstance(value, EnumProxy): value = value._get(True) elif not isinstance(value, EnumValue): value = operator.index(value) except TypeError: return super().__setitem__(key, value) try: o = super().__getitem__(key) if isinstance(o, EnumProxy): o.values.append(value) except KeyError: if isinstance(value, EnumProxy): int.__setattr__(value, 'key', key) else: value = EnumProxy(key, value) super().__setitem__(value.key, value) class EnumMeta(type): @classmethod def __prepare__(metacls, name, bases): return EnumValues() def __new__(cls, name, bases, classdict): classdict.sealed = True del_list = [] for v in classdict.values(): if isinstance(v, EnumProxy) and v.used: del_list.append(v) for v in del_list: del classdict[v.key] result = type.__new__(cls, name, bases, dict(classdict)) value = 0 keys = {} values = {} for v in classdict.values(): if isinstance(v, EnumProxy) and not v.used: if len(v.values) > 1: raise AttributeError("Duplicate enum key '%s.%s'" % (result.__qualname__, v.key,)) if v.values[0] is not None: value = v.values[0] if isinstance(value, EnumValue): if (value.key is not None) and (value.key != v.key): raise AttributeError("Assigned enum value to non-matching key '%s': %r" % (v.key, value)) if value.owner is not None: raise AttributeError("Assigned owned enum value to key '%s': %r" % (v.key, value)) int.__setattr__(value, 'key', v.key) v = value else: v = EnumValue(v.key, value) setattr(result, v.key, v) value += 1 if isinstance(v, EnumValue): int.__setattr__(v, 'owner', result) if v in values: raise AttributeError("Duplicate enum value %d for keys: '%s' and '%s'" % (int(v), values[v].key, v.key)) keys[v.key] = v values[v] = v enum = sorted(values) result._key_to_enum = collections.OrderedDict() result._value_to_enum = values for e in enum: result._key_to_enum[e.key] = e return result def __getitem__(self, key): try: key = operator.index(key) except TypeError: return self._key_to_enum[key] else: return self._value_to_enum[key] def _items(self): return self._key_to_enum.items() def _keys(self): return self._key_to_enum.keys() def _values(self): return self._key_to_enum.values() def items(self): return self._items() def keys(self): return self._keys() def values(self): return self._values() def __iter__(self): return iter(self.values()) def __repr__(self): r = super().__repr__() r = ['') return ''.join(r) def __str__(self): s = ['{'] for k, v in self.items(): if s[-1][-1:] != '{': s.append(', ') s.extend([k, ':', str(int(v))]) s.append('}') return ''.join(s) class Enum(metaclass=EnumMeta): def __getitem__(self, key): cls = type(self) return type(cls).__getitem__(cls, key) @classmethod def items(cls): return cls._items() @classmethod def keys(cls): return cls._keys() @classmethod def values(cls): return cls._values() def __iter__(self): return iter(self.values()) def __repr__(self): r = super().__repr__() r = r.replace('object at 0x', 'enum at 0x') r = [r[:-1], ' '] r.append(str(self)) r.append('>') return ''.join(r) def __str__(self): return str(type(self)) if __name__ == '__main__': class Color(Enum): RED, GREEN, BLUE ORANGE = "orange" CYAN = 10 MAGENTA YELLOW BLACK import unittest class TestEnum(unittest.TestCase): EXPECTED_KEY_ORDER = ('RED', 'GREEN', 'BLUE', 'CYAN', 'MAGENTA', 'YELLOW', 'BLACK') EXPECTED_INT_VALUE_ORDER = (0, 1, 2, 10, 11, 12, 13) EXPECTED_ENUM_VALUE_ORDER = (Color.RED, Color.GREEN, Color.BLUE, Color.CYAN, Color.MAGENTA, Color.YELLOW, Color.BLACK) EXPECTED_ITEMS_ORDER = tuple(zip(EXPECTED_KEY_ORDER, EXPECTED_ENUM_VALUE_ORDER)) def test_type(self): self.assertIsInstance(Color.RED, int) def test_class_enum_values(self): self.assertEqual(0, Color.RED) self.assertEqual(1, Color.GREEN) self.assertEqual(2, Color.BLUE) self.assertEqual(10, Color.CYAN) self.assertEqual(11, Color.MAGENTA) self.assertEqual(12, Color.YELLOW) self.assertEqual(13, Color.BLACK) self.assertEqual("orange", Color.ORANGE) def test_instance_enum_values(self): e = Color() self.assertIs(Color.RED, e.RED) self.assertIs(Color.GREEN, e.GREEN) self.assertIs(Color.BLUE, e.BLUE) self.assertIs(Color.CYAN, e.CYAN) self.assertIs(Color.MAGENTA, e.MAGENTA) self.assertIs(Color.YELLOW, e.YELLOW) self.assertIs(Color.BLACK, e.BLACK) self.assertIs(Color.ORANGE, e.ORANGE) def test_class_indexing(self): self.assertIs(Color.CYAN, Color['CYAN']) self.assertIs(Color.CYAN, Color[10]) def test_instance_indexing(self): e = Color() self.assertIs(Color.CYAN, e['CYAN']) self.assertIs(Color.CYAN, e[10]) def test_class_keys(self): self.assertEqual(self.EXPECTED_KEY_ORDER, tuple(Color.keys())) def test_instance_keys(self): self.assertEqual(tuple(Color.keys()), tuple(Color().keys())) def test_class_values(self): self.assertEqual(self.EXPECTED_INT_VALUE_ORDER, tuple(Color.values())) self.assertEqual(self.EXPECTED_ENUM_VALUE_ORDER, tuple(Color.values())) def test_instance_values(self): self.assertEqual(tuple(Color.values()), tuple(Color().values())) def test_class_items(self): self.assertEqual(self.EXPECTED_ITEMS_ORDER, tuple(Color.items())) def test_instance_items(self): self.assertEqual(tuple(Color.items()), tuple(Color().items())) def test_owner(self): for e in Color: self.assertIs(e.owner, Color) def test_class_str(self): s = str(Color) for e in Color: self.assertIn('%s:%d' % (e.key, int(e)), s) def test_instance_str(self): self.assertEqual(str(Color), str(Color())) def test_class_repr(self): r = repr(Color) self.assertIn(Color.__qualname__, r) self.assertIn(str(Color), r) def test_instance_repr(self): e = Color() r = repr(e) self.assertIn(Color.__qualname__, r) self.assertIn('at 0x', r) self.assertIn(str(Color()), r) def _create_duplicate_key(self): class DuplicateKey(Enum): KEY, KEY def test_duplicate_key(self): self.assertRaises(AttributeError, self._create_duplicate_key) def _create_duplicate_value(self): class DuplicateValue(Enum): KEY1, KEY2 = 0 def test_duplicate_value(self): self.assertRaises(AttributeError, self._create_duplicate_value) def _assign_wrong_key(self): class WrongKey(Enum): KEY1 = EnumValue('KEY2', 0) def test_wrong_key(self): self.assertRaises(AttributeError, self._assign_wrong_key) def test_unnamed_key1(self): class UnnamedKey(Enum): KEY1 = EnumValue(None, 5) self.assertEqual(UnnamedKey.KEY1, 5) self.assertIs(UnnamedKey, UnnamedKey.KEY1.owner) def test_unnamed_key1(self): unnamed = EnumValue(None, 5) class UnnamedKey(Enum): KEY1 = unnamed self.assertEqual(UnnamedKey.KEY1, 5) self.assertIs(UnnamedKey.KEY1, unnamed) self.assertIs(UnnamedKey, UnnamedKey.KEY1.owner) def _assign_wrong_owner(self): class WrongOwner(Enum): KEY1 = Color.RED def test_wrong_owner(self): self.assertRaises(AttributeError, self._assign_wrong_owner) unittest.main() Tim Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: From timothy.c.delaney at gmail.com Fri Feb 1 05:18:57 2013 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Fri, 1 Feb 2013 15:18:57 +1100 Subject: [Python-ideas] constant/enum type in stdlib In-Reply-To: References: <51085AAB.6090303@canterbury.ac.nz> <5108A87D.9000207@canterbury.ac.nz> <20130130082639.0b28d7eb@pitrou.net> Message-ID: Last version (for now). I'm really interested in people's opinions on this. For this version I've taken some inspiration from flufl.enum (but there remains the major difference that these enums subclass int). - Enums are now subclassable; - Added an Enum.make() method - a bit different to flufl.enum.make since my enums have different semantics - each element must either be a name or a (name, value) pair, and you can have a mix; - Instantiating an enum now returns the appropriate EnumValue - Enums now compare not equal with any enum that is not the same object (but continue to compare equal with ints); - Changed EnumValue.key -> EnumValue.name and EnumValue.owner -> EnumValue.enum. I didn't add __members__ as that use case is covered by having the Enum be iterable + the immutable mapping interface. #!/usr/bin/env python3 import builtins import collections import operator class EnumValue(int): def __new__(cls, key, value): e = super().__new__(cls, value) super().__setattr__(e, 'name', key) super().__setattr__(e, 'enum', None) return e def __setattr__(self, key, value): raise TypeError("can't set attribute") def __eq__(self, other): if isinstance(other, EnumValue): return self is other return int(self) == other def __ne__(self, other): return not (self == other) def __hash__(self): return super().__hash__() def __str__(self): if self.enum is not None: return "%s.%s" % (self.enum.__name__, self.name) return self.name def __repr__(self): if self.enum is not None: return "<%s '%s.%s': %d>" % (self.__qualname__, self.enum.__qualname__, self.name, int(self)) return "<%s '%s': %d>" % (self.__qualname__, self.name, int(self)) class _EnumProxy(object): def __init__(self, key, value=None): self.key = key self.values = [value] self.used = False def __repr__(self): return "<%s '%s': %s>" % (self.__qualname__, self.key, self.values) def _get(self, used=None): if used: self.used = True try: return locals()[self.key] except KeyError: try: return globals()[self.key] except KeyError: try: return getattr(builtins, self.key) except KeyError: raise NameError(self.key, self.values) def __call__(self, *p, **kw): return self._get(True)(*p, **kw) def __getattr__(self, name): return getattr(self._get(True), name) class EnumValues(collections.OrderedDict): def __init__(self): super().__init__() self.sealed = False def __getitem__(self, key): try: obj = super().__getitem__(key) if not self.sealed and isinstance(obj, _EnumProxy): obj.values.append(None) return obj except KeyError: # Don't do anything with __dunder__ attributes if key[:2] == '__' and key[-2:] == '__': raise proxy = _EnumProxy(key, None) super().__setitem__(key, proxy) return proxy def __setitem__(self, key, value): if key[:2] == '__' and key[-2:] == '__': return super().__setitem__(key, value) try: if isinstance(value, _EnumProxy): value = value._get(True) elif not isinstance(value, EnumValue): value = operator.index(value) except TypeError: return super().__setitem__(key, value) try: o = super().__getitem__(key) if isinstance(o, _EnumProxy): o.values.append(value) except KeyError: if not isinstance(value, _EnumProxy): value = _EnumProxy(key, value) super().__setitem__(value.key, value) class EnumMeta(type): @classmethod def __prepare__(metacls, name, bases): return EnumValues() def __new__(cls, name, bases, classdict): classdict.sealed = True del_list = [] for v in classdict.values(): if isinstance(v, _EnumProxy) and v.used: del_list.append(v) for v in del_list: del classdict[v.key] result = type.__new__(cls, name, bases, dict(classdict)) keys = {} values = {} result._key_to_enum = collections.OrderedDict() result._value_to_enum = values value = 0 for b in result.__bases__: if isinstance(b, EnumMeta): keys.update(b._key_to_enum) values.update(b._value_to_enum) if values: value = max(values) + 1 for v in classdict.values(): if isinstance(v, _EnumProxy) and not v.used: if len(v.values) > 1: raise AttributeError("Duplicate enum key '%s.%s'" % (name, v.key,)) elif v.key in keys: raise AttributeError("Duplicate enum key '%s.%s' (overriding '%s')" % (result.__name__, v.key, keys[v.key])) if v.values[0] is not None: value = v.values[0] if isinstance(value, EnumValue): if (value.name is not None) and (value.name != v.key): raise AttributeError("Assigned enum value to non-matching key '%s': %r" % (v.key, value)) if value.enum is not None: raise AttributeError("Assigned owned enum value to key '%s': %r" % (v.key, value)) int.__setattr__(value, 'name', v.key) v = value else: v = EnumValue(v.key, value) setattr(result, v.name, v) value += 1 if isinstance(v, EnumValue): int.__setattr__(v, 'enum', result) int_v = int(v) if int_v in values: raise AttributeError("Duplicate enum value %d for keys: '%s.%s' and '%s.%s'" % ( int_v, values[int_v].enum.__name__, values[int_v].name, result.__name__, v.name)) keys[v.name] = v values[v] = v enum = sorted(values) for e in enum: result._key_to_enum[e.name] = e return result def __getitem__(self, key): try: key = operator.index(key) except TypeError: return self._key_to_enum[key] else: return self._value_to_enum[key] def items(self): return self._key_to_enum.items() def keys(self): return self._key_to_enum.keys() def values(self): return self._key_to_enum.values() def __iter__(self): return iter(self.values()) def __repr__(self): r = super().__repr__() r = ['') return ''.join(r) def __str__(self): s = ['{'] for k, v in self.items(): if s[-1][-1:] != '{': s.append(', ') s.extend([k, ':', str(int(v))]) s.append('}') return ''.join(s) class Enum(metaclass=EnumMeta): def __new__(cls, value): return type(cls).__getitem__(cls, value) @staticmethod def make(clsname, elements): classdict = collections.OrderedDict() for e in elements: if isinstance(e, tuple): k, v = e else: k, v = e, None try: e = classdict[k] e.values.append(v) except KeyError: classdict[k] = _EnumProxy(k, v) result = EnumMeta(clsname, (Enum,), classdict) # For some reason, this is set to 'Enum' ... result.__qualname__ = clsname return result if __name__ == '__main__': import unittest class Color(Enum): RED, GREEN, BLUE ORANGE = "orange" CYAN = 10 MAGENTA YELLOW BLACK class TestEnum(unittest.TestCase): EXPECTED_KEY_ORDER = ('RED', 'GREEN', 'BLUE', 'CYAN', 'MAGENTA', 'YELLOW', 'BLACK') EXPECTED_INT_VALUE_ORDER = (0, 1, 2, 10, 11, 12, 13) EXPECTED_ENUM_VALUE_ORDER = (Color.RED, Color.GREEN, Color.BLUE, Color.CYAN, Color.MAGENTA, Color.YELLOW, Color.BLACK) EXPECTED_ITEMS_ORDER = tuple(zip(EXPECTED_KEY_ORDER, EXPECTED_ENUM_VALUE_ORDER)) def test_type(self): self.assertIsInstance(Color.RED, int) def test_enum_values(self): self.assertEqual(0, Color.RED) self.assertEqual(1, Color.GREEN) self.assertEqual(2, Color.BLUE) self.assertEqual(10, Color.CYAN) self.assertEqual(11, Color.MAGENTA) self.assertEqual(12, Color.YELLOW) self.assertEqual(13, Color.BLACK) self.assertEqual("orange", Color.ORANGE) def test_indexing(self): self.assertIs(Color.CYAN, Color['CYAN']) self.assertIs(Color.CYAN, Color[10]) def test_keys(self): self.assertEqual(self.EXPECTED_KEY_ORDER, tuple(Color.keys())) def test_values(self): self.assertEqual(self.EXPECTED_INT_VALUE_ORDER, tuple(Color.values())) self.assertEqual(self.EXPECTED_ENUM_VALUE_ORDER, tuple(Color.values())) def test_items(self): self.assertEqual(self.EXPECTED_ITEMS_ORDER, tuple(Color.items())) def test_owner(self): for e in Color: self.assertIs(e.enum, Color) def test_equality(self): self.assertEqual(0, Color.RED) self.assertEqual(Color.RED, 0) self.assertEqual(Color.RED, Color.RED) self.assertNotEqual(Color.RED, Color.GREEN) class Color2(Enum): RED self.assertEqual(0, Color2.RED) self.assertEqual(Color2.RED, 0) self.assertNotEqual(Color.RED, Color2.RED) def test_str(self): s = str(Color) for e in Color: self.assertIn('%s:%d' % (e.name, int(e)), s) def test_repr(self): r = repr(Color) self.assertIn(Color.__qualname__, r) self.assertIn(str(Color), r) def test_instances(self): for e in Color: self.assertIs(e, Color(e)) self.assertIs(e, Color(int(e))) self.assertIs(e, Color(e.name)) def _create_duplicate_key(self): class DuplicateKey(Enum): KEY, KEY def test_duplicate_key(self): self.assertRaises(AttributeError, self._create_duplicate_key) def _create_duplicate_value(self): class DuplicateValue(Enum): KEY1, KEY2 = 0 def test_duplicate_value(self): self.assertRaises(AttributeError, self._create_duplicate_value) def _assign_wrong_key(self): class WrongKey(Enum): KEY1 = EnumValue('KEY2', 0) def test_wrong_key(self): self.assertRaises(AttributeError, self._assign_wrong_key) def test_unnamed_key1(self): class UnnamedKey(Enum): KEY1 = EnumValue(None, 5) self.assertEqual(UnnamedKey.KEY1, 5) self.assertIs(UnnamedKey, UnnamedKey.KEY1.enum) def test_unnamed_key1(self): unnamed = EnumValue(None, 5) class UnnamedKey(Enum): KEY1 = unnamed self.assertEqual(UnnamedKey.KEY1, 5) self.assertIs(UnnamedKey.KEY1, unnamed) self.assertIs(UnnamedKey, UnnamedKey.KEY1.enum) def _assign_wrong_owner(self): class WrongOwner(Enum): KEY1 = Color.RED def test_wrong_owner(self): self.assertRaises(AttributeError, self._assign_wrong_owner) def test_subclassing(self): class ExtendedColor1(Color): PINK, GREY, WHITE = 5 self.assertIs(Color.RED, ExtendedColor1.RED) for e in Color: self.assertIs(e, ExtendedColor1[e]) self.assertEqual(14, ExtendedColor1.PINK) self.assertEqual(15, ExtendedColor1.GREY) self.assertEqual(5, ExtendedColor1.WHITE) class ExtendedColor2(Color): PINK, GREY, WHITE = 5 self.assertIsNot(ExtendedColor1.PINK, ExtendedColor2.PINK) self.assertIs(ExtendedColor1.RED, ExtendedColor2.RED) def _create_duplicate_key_subclass(self): class ExtendedColor1(Color): RED def test_duplicate_key_subclass(self): self.assertRaises(AttributeError, self._create_duplicate_key_subclass) def _create_duplicate_value_subclass(self): class ExtendedColor1(Color): PINK = 0 def test_duplicate_key_subclass(self): self.assertRaises(AttributeError, self._create_duplicate_value_subclass) def test_make(self): e = Enum.make('e', ('a', 'b', ('c', 3), 'd')) self.assertEqual(0, e.a) self.assertEqual(1, e.b) self.assertEqual(3, e.c) self.assertEqual(4, e.d) def test_duplicate_key_make(self): self.assertRaises(AttributeError, Enum.make, 'e', ('a', 'b', ('b', 3), 'd')) self.assertRaises(AttributeError, Enum.make, 'e', ('a', 'b', ('c', 3), 'b')) def test_duplicate_value_make(self): self.assertRaises(AttributeError, Enum.make, 'e', ('a', 'b', ('c', 1), 'd')) unittest.main() Tim Delaney On 1 February 2013 11:19, Tim Delaney wrote: > Wow - this ended up being more difficult that I'd anticipated. Ensuring > that decorators work in Michael Foord-inspired enums, etc mandated a fairly > big redesign. As it was you couldn't do something like: > > class A(): > a = 1 > > class B(Enum): > B = A.a > > because when the name 'A' was resolved in the definition of class B, it > was returning an int (the next enum value). > > The end result may well still have some holes, but it's looking pretty > good to me. > > Enums are constructed like: > > class Color(Enum): > RED, GREEN, BLUE > CYAN = 10 > MAGENTA > YELLOW > BLACK > > where the assigned enum starts a new count i.e. the above is 0, 1, 2, 10, > 11, 12, 13. Arbitrary attributes may be assigned and not contribute to the > enumeration so long as the object being assigned does not implement > __index__ (if it does, it creates a discontiguous enumeration). > > #!/usr/bin/env python3 > > import builtins > import collections > import operator > > class EnumValue(int): > def __new__(cls, key, value): > e = super().__new__(cls, value) > super().__setattr__(e, 'key', key) > super().__setattr__(e, 'owner', None) > return e > > def __setattr__(self, key, value): > raise TypeError("Cannot set attribute of type %r" % (type(self),)) > > def __str__(self): > return "%s.%s" % (self.owner.__name__, self.key) > > def __repr__(self): > if self.owner is not None: > return "<%s '%s.%s': %d>" % (self.__qualname__, > self.owner.__qualname__, self.key, int(self)) > > return "<%s '%s': %d>" % (self.__qualname__, self.key, int(self)) > > class EnumProxy(object): > def __init__(self, key, value=None): > self.key = key > self.values = [value] > self.used = False > > def __repr__(self): > return "<%s '%s': %s>" % (self.__qualname__, self.key, self.values) > > def _get(self, used=None): > if used: > self.used = True > > try: > return locals()[self.key] > except KeyError: > try: > return globals()[self.key] > except KeyError: > try: > return getattr(builtins, self.key) > except KeyError: > raise NameError(self.key, self.values) > > def __call__(self, *p, **kw): > return self._get(True)(*p, **kw) > > def __getattr__(self, name): > return getattr(self._get(True), name) > > class EnumValues(collections.OrderedDict): > def __init__(self): > super().__init__() > self.sealed = False > > def __getitem__(self, key): > try: > obj = super().__getitem__(key) > > if not self.sealed and isinstance(obj, EnumProxy): > obj.values.append(None) > > return obj > > except KeyError: > # Don't do anything with __dunder__ attributes > if key[:2] == '__' and key[-2:] == '__': > raise > > proxy = EnumProxy(key, None) > super().__setitem__(key, proxy) > return proxy > > def __setitem__(self, key, value): > if key[:2] == '__' and key[-2:] == '__': > return super().__setitem__(key, value) > > try: > if isinstance(value, EnumProxy): > value = value._get(True) > > elif not isinstance(value, EnumValue): > value = operator.index(value) > > except TypeError: > return super().__setitem__(key, value) > > try: > o = super().__getitem__(key) > > if isinstance(o, EnumProxy): > o.values.append(value) > > except KeyError: > if isinstance(value, EnumProxy): > int.__setattr__(value, 'key', key) > else: > value = EnumProxy(key, value) > > super().__setitem__(value.key, value) > > class EnumMeta(type): > > @classmethod > def __prepare__(metacls, name, bases): > return EnumValues() > > def __new__(cls, name, bases, classdict): > classdict.sealed = True > > del_list = [] > > for v in classdict.values(): > if isinstance(v, EnumProxy) and v.used: > del_list.append(v) > > for v in del_list: > del classdict[v.key] > > result = type.__new__(cls, name, bases, dict(classdict)) > value = 0 > keys = {} > values = {} > > for v in classdict.values(): > if isinstance(v, EnumProxy) and not v.used: > if len(v.values) > 1: > raise AttributeError("Duplicate enum key '%s.%s'" % > (result.__qualname__, v.key,)) > > if v.values[0] is not None: > value = v.values[0] > > if isinstance(value, EnumValue): > > if (value.key is not None) and (value.key != v.key): > raise AttributeError("Assigned enum value to > non-matching key '%s': %r" % (v.key, value)) > > if value.owner is not None: > raise AttributeError("Assigned owned enum value to > key '%s': %r" % (v.key, value)) > > int.__setattr__(value, 'key', v.key) > v = value > > else: > v = EnumValue(v.key, value) > > setattr(result, v.key, v) > value += 1 > > if isinstance(v, EnumValue): > int.__setattr__(v, 'owner', result) > > if v in values: > raise AttributeError("Duplicate enum value %d for > keys: '%s' and '%s'" % (int(v), values[v].key, v.key)) > > keys[v.key] = v > values[v] = v > > enum = sorted(values) > > result._key_to_enum = collections.OrderedDict() > result._value_to_enum = values > > for e in enum: > result._key_to_enum[e.key] = e > > return result > > def __getitem__(self, key): > try: > key = operator.index(key) > except TypeError: > return self._key_to_enum[key] > else: > return self._value_to_enum[key] > > def _items(self): > return self._key_to_enum.items() > > def _keys(self): > return self._key_to_enum.keys() > > def _values(self): > return self._key_to_enum.values() > > def items(self): > return self._items() > > def keys(self): > return self._keys() > > def values(self): > return self._values() > > def __iter__(self): > return iter(self.values()) > > def __repr__(self): > r = super().__repr__() > r = [' r.append(str(self)) > r.append('>') > return ''.join(r) > > def __str__(self): > s = ['{'] > > for k, v in self.items(): > if s[-1][-1:] != '{': > s.append(', ') > > s.extend([k, ':', str(int(v))]) > > s.append('}') > return ''.join(s) > > class Enum(metaclass=EnumMeta): > def __getitem__(self, key): > cls = type(self) > return type(cls).__getitem__(cls, key) > > @classmethod > def items(cls): > return cls._items() > > @classmethod > def keys(cls): > return cls._keys() > > @classmethod > def values(cls): > return cls._values() > > def __iter__(self): > return iter(self.values()) > > def __repr__(self): > r = super().__repr__() > r = r.replace('object at 0x', 'enum at 0x') > r = [r[:-1], ' '] > r.append(str(self)) > r.append('>') > return ''.join(r) > > def __str__(self): > return str(type(self)) > > if __name__ == '__main__': > > class Color(Enum): > RED, GREEN, BLUE > ORANGE = "orange" > CYAN = 10 > MAGENTA > YELLOW > BLACK > > import unittest > > class TestEnum(unittest.TestCase): > > EXPECTED_KEY_ORDER = ('RED', 'GREEN', 'BLUE', 'CYAN', 'MAGENTA', > 'YELLOW', 'BLACK') > EXPECTED_INT_VALUE_ORDER = (0, 1, 2, 10, 11, 12, 13) > EXPECTED_ENUM_VALUE_ORDER = (Color.RED, Color.GREEN, Color.BLUE, > Color.CYAN, Color.MAGENTA, Color.YELLOW, Color.BLACK) > EXPECTED_ITEMS_ORDER = tuple(zip(EXPECTED_KEY_ORDER, > EXPECTED_ENUM_VALUE_ORDER)) > > def test_type(self): > self.assertIsInstance(Color.RED, int) > > def test_class_enum_values(self): > self.assertEqual(0, Color.RED) > self.assertEqual(1, Color.GREEN) > self.assertEqual(2, Color.BLUE) > self.assertEqual(10, Color.CYAN) > self.assertEqual(11, Color.MAGENTA) > self.assertEqual(12, Color.YELLOW) > self.assertEqual(13, Color.BLACK) > self.assertEqual("orange", Color.ORANGE) > > def test_instance_enum_values(self): > e = Color() > self.assertIs(Color.RED, e.RED) > self.assertIs(Color.GREEN, e.GREEN) > self.assertIs(Color.BLUE, e.BLUE) > self.assertIs(Color.CYAN, e.CYAN) > self.assertIs(Color.MAGENTA, e.MAGENTA) > self.assertIs(Color.YELLOW, e.YELLOW) > self.assertIs(Color.BLACK, e.BLACK) > self.assertIs(Color.ORANGE, e.ORANGE) > > def test_class_indexing(self): > self.assertIs(Color.CYAN, Color['CYAN']) > self.assertIs(Color.CYAN, Color[10]) > > def test_instance_indexing(self): > e = Color() > self.assertIs(Color.CYAN, e['CYAN']) > self.assertIs(Color.CYAN, e[10]) > > def test_class_keys(self): > self.assertEqual(self.EXPECTED_KEY_ORDER, tuple(Color.keys())) > > def test_instance_keys(self): > self.assertEqual(tuple(Color.keys()), tuple(Color().keys())) > > def test_class_values(self): > self.assertEqual(self.EXPECTED_INT_VALUE_ORDER, > tuple(Color.values())) > self.assertEqual(self.EXPECTED_ENUM_VALUE_ORDER, > tuple(Color.values())) > > def test_instance_values(self): > self.assertEqual(tuple(Color.values()), > tuple(Color().values())) > > def test_class_items(self): > self.assertEqual(self.EXPECTED_ITEMS_ORDER, > tuple(Color.items())) > > def test_instance_items(self): > self.assertEqual(tuple(Color.items()), tuple(Color().items())) > > def test_owner(self): > for e in Color: > self.assertIs(e.owner, Color) > > def test_class_str(self): > s = str(Color) > > for e in Color: > self.assertIn('%s:%d' % (e.key, int(e)), s) > > def test_instance_str(self): > self.assertEqual(str(Color), str(Color())) > > def test_class_repr(self): > r = repr(Color) > self.assertIn(Color.__qualname__, r) > self.assertIn(str(Color), r) > > def test_instance_repr(self): > e = Color() > r = repr(e) > self.assertIn(Color.__qualname__, r) > self.assertIn('at 0x', r) > self.assertIn(str(Color()), r) > > def _create_duplicate_key(self): > class DuplicateKey(Enum): > KEY, > KEY > > def test_duplicate_key(self): > self.assertRaises(AttributeError, self._create_duplicate_key) > > def _create_duplicate_value(self): > class DuplicateValue(Enum): > KEY1, > KEY2 = 0 > > def test_duplicate_value(self): > self.assertRaises(AttributeError, self._create_duplicate_value) > > def _assign_wrong_key(self): > class WrongKey(Enum): > KEY1 = EnumValue('KEY2', 0) > > def test_wrong_key(self): > self.assertRaises(AttributeError, self._assign_wrong_key) > > def test_unnamed_key1(self): > class UnnamedKey(Enum): > KEY1 = EnumValue(None, 5) > > self.assertEqual(UnnamedKey.KEY1, 5) > self.assertIs(UnnamedKey, UnnamedKey.KEY1.owner) > > def test_unnamed_key1(self): > unnamed = EnumValue(None, 5) > > class UnnamedKey(Enum): > KEY1 = unnamed > > self.assertEqual(UnnamedKey.KEY1, 5) > self.assertIs(UnnamedKey.KEY1, unnamed) > self.assertIs(UnnamedKey, UnnamedKey.KEY1.owner) > > def _assign_wrong_owner(self): > class WrongOwner(Enum): > KEY1 = Color.RED > > def test_wrong_owner(self): > self.assertRaises(AttributeError, self._assign_wrong_owner) > > unittest.main() > > Tim Delaney > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben+python at benfinney.id.au Fri Feb 1 05:28:33 2013 From: ben+python at benfinney.id.au (Ben Finney) Date: Fri, 01 Feb 2013 15:28:33 +1100 Subject: [Python-ideas] Definition Symmetry References: <510AFD06.1090305@pearwood.info> Message-ID: <7wa9roejym.fsf@benfinney.id.au> Jason Keene writes: > In a way they do the same thing, they both create an object > (function/class) from a suite and assign it to the name given after the > keyword (def/class). Sure they do totally different things with the suite > in creating the object, but in essence it's a name assignment. In a way, ?import? and ?break? do the same thing, they trigger the compiler to compile a particular set of code bytes. Sure the code bytes do totally different things, but in essence it's a statement. Less facetiously: You can blur your vision as much as you like to make ?class? and ?def? look similar, but that doesn't diminish the importance of the distinctions you're ignoring. -- \ ?Always do right. This will gratify some people, and astonish | `\ the rest.? ?Mark Twain | _o__) | Ben Finney From jasonkeene at gmail.com Fri Feb 1 06:21:07 2013 From: jasonkeene at gmail.com (Jason Keene) Date: Fri, 1 Feb 2013 00:21:07 -0500 Subject: [Python-ideas] Definition Symmetry In-Reply-To: <7wa9roejym.fsf@benfinney.id.au> References: <510AFD06.1090305@pearwood.info> <7wa9roejym.fsf@benfinney.id.au> Message-ID: Both class and function definitions produce callable objects that are assigned to a name with nearly identical syntax. I don't think your analogy with import/break statements is valid. Additionally, both definitions produce very similar byte codes: 3 MAKE_FUNCTION 0 6 STORE_FAST 0 (my_func) 9 MAKE_FUNCTION 0 12 CALL_FUNCTION 0 15 BUILD_CLASS 16 STORE_FAST 0 (MyClass) The only argument I see for requiring empty parens in function definitions is a historical one. On Thursday, January 31, 2013, Ben Finney wrote: > Jason Keene > writes: > > > In a way they do the same thing, they both create an object > > (function/class) from a suite and assign it to the name given after the > > keyword (def/class). Sure they do totally different things with the > suite > > in creating the object, but in essence it's a name assignment. > > In a way, ?import? and ?break? do the same thing, they trigger the > compiler to compile a particular set of code bytes. Sure the code bytes > do totally different things, but in essence it's a statement. > > Less facetiously: You can blur your vision as much as you like to make > ?class? and ?def? look similar, but that doesn't diminish the importance > of the distinctions you're ignoring. > > -- > \ ?Always do right. This will gratify some people, and astonish | > `\ the rest.? ?Mark Twain | > _o__) | > Ben Finney > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jasonkeene at gmail.com Fri Feb 1 06:41:32 2013 From: jasonkeene at gmail.com (Jason Keene) Date: Fri, 1 Feb 2013 00:41:32 -0500 Subject: [Python-ideas] Definition Symmetry In-Reply-To: References: <510AFD06.1090305@pearwood.info> <7wa9roejym.fsf@benfinney.id.au> Message-ID: >>> class MyClass(): pass >>> class MyClass: pass >>> def my_func(): pass >>> def my_func: pass File "", line 1 def my_func: ^ SyntaxError: invalid syntax http://www.youtube.com/watch?v=etuPF1yJRzg If only more of us grew up watching sesame street. On Friday, February 1, 2013, Jason Keene wrote: > Both class and function definitions produce callable objects that are > assigned to a name with nearly identical syntax. I don't think your > analogy with import/break statements is valid. Additionally, both > definitions produce very similar byte codes: > > 3 MAKE_FUNCTION 0 > 6 STORE_FAST 0 (my_func) > > 9 MAKE_FUNCTION 0 > 12 CALL_FUNCTION 0 > 15 BUILD_CLASS > 16 STORE_FAST 0 (MyClass) > > The only argument I see for requiring empty parens in function > definitions is a historical one. > > On Thursday, January 31, 2013, Ben Finney wrote: > >> Jason Keene writes: >> >> > In a way they do the same thing, they both create an object >> > (function/class) from a suite and assign it to the name given after the >> > keyword (def/class). Sure they do totally different things with the >> suite >> > in creating the object, but in essence it's a name assignment. >> >> In a way, ?import? and ?break? do the same thing, they trigger the >> compiler to compile a particular set of code bytes. Sure the code bytes >> do totally different things, but in essence it's a statement. >> >> Less facetiously: You can blur your vision as much as you like to make >> ?class? and ?def? look similar, but that doesn't diminish the importance >> of the distinctions you're ignoring. >> >> -- >> \ ?Always do right. This will gratify some people, and astonish | >> `\ the rest.? ?Mark Twain | >> _o__) | >> Ben Finney >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> http://mail.python.org/mailman/listinfo/python-ideas >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.brandl at gmx.net Fri Feb 1 10:12:24 2013 From: g.brandl at gmx.net (Georg Brandl) Date: Fri, 01 Feb 2013 10:12:24 +0100 Subject: [Python-ideas] constant/enum type in stdlib In-Reply-To: <20130130172627.32f64e71@pitrou.net> References: <51085AAB.6090303@canterbury.ac.nz> <5108A87D.9000207@canterbury.ac.nz> <20130130082639.0b28d7eb@pitrou.net> <20130130172627.32f64e71@pitrou.net> Message-ID: Am 30.01.2013 17:26, schrieb Antoine Pitrou: >> With a Python 3 metaclass that provides default values for *looked up* >> entries you could have this: >> >> class Color(Enum): >> RED, WHITE, BLUE > > This relies on tuple evaluation order, and would also evaluate any > other symbol looked up from inside the class body (which means I > cannot add anything else than enum symbols to the class). > > In other words, I'm afraid it would be somewhat fragile ;) And it breaks static code checkers like pyflakes. Georg From storchaka at gmail.com Fri Feb 1 10:27:16 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Fri, 01 Feb 2013 11:27:16 +0200 Subject: [Python-ideas] constant/enum type in stdlib In-Reply-To: References: Message-ID: I think that instead of the invention of implementation which covers all possible hypothetic scenarios, much more useful for the enums promotion would be if someone were to transform the existing constants in stdlib into enums with a minimum of the necessary capabilities. This will show which features are the most important, which interface is more convenient from user's point of view, and what difficulties have to be faced. From timothy.c.delaney at gmail.com Fri Feb 1 12:41:58 2013 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Fri, 1 Feb 2013 22:41:58 +1100 Subject: [Python-ideas] constant/enum type in stdlib In-Reply-To: References: Message-ID: On 1 February 2013 20:27, Serhiy Storchaka wrote: > I think that instead of the invention of implementation which covers all > possible hypothetic scenarios The scenarios I'm really interested in here is DRY; automatic assignment (i.e. not having to specify any values); manual assignment (which should allow any expression that is legal within a class definition) and being able to introspect the enum. Everything else falls out of supporting those scenarios within the normal restrictions of enums (in particular 1:1 mapping) and conforming to obvious interfaces. Tim Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Fri Feb 1 16:36:56 2013 From: barry at python.org (Barry Warsaw) Date: Fri, 1 Feb 2013 10:36:56 -0500 Subject: [Python-ideas] constant/enum type in stdlib References: <51085AAB.6090303@canterbury.ac.nz> <5108A87D.9000207@canterbury.ac.nz> <20130130082639.0b28d7eb@pitrou.net> Message-ID: <20130201103656.0b24c40d@anarchist.wooz.org> On Feb 01, 2013, at 03:18 PM, Tim Delaney wrote: >Last version (for now). I'm really interested in people's opinions on this. >For this version I've taken some inspiration from flufl.enum (but there >remains the major difference that these enums subclass int). Why not package it up and put it in PyPI? Better there than sitting in an email thread of some mailing list full of crazy people. :) -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From ned at nedbatchelder.com Fri Feb 1 17:43:55 2013 From: ned at nedbatchelder.com (Ned Batchelder) Date: Fri, 01 Feb 2013 11:43:55 -0500 Subject: [Python-ideas] Definition Symmetry In-Reply-To: References: <510AFD06.1090305@pearwood.info> <7wa9roejym.fsf@benfinney.id.au> Message-ID: <510BF0CB.6020705@nedbatchelder.com> On 2/1/2013 12:21 AM, Jason Keene wrote: > Both class and function definitions produce callable objects that are > assigned to a name with nearly identical syntax. I don't think your > analogy with import/break statements is valid. Additionally, both > definitions produce very similar byte codes: > > 3 MAKE_FUNCTION 0 > 6 STORE_FAST 0 (my_func) > > 9 MAKE_FUNCTION 0 > 12 CALL_FUNCTION 0 > 15 BUILD_CLASS > 16 STORE_FAST 0 (MyClass) > > The only argument I see for requiring empty parens in function > definitions is a historical one. That's because you're ignoring the main one: function calls require parens, so function definitions also require them. --Ned. > > On Thursday, January 31, 2013, Ben Finney wrote: > > Jason Keene > writes: > > > In a way they do the same thing, they both create an object > > (function/class) from a suite and assign it to the name given > after the > > keyword (def/class). Sure they do totally different things with > the suite > > in creating the object, but in essence it's a name assignment. > > In a way, 'import' and 'break' do the same thing, they trigger the > compiler to compile a particular set of code bytes. Sure the code > bytes > do totally different things, but in essence it's a statement. > > Less facetiously: You can blur your vision as much as you like to make > 'class' and 'def' look similar, but that doesn't diminish the > importance > of the distinctions you're ignoring. > > -- > \ "Always do right. This will gratify some people, and > astonish | > `\ the rest." ---Mark > Twain | > _o__) | > Ben Finney > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericsnowcurrently at gmail.com Fri Feb 1 20:13:41 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Fri, 1 Feb 2013 12:13:41 -0700 Subject: [Python-ideas] Definition Symmetry In-Reply-To: References: <510AFD06.1090305@pearwood.info> <7wa9roejym.fsf@benfinney.id.au> Message-ID: On Thu, Jan 31, 2013 at 10:21 PM, Jason Keene wrote: > Additionally, both definitions > produce very similar byte codes: > > 3 MAKE_FUNCTION 0 > 6 STORE_FAST 0 (my_func) > > 9 MAKE_FUNCTION 0 > 12 CALL_FUNCTION 0 > 15 BUILD_CLASS > 16 STORE_FAST 0 (MyClass) That's just an implementation detail. -eric From eliben at gmail.com Fri Feb 1 20:30:23 2013 From: eliben at gmail.com (Eli Bendersky) Date: Fri, 1 Feb 2013 11:30:23 -0800 Subject: [Python-ideas] constant/enum type in stdlib In-Reply-To: <20130201103656.0b24c40d@anarchist.wooz.org> References: <51085AAB.6090303@canterbury.ac.nz> <5108A87D.9000207@canterbury.ac.nz> <20130130082639.0b28d7eb@pitrou.net> <20130201103656.0b24c40d@anarchist.wooz.org> Message-ID: On Fri, Feb 1, 2013 at 7:36 AM, Barry Warsaw wrote: > On Feb 01, 2013, at 03:18 PM, Tim Delaney wrote: > > >Last version (for now). I'm really interested in people's opinions on > this. > >For this version I've taken some inspiration from flufl.enum (but there > >remains the major difference that these enums subclass int). > > Why not package it up and put it in PyPI? Better there than sitting in an > email thread of some mailing list full of crazy people. :) > > I would actually prefer a place where it's easy to see the code, comment on it and fork it like Bitbucket or Github. PyPI is good for other purposes... Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsbueno at python.org.br Fri Feb 1 20:47:14 2013 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Fri, 1 Feb 2013 17:47:14 -0200 Subject: [Python-ideas] constant/enum type in stdlib In-Reply-To: References: <51085AAB.6090303@canterbury.ac.nz> <5108A87D.9000207@canterbury.ac.nz> <20130130082639.0b28d7eb@pitrou.net> <20130201103656.0b24c40d@anarchist.wooz.org> Message-ID: On 1 February 2013 17:30, Eli Bendersky wrote: > > > > On Fri, Feb 1, 2013 at 7:36 AM, Barry Warsaw wrote: > >> On Feb 01, 2013, at 03:18 PM, Tim Delaney wrote: >> >> >Last version (for now). I'm really interested in people's opinions on >> this. >> >For this version I've taken some inspiration from flufl.enum (but there >> >remains the major difference that these enums subclass int). >> >> Why not package it up and put it in PyPI? Better there than sitting in an >> email thread of some mailing list full of crazy people. :) >> >> > I would actually prefer a place where it's easy to see the code, comment > on it and fork it like Bitbucket or Github. PyPI is good for other > purposes... > Indeed - there are plenty of nice enums on pypi already - my call for getting then to the stdlib is that we can get a reliable enum/constant behavior to be always there - not only for the constants in the stdlib, but for projects for which adding external dependencies would be expensive (as they currently have none) js -><- > > > Eli > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericsnowcurrently at gmail.com Fri Feb 1 21:09:48 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Fri, 1 Feb 2013 13:09:48 -0700 Subject: [Python-ideas] constant/enum type in stdlib In-Reply-To: References: <51085AAB.6090303@canterbury.ac.nz> <5108A87D.9000207@canterbury.ac.nz> <20130130082639.0b28d7eb@pitrou.net> <20130201103656.0b24c40d@anarchist.wooz.org> Message-ID: On Fri, Feb 1, 2013 at 12:30 PM, Eli Bendersky wrote: > On Fri, Feb 1, 2013 at 7:36 AM, Barry Warsaw wrote: >> Why not package it up and put it in PyPI? Better there than sitting in an >> email thread of some mailing list full of crazy people. :) > > I would actually prefer a place where it's easy to see the code, comment on > it and fork it like Bitbucket or Github. PyPI is good for other purposes... +1 And put it on PyPI to make it more accessible. -eric From ericsnowcurrently at gmail.com Fri Feb 1 21:11:39 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Fri, 1 Feb 2013 13:11:39 -0700 Subject: [Python-ideas] constant/enum type in stdlib In-Reply-To: References: Message-ID: On Fri, Feb 1, 2013 at 2:27 AM, Serhiy Storchaka wrote: > I think that instead of the invention of implementation which covers all > possible hypothetic scenarios, much more useful for the enums promotion > would be if someone were to transform the existing constants in stdlib into > enums with a minimum of the necessary capabilities. This will show which > features are the most important, which interface is more convenient from > user's point of view, and what difficulties have to be faced. +1 There's a lot of precedent (and good reasons) for starting with a minimal/basic implementation in the stdlib. -eric From jsbueno at python.org.br Fri Feb 1 21:19:49 2013 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Fri, 1 Feb 2013 18:19:49 -0200 Subject: [Python-ideas] constant/enum type in stdlib In-Reply-To: References: <51085AAB.6090303@canterbury.ac.nz> <5108A87D.9000207@canterbury.ac.nz> <20130130082639.0b28d7eb@pitrou.net> <20130201103656.0b24c40d@anarchist.wooz.org> Message-ID: On 1 February 2013 18:09, Eric Snow wrote: > On Fri, Feb 1, 2013 at 12:30 PM, Eli Bendersky wrote: > > On Fri, Feb 1, 2013 at 7:36 AM, Barry Warsaw wrote: > >> Why not package it up and put it in PyPI? Better there than sitting in > an > >> email thread of some mailing list full of crazy people. :) > > > > I would actually prefer a place where it's easy to see the code, comment > on > > it and fork it like Bitbucket or Github. PyPI is good for other > purposes... > > +1 > > And put it on PyPI to make it more accessible. > As soon as there are some more goodies in there beyond the enumerator and constants themselves. I can see that a namedtuple - like utility function could be interesting to create the enums - and a way to import the generated constatnts to the current global namespace that does not violate good pratices. I think that something like: class MyEnum(Enum): RED, GREEN, BLUE load_constants(MyEnum, globals() ) is a good solution that could respect both DRY and "explicit is better than implicit" and "Special cases aren't special enough to break the rules." js -><- > > -eric > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > -------------- next part -------------- An HTML attachment was scrubbed... URL: From timothy.c.delaney at gmail.com Fri Feb 1 21:28:57 2013 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Sat, 2 Feb 2013 07:28:57 +1100 Subject: [Python-ideas] constant/enum type in stdlib In-Reply-To: <20130201103656.0b24c40d@anarchist.wooz.org> References: <51085AAB.6090303@canterbury.ac.nz> <5108A87D.9000207@canterbury.ac.nz> <20130130082639.0b28d7eb@pitrou.net> <20130201103656.0b24c40d@anarchist.wooz.org> Message-ID: On 2 February 2013 02:36, Barry Warsaw wrote: > On Feb 01, 2013, at 03:18 PM, Tim Delaney wrote: > > >Last version (for now). I'm really interested in people's opinions on > this. > >For this version I've taken some inspiration from flufl.enum (but there > >remains the major difference that these enums subclass int). > > Why not package it up and put it in PyPI? Better there than sitting in an > email thread of some mailing list full of crazy people. :) > Yep - I intend to. Was just hacking away initially to see if what I wanted to achieve was feasible using Michael's metaclass as a base. It's in a Mercurial repo so I'll put it up on BitBucket in the next day or two and look at cleaning it up (documentation!) to put on PyPI. Tim Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsbueno at python.org.br Fri Feb 1 21:39:28 2013 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Fri, 1 Feb 2013 18:39:28 -0200 Subject: [Python-ideas] constant/enum type in stdlib In-Reply-To: References: <51085AAB.6090303@canterbury.ac.nz> <5108A87D.9000207@canterbury.ac.nz> <20130130082639.0b28d7eb@pitrou.net> <20130201103656.0b24c40d@anarchist.wooz.org> Message-ID: On 1 February 2013 18:28, Tim Delaney wrote: > On 2 February 2013 02:36, Barry Warsaw wrote: > >> On Feb 01, 2013, at 03:18 PM, Tim Delaney wrote: >> >> >Last version (for now). I'm really interested in people's opinions on >> this. >> >For this version I've taken some inspiration from flufl.enum (but there >> >remains the major difference that these enums subclass int). >> >> Why not package it up and put it in PyPI? Better there than sitting in an >> email thread of some mailing list full of crazy people. :) >> > > Yep - I intend to. Was just hacking away initially to see if what I wanted > to achieve was feasible using Michael's metaclass as a base. It's in a > Mercurial repo so I'll put it up on BitBucket in the next day or two and > look at cleaning it up (documentation!) to put on PyPI. > Indeed . .as I said, we'd better put together a pre-PEP and making this proof of concept impement most of it before going to Pypy - There are other requisites that I think aren't taken care of that come form the 2010 thread - for example, pickle-ability. js -><- > > Tim Delaney > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Fri Feb 1 22:04:46 2013 From: barry at python.org (Barry Warsaw) Date: Fri, 1 Feb 2013 16:04:46 -0500 Subject: [Python-ideas] constant/enum type in stdlib In-Reply-To: References: <51085AAB.6090303@canterbury.ac.nz> <5108A87D.9000207@canterbury.ac.nz> <20130130082639.0b28d7eb@pitrou.net> <20130201103656.0b24c40d@anarchist.wooz.org> Message-ID: <20130201160446.081bb6d3@anarchist.wooz.org> On Feb 01, 2013, at 11:30 AM, Eli Bendersky wrote: >I would actually prefer a place where it's easy to see the code, comment on >it and fork it like Bitbucket or Github. PyPI is good for other purposes... Well, sure, having the code in a publicly available vcs is always a good idea. My point really was that stuff ordinarily doesn't land in the stdlib until it's lived on PyPI for a while. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From ubershmekel at gmail.com Fri Feb 1 22:53:14 2013 From: ubershmekel at gmail.com (Yuval Greenfield) Date: Fri, 1 Feb 2013 23:53:14 +0200 Subject: [Python-ideas] PEP 3156: getting the socket or peer name from the transport In-Reply-To: References: <20130127122121.6b779ada@pitrou.net> <1359288997.3488.2.camel@localhost.localdomain> Message-ID: On Fri, Feb 1, 2013 at 3:58 AM, Guido van Rossum wrote: > hasattr() smells bad. It also has namespace issues (hasattr(transport, > "write") returns true) and if people forget to use it (perhaps because > the transport they normally use always has a certain attribute) their > code is brittle. Defining a new API with a string key signals clearly > that the value may or may not exist, and reminds people to test the > result for None. (Of course they can forget that too. But it still > feels different to me.) I understand. It's a good solution for clearly separating the standard transport api from the extra-and-specific api. We want to make it obvious when a protocol is breaking the transport abstraction - to promote playing nicely with different transports. -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Sat Feb 2 01:53:03 2013 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 02 Feb 2013 13:53:03 +1300 Subject: [Python-ideas] constant/enum type in stdlib In-Reply-To: References: <5108A87D.9000207@canterbury.ac.nz> <20130130082639.0b28d7eb@pitrou.net> <20130201103656.0b24c40d@anarchist.wooz.org> Message-ID: <510C636F.6060304@canterbury.ac.nz> Joao S. O. Bueno wrote: > class MyEnum(Enum): > RED, GREEN, BLUE > > load_constants(MyEnum, globals() ) The "obvious" way to spell this would be from MyEnum import * but it would be challenging to make that work, I suspect. :-( -- Greg From ethan at stoneleaf.us Sat Feb 2 02:18:53 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 01 Feb 2013 17:18:53 -0800 Subject: [Python-ideas] constant/enum type in stdlib In-Reply-To: <510C636F.6060304@canterbury.ac.nz> References: <5108A87D.9000207@canterbury.ac.nz> <20130130082639.0b28d7eb@pitrou.net> <20130201103656.0b24c40d@anarchist.wooz.org> <510C636F.6060304@canterbury.ac.nz> Message-ID: <510C697D.6020400@stoneleaf.us> On 02/01/2013 04:53 PM, Greg Ewing wrote: > Joao S. O. Bueno wrote: >> class MyEnum(Enum): >> RED, GREEN, BLUE >> >> load_constants(MyEnum, globals() ) > > The "obvious" way to spell this would be > > from MyEnum import * > > but it would be challenging to make that work, I suspect. :-( It's not too tough: 8<---- constants.py -------------------------------------------------- class Colors(object): BLACK = 0 RED = 1 GREEN = 2 BLUE = 3 __all__ = ('BLACK','RED','GREEN','BLUE') @classmethod def register(cls): import sys sys.modules['%s.%s' % (__name__, cls.__name__)] = cls() Colors.register() 8<---- constants.py -------------------------------------------------- --> from constants.Colors import * --> RED 1 --> BLUE 3 ~Ethan~ From greg.ewing at canterbury.ac.nz Sat Feb 2 02:41:01 2013 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 02 Feb 2013 14:41:01 +1300 Subject: [Python-ideas] constant/enum type in stdlib In-Reply-To: <510C697D.6020400@stoneleaf.us> References: <5108A87D.9000207@canterbury.ac.nz> <20130130082639.0b28d7eb@pitrou.net> <20130201103656.0b24c40d@anarchist.wooz.org> <510C636F.6060304@canterbury.ac.nz> <510C697D.6020400@stoneleaf.us> Message-ID: <510C6EAD.80009@canterbury.ac.nz> Ethan Furman wrote: > On 02/01/2013 04:53 PM, Greg Ewing wrote: > >> The "obvious" way to spell this would be >> >> from MyEnum import * >> >> but it would be challenging to make that work, I suspect. :-( > > > It's not too tough: Yeah, I just took up my own challenge and came up with something similar (apologies for the Python 2): class MetaEnum(type): def __init__(self, name, bases, dict): type.__init__(self, name, bases, dict) import sys sys.modules[name] = self class Enum(object): __metaclass__ = MetaEnum class MyEnum(Enum): RED = 0 GREEN = 1 BLUE = 2 from MyEnum import * print RED print GREEN print BLUE I left off the module name so that you don't have to qualify the import. A more general version would qualify it with all but the last component of the module name, so you can import it relative to the containing module. -- Greg From gotoalanlu at gmail.com Sat Feb 2 07:24:34 2013 From: gotoalanlu at gmail.com (Hua Lu) Date: Sat, 2 Feb 2013 00:24:34 -0600 Subject: [Python-ideas] frozenset literals Message-ID: This is similar to another proposal: http://mail.python.org/pipermail/python-3000/2008-January/011798.html Anyway, I was using ast.literal_eval and attempted to use frozenset({...}) as a key in a dictionary, which failed, because frozenset isn't a literal (though putting frozenset in the environment would be a security risk). I am currently working around this with tuples, but I'd like a literal for representing frozensets as well. I also use frozensets elsewhere in the code in ways similar to Raymond's original suggestion. Perhaps something like f{...} for declaring frozenset( comprehension)? literals? -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Sat Feb 2 08:34:27 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 02 Feb 2013 08:34:27 +0100 Subject: [Python-ideas] frozenset literals In-Reply-To: References: Message-ID: Hua Lu, 02.02.2013 07:24: > This is similar to another proposal: > http://mail.python.org/pipermail/python-3000/2008-January/011798.html > > Anyway, I was using ast.literal_eval and attempted to use frozenset({...}) > as a key in a dictionary, which failed, because frozenset isn't a literal This has nothing to do with being a literal or not. The way you created your frozenset doesn't impact its behaviour. Could you give an example of what's not working for you? Frozensets as dict keys work just fine for me: Python 3.2.3 (default, Oct 19 2012, 19:53:16) [GCC 4.7.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> frozenset({1,2,3}) frozenset({1, 2, 3}) >>> hash(frozenset({1,2,3})) -7699079583225461316 >>> { frozenset({1,2,3}) : 1 } {frozenset({1, 2, 3}): 1} Stefan From gotoalanlu at gmail.com Sat Feb 2 08:38:06 2013 From: gotoalanlu at gmail.com (Hua Lu) Date: Sat, 2 Feb 2013 01:38:06 -0600 Subject: [Python-ideas] frozenset literals In-Reply-To: References: Message-ID: >>> import ast >>> ast.literal_eval("{ 'key': 'val' }") {'key': 'val'} >>> ast.literal_eval("{ ('key',): 'val' }") {('key',): 'val'} >>> ast.literal_eval("{ frozenset({'key'}): 'val' }") Traceback (most recent call last): File "", line 1, in File "/usr/lib/python3.3/ast.py", line 86, in literal_eval return _convert(node_or_string) File "/usr/lib/python3.3/ast.py", line 63, in _convert in zip(node.keys, node.values)) File "/usr/lib/python3.3/ast.py", line 62, in return dict((_convert(k), _convert(v)) for k, v File "/usr/lib/python3.3/ast.py", line 85, in _convert raise ValueError('malformed node or string: ' + repr(node)) ValueError: malformed node or string: <_ast.Call object at 0x7f865a8c1450> On Sat, Feb 2, 2013 at 1:34 AM, Stefan Behnel wrote: > Hua Lu, 02.02.2013 07:24: > > This is similar to another proposal: > > http://mail.python.org/pipermail/python-3000/2008-January/011798.html > > > > Anyway, I was using ast.literal_eval and attempted to use > frozenset({...}) > > as a key in a dictionary, which failed, because frozenset isn't a literal > > This has nothing to do with being a literal or not. The way you created > your frozenset doesn't impact its behaviour. > > Could you give an example of what's not working for you? Frozensets as dict > keys work just fine for me: > > Python 3.2.3 (default, Oct 19 2012, 19:53:16) > [GCC 4.7.2] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> frozenset({1,2,3}) > frozenset({1, 2, 3}) > >>> hash(frozenset({1,2,3})) > -7699079583225461316 > >>> { frozenset({1,2,3}) : 1 } > {frozenset({1, 2, 3}): 1} > > Stefan > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > -------------- next part -------------- An HTML attachment was scrubbed... URL: From shane at umbrellacode.com Sat Feb 2 12:46:28 2013 From: shane at umbrellacode.com (Shane Green) Date: Sat, 02 Feb 2013 03:46:28 -0800 Subject: [Python-ideas] for line in input with open(path) as input... Message-ID: <510CFC94.5000502@umbrellacode.com> with open(path) as input: for line in input: do(line) Using with to create reference to opened file returned by open() so it could temporarily be assigned to input for the sole purpose of iterating its contents never sat very well with me. * The context manager returned by open() exists only to create the context and return reference "input"; * the context and code block created by the "with" only exists for inner "for" loop's code block to execute in. Now, given a generator function: def iterwith(cm): with cm as context: if context is None: context = cm for item in context: yield item The previous code can be turned into: for line in iterwith(open(path)): do(line) So, questions: - Is there anything inherently wrong with the idea, or does it exist? - Is it a generally useful tool, or are the examples limited to files? - Is it possible a more general mechanism could have value, such as: for line in file with open(path) as file: do(line) The preceding could be leveraged to different effect: for line in file with locked(path): write(path + ".out", line) Or, for line in input with nested(open(path),lock,open(opath)) as input,locked,output: output.write(line) To revisit the original purpose of "with", this seems to cleanly address a very common scenario wherein: resource = create_resource() try: for item in resource: do_something(resource, item) except: raise finally: cleanup() # Standard with approach... with create_resource() as resource: for item in resource: do_something(resource, item) # With for loop as context... for item in resource with create_resource() as resource: do_something(resource, item) And, given the translation into statements, maybe even crazy stuff... [line for line in file with open(path) as file] J/K, I think. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sat Feb 2 13:46:26 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 02 Feb 2013 23:46:26 +1100 Subject: [Python-ideas] for line in input with open(path) as input... In-Reply-To: <510CFC94.5000502@umbrellacode.com> References: <510CFC94.5000502@umbrellacode.com> Message-ID: <510D0AA2.2060805@pearwood.info> On 02/02/13 22:46, Shane Green wrote: > with open(path) as input: > for line in input: > do(line) > > Using with to create reference to opened file returned by open() so it could temporarily be assigned to input for the sole purpose of iterating its contents never sat very well with me. It's not the *sole* purpose. If all you want it to iterate over the file, you can do this: for line in open(path): ... and no context manager is created. The context manager is also responsible for closing the file immediately you exit the block, without waiting for the caller to manually close it, or the garbage collector to (eventually) close it. So it is not *solely* for iteration. File context managers can also be used for more than just iteration: with open(path) as input: text = input.read() with open(path, 'r+') as output: output.write('ZZ') and so forth. > * The context manager returned by open() exists only to create the > context and return reference "input"; > * the context and code block created by the "with" only exists for > inner "for" loop's code block to execute in. I don't understand that objection. As I see it, that's a bit like saying "the len function exists only to get the length of objects". What did you expect the context manager to exist for if not to do the things you say? What am I missing? -- Steven From rosuav at gmail.com Sat Feb 2 14:01:00 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 3 Feb 2013 00:01:00 +1100 Subject: [Python-ideas] for line in input with open(path) as input... In-Reply-To: <510D0AA2.2060805@pearwood.info> References: <510CFC94.5000502@umbrellacode.com> <510D0AA2.2060805@pearwood.info> Message-ID: On Sat, Feb 2, 2013 at 11:46 PM, Steven D'Aprano wrote: > It's not the *sole* purpose. If all you want it to iterate over the file, > you can do this: > > for line in open(path): > ... If I understand the OP, the issue is that the 'with' creates a name binding and an indentation level for no purpose; it's like doing this: f = open(path) for line in f: ... In that instance, it's possible to inline the function call and use its result directly; it would be nice to be able to do the same with a context manager. However, since 'with' isn't an expression, it's not possible to directly inline the two. I think the utility function iterwith() is a good - and probably the best - method; it demands nothing special from the language, and works quite happily. ChrisA From stefan_ml at behnel.de Sat Feb 2 14:06:31 2013 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 02 Feb 2013 14:06:31 +0100 Subject: [Python-ideas] frozenset literals In-Reply-To: References: Message-ID: Hua Lu, 02.02.2013 08:38: > >>> import ast > >>> ast.literal_eval("{ 'key': 'val' }") > {'key': 'val'} > >>> ast.literal_eval("{ ('key',): 'val' }") > {('key',): 'val'} > >>> ast.literal_eval("{ frozenset({'key'}): 'val' }") > Traceback (most recent call last): > File "", line 1, in > File "/usr/lib/python3.3/ast.py", line 86, in literal_eval > return _convert(node_or_string) > File "/usr/lib/python3.3/ast.py", line 63, in _convert > in zip(node.keys, node.values)) > File "/usr/lib/python3.3/ast.py", line 62, in > return dict((_convert(k), _convert(v)) for k, v > File "/usr/lib/python3.3/ast.py", line 85, in _convert > raise ValueError('malformed node or string: ' + repr(node)) > ValueError: malformed node or string: <_ast.Call object at 0x7f865a8c1450> So, why exactly are you using ast.literal_eval() to evaluate a non-literal expression? Stefan PS: please reply without top-posting. From shane at umbrellacode.com Sat Feb 2 14:21:38 2013 From: shane at umbrellacode.com (Shane Green) Date: Sat, 02 Feb 2013 05:21:38 -0800 Subject: [Python-ideas] for line in input with open(path) as input... In-Reply-To: <510D0AA2.2060805@pearwood.info> References: <510CFC94.5000502@umbrellacode.com> <510D0AA2.2060805@pearwood.info> Message-ID: <510D12E2.3040000@umbrellacode.com> Sorry, I was definitely unclear: I understand that isn't their sole purpose, and what role they play, in both the iteration examples and others, and I don't mean to "object" to anything, nor was I suggesting context-manager current implementation was uneeded, etc. My point is that, in the examples I listed, the sole purpose of the context-manager is to provide a context the for loop will execute in. with manager as context: # No statements here... for items in context: # blah # no statements here... I was suggesting this scenario, wherein the body of a for loop is, in fact, the block of code that acts upon some resource or set of resource being managed by a context manager, might be common enough to warrant a closer look, as the more "pythonic" approach might accept the for loop's body as the with statement's code block context. with open(file) as input: # why have input exist here, if it's not used, other than... for line in input: # do something... # why have input--the context--exist outside of me? # why have input exist here? And finally, why define the outer with block, for the sole pupose of containing the inner "for in.." loop's block? > Steven D'Aprano > February 2, 2013 4:46 AM > On 02/02/13 22:46, Shane Green wrote: >> with open(path) as input: >> for line in input: >> do(line) >> >> Using with to create reference to opened file returned by open() so >> it could temporarily be assigned to input for the sole purpose of >> iterating its contents never sat very well with me. > > It's not the *sole* purpose. If all you want it to iterate over the > file, you can do this: > > for line in open(path): > ... > > > and no context manager is created. The context manager is also > responsible for closing the file immediately you exit the block, > without waiting for the caller to manually close it, or the garbage > collector to (eventually) close it. So it is not *solely* for iteration. > > File context managers can also be used for more than just iteration: > > with open(path) as input: > text = input.read() > > > with open(path, 'r+') as output: > output.write('ZZ') > > and so forth. > > >> * The context manager returned by open() exists only to create the >> context and return reference "input"; >> * the context and code block created by the "with" only exists for >> inner "for" loop's code block to execute in. > > > I don't understand that objection. As I see it, that's a bit like > saying "the len function exists only to get the length of objects". > What did you expect the context manager to exist for if not to do the > things you say? > > What am I missing? > > > > > Shane Green > February 2, 2013 3:46 AM > with open(path) as input: > for line in input: > do(line) > > Using with to create reference to opened file returned by open() so it > could temporarily be assigned to input for the sole purpose of > iterating its contents never sat very well with me. > > * The context manager returned by open() exists only to create the > context and return reference "input"; > * the context and code block created by the "with" only exists for > inner "for" loop's code block to execute in. > > Now, given a generator function: > > def iterwith(cm): > with cm as context: > if context is None: > context = cm > for item in context: > yield item > > The previous code can be turned into: > > for line in iterwith(open(path)): > do(line) > > So, questions: > - Is there anything inherently wrong with the idea, or does it exist? > - Is it a generally useful tool, or are the examples limited to files? > - Is it possible a more general mechanism could have value, such as: > > for line in file with open(path) as file: > do(line) > > > The preceding could be leveraged to different effect: > > for line in file with locked(path): > write(path + ".out", line) > > Or, > for line in input with nested(open(path),lock,open(opath)) as > input,locked,output: > output.write(line) > > To revisit the original purpose of "with", this seems to cleanly > address a very common scenario wherein: > > resource = create_resource() > try: > for item in resource: > do_something(resource, item) > except: > raise > finally: > cleanup() > > # Standard with approach... > with create_resource() as resource: > for item in resource: > do_something(resource, item) > > # With for loop as context... > for item in resource with create_resource() as resource: > do_something(resource, item) > > And, given the translation into statements, maybe even crazy stuff... > > [line for line in file with open(path) as file] > > J/K, I think. > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: compose-unknown-contact.jpg Type: image/jpeg Size: 770 bytes Desc: not available URL: From gotoalanlu at gmail.com Sat Feb 2 14:33:17 2013 From: gotoalanlu at gmail.com (Hua Lu) Date: Sat, 2 Feb 2013 07:33:17 -0600 Subject: [Python-ideas] frozenset literals In-Reply-To: References: Message-ID: > So, why exactly are you using ast.literal_eval() to evaluate a non-literal > expression? > > Stefan > > PS: please reply without top-posting. Sorry, I'm quite new to mailing lists. I want to be able to use frozensets in ast.literal_eval portions (especially as semantically correct keys for certain domains where tuples have too much structure). Writing an ast filter just to accommodate a builtin type feels way too hacky to me, and there are legitimate reasons for having frozenset literals exist, some of which are discussed in the link to that 2008 discussion. Also, there are serious hacks for eval, and after seeing some of those... ay caramba. Thanks for being patient with my newbishness. From storchaka at gmail.com Sat Feb 2 14:50:27 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 02 Feb 2013 15:50:27 +0200 Subject: [Python-ideas] for line in input with open(path) as input... In-Reply-To: References: <510CFC94.5000502@umbrellacode.com> <510D0AA2.2060805@pearwood.info> Message-ID: On 02.02.13 15:01, Chris Angelico wrote: > I think the utility function iterwith() is a good - and probably the > best - method; it demands nothing special from the language, and works > quite happily. When should iterwith() call __exit__()? From rosuav at gmail.com Sat Feb 2 14:52:50 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 3 Feb 2013 00:52:50 +1100 Subject: [Python-ideas] for line in input with open(path) as input... In-Reply-To: References: <510CFC94.5000502@umbrellacode.com> <510D0AA2.2060805@pearwood.info> Message-ID: On Sun, Feb 3, 2013 at 12:50 AM, Serhiy Storchaka wrote: > On 02.02.13 15:01, Chris Angelico wrote: >> >> I think the utility function iterwith() is a good - and probably the >> best - method; it demands nothing special from the language, and works >> quite happily. > > > When should iterwith() call __exit__()? According to the OP's posted code, as soon as the iterable runs out. Not sure what happens if you don't exhaust it but I'm sure generator functions have already solved that, too. ChrisA From storchaka at gmail.com Sat Feb 2 14:53:15 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 02 Feb 2013 15:53:15 +0200 Subject: [Python-ideas] frozenset literals In-Reply-To: References: Message-ID: On 02.02.13 08:24, Hua Lu wrote: > Perhaps something like f{...} for declaring frozenset( comprehension)? > literals? You need also a syntax to specify an empty set and an empty frozenset. From shane at umbrellacode.com Sat Feb 2 14:55:14 2013 From: shane at umbrellacode.com (Shane Green) Date: Sat, 02 Feb 2013 05:55:14 -0800 Subject: [Python-ideas] for line in input with open(path) as input... In-Reply-To: References: <510CFC94.5000502@umbrellacode.com> <510D0AA2.2060805@pearwood.info> Message-ID: <510D1AC2.1000304@umbrellacode.com> iterwith() doesn't need to: __exit__() is invoked automatically when iteration completes and the loop exits. > Serhiy Storchaka > February 2, 2013 5:50 AM > > > When should iterwith() call __exit__()? > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > Chris Angelico > February 2, 2013 5:01 AM > > If I understand the OP, the issue is that the 'with' creates a name > binding and an indentation level for no purpose; it's like doing this: > > f = open(path) > for line in f: > ... > > In that instance, it's possible to inline the function call and use > its result directly; it would be nice to be able to do the same with a > context manager. However, since 'with' isn't an expression, it's not > possible to directly inline the two. > > I think the utility function iterwith() is a good - and probably the > best - method; it demands nothing special from the language, and works > quite happily. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > Steven D'Aprano > February 2, 2013 4:46 AM > On 02/02/13 22:46, Shane Green wrote: >> with open(path) as input: >> for line in input: >> do(line) >> >> Using with to create reference to opened file returned by open() so >> it could temporarily be assigned to input for the sole purpose of >> iterating its contents never sat very well with me. > > It's not the *sole* purpose. If all you want it to iterate over the > file, you can do this: > > for line in open(path): > ... > > > and no context manager is created. The context manager is also > responsible for closing the file immediately you exit the block, > without waiting for the caller to manually close it, or the garbage > collector to (eventually) close it. So it is not *solely* for iteration. > > File context managers can also be used for more than just iteration: > > with open(path) as input: > text = input.read() > > > with open(path, 'r+') as output: > output.write('ZZ') > > and so forth. > > >> * The context manager returned by open() exists only to create the >> context and return reference "input"; >> * the context and code block created by the "with" only exists for >> inner "for" loop's code block to execute in. > > > I don't understand that objection. As I see it, that's a bit like > saying "the len function exists only to get the length of objects". > What did you expect the context manager to exist for if not to do the > things you say? > > What am I missing? > > > > > Shane Green > February 2, 2013 3:46 AM > with open(path) as input: > for line in input: > do(line) > > Using with to create reference to opened file returned by open() so it > could temporarily be assigned to input for the sole purpose of > iterating its contents never sat very well with me. > > * The context manager returned by open() exists only to create the > context and return reference "input"; > * the context and code block created by the "with" only exists for > inner "for" loop's code block to execute in. > > Now, given a generator function: > > def iterwith(cm): > with cm as context: > if context is None: > context = cm > for item in context: > yield item > > The previous code can be turned into: > > for line in iterwith(open(path)): > do(line) > > So, questions: > - Is there anything inherently wrong with the idea, or does it exist? > - Is it a generally useful tool, or are the examples limited to files? > - Is it possible a more general mechanism could have value, such as: > > for line in file with open(path) as file: > do(line) > > > The preceding could be leveraged to different effect: > > for line in file with locked(path): > write(path + ".out", line) > > Or, > for line in input with nested(open(path),lock,open(opath)) as > input,locked,output: > output.write(line) > > To revisit the original purpose of "with", this seems to cleanly > address a very common scenario wherein: > > resource = create_resource() > try: > for item in resource: > do_something(resource, item) > except: > raise > finally: > cleanup() > > # Standard with approach... > with create_resource() as resource: > for item in resource: > do_something(resource, item) > > # With for loop as context... > for item in resource with create_resource() as resource: > do_something(resource, item) > > And, given the translation into statements, maybe even crazy stuff... > > [line for line in file with open(path) as file] > > J/K, I think. > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: compose-unknown-contact.jpg Type: image/jpeg Size: 770 bytes Desc: not available URL: From shane at umbrellacode.com Sat Feb 2 15:05:19 2013 From: shane at umbrellacode.com (Shane Green) Date: Sat, 02 Feb 2013 06:05:19 -0800 Subject: [Python-ideas] for line in input with open(path) as input... In-Reply-To: <510D1C4B.1020806@umbrellacode.com> References: <510CFC94.5000502@umbrellacode.com> <510D0AA2.2060805@pearwood.info> <510D1AC2.1000304@umbrellacode.com> <510D1C4B.1020806@umbrellacode.com> Message-ID: <510D1D1F.4080206@umbrellacode.com> *From:* Serhiy Storchaka *Date:* February 2, 2013 5:50 AM *To:* python-ideas at python.org *Subject:* [Python-ideas] for line in input with open(path) as input... When should iterwith() call __exit__()? *From:* Shane Green *Date:* February 2, 2013 5:55 AM *To:* Serhiy Storchaka *CC:* python-ideas at python.org *Subject:* [Python-ideas] for line in input with open(path) as input... iterwith() doesn't need to: __exit__() is invoked automatically when iteration completes and the loop exits. > I hadn't thought about the fact the context was created inside the > generator... Nonetheless, cleanup would be automatic, always; and the > generator should exit eventually on close() if not exhausted, I > believe (need to review the details on gen cleanup). > > > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: compose-unknown-contact.jpg Type: image/jpeg Size: 770 bytes Desc: not available URL: From ncoghlan at gmail.com Sat Feb 2 15:18:23 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 3 Feb 2013 00:18:23 +1000 Subject: [Python-ideas] for line in input with open(path) as input... In-Reply-To: References: <510CFC94.5000502@umbrellacode.com> <510D0AA2.2060805@pearwood.info> Message-ID: The with statement block is needed to define *when* cleanup happens (unconditionally at the end of the block). The "iterwith" generator is currently pointless, as it results in nondeterministic cleanup of the context manager, so you may as well not bother and just rely on the underlying iterable's nondeterministic cleanup. We're never going to add cleanup semantics directly to for loops because: - separation of concerns is a good design principle - Indentation levels are not a limited resource (anyone that thinks they are may be forgetting that factoring out context managers, iterators and subfunctions gives you more of them, and that judicious use of early returns and continue statements can avoid wasting them) - we already considered it when initially designing the with statement and decided it was a bad idea. I forget where that last part is written up. If it's not in PEP 343, 342, 346 or 340 (the full set of PEPs that led to the current with statement and contextlib.contextmanager designs), it should be in one of the threads they reference. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sat Feb 2 15:20:38 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 03 Feb 2013 01:20:38 +1100 Subject: [Python-ideas] frozenset literals In-Reply-To: References: Message-ID: <510D20B6.5040705@pearwood.info> On 03/02/13 00:06, Stefan Behnel wrote: > Hua Lu, 02.02.2013 08:38: >>>>> import ast >>>>> ast.literal_eval("{ 'key': 'val' }") >> {'key': 'val'} >>>>> ast.literal_eval("{ ('key',): 'val' }") >> {('key',): 'val'} >>>>> ast.literal_eval("{ frozenset({'key'}): 'val' }") >> Traceback (most recent call last): >> File "", line 1, in >> File "/usr/lib/python3.3/ast.py", line 86, in literal_eval >> return _convert(node_or_string) >> File "/usr/lib/python3.3/ast.py", line 63, in _convert >> in zip(node.keys, node.values)) >> File "/usr/lib/python3.3/ast.py", line 62, in >> return dict((_convert(k), _convert(v)) for k, v >> File "/usr/lib/python3.3/ast.py", line 85, in _convert >> raise ValueError('malformed node or string: ' + repr(node)) >> ValueError: malformed node or string:<_ast.Call object at 0x7f865a8c1450> > > So, why exactly are you using ast.literal_eval() to evaluate a non-literal > expression? Stefan, you did ask Hua Lu to show an example of what isn't working. It's just a demonstration of what doesn't work -- you can't create a frozenset using ast.literal_eval. I think Hua Lu's original post made it quite clear. He wishes there to be a frozenset literal, because currently there is no way to have ast.literal_eval evaluate something containing a frozenset. Because there is no frozenset literal. I think Raymond Hettinger's proposal back in 2008: http://mail.python.org/pipermail/python-3000/2008-January/011798.html and the following thread is worth reading. Guido even pronounced his agreement: http://mail.python.org/pipermail/python-3000/2008-January/011814.html but then changed his mind (as did Raymond). So the status quo remains. Unfortunately the proposal to use f{ ... } for frozen sets cannot work within the constraints of Python's lexer: http://mail.python.org/pipermail/python-3000/2008-January/011838.html Unfortunately we're running out of useful, easy to enter symbols for literals. Until such time (Python4000 perhaps, or more likely Python5000) as we can use a rich set of Unicode literals, I don't think there is any clear way to have a frozenset literal. -- Steven From ncoghlan at gmail.com Sat Feb 2 15:28:55 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 3 Feb 2013 00:28:55 +1000 Subject: [Python-ideas] frozenset literals In-Reply-To: References: Message-ID: FWIW, I could personally tolerate the introduction of s{} and fs{} literals. We'd just declare the "s" prefix optional for non-empty sets to match the current rules. Encouraging the use of ast.literal_eval() over the security nightmare that is eval() would be more than enough justification for me. (As a syntax change, the idea would still need a PEP, though) Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sat Feb 2 15:28:56 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 2 Feb 2013 15:28:56 +0100 Subject: [Python-ideas] frozenset literals References: <510D20B6.5040705@pearwood.info> Message-ID: <20130202152856.0ab823b9@pitrou.net> On Sun, 03 Feb 2013 01:20:38 +1100 Steven D'Aprano wrote: > > Unfortunately the proposal to use f{ ... } for frozen sets cannot work within the constraints of Python's lexer: > > http://mail.python.org/pipermail/python-3000/2008-January/011838.html > > Unfortunately we're running out of useful, easy to enter symbols for literals. Until such time (Python4000 perhaps, or more likely Python5000) as we can use a rich set of Unicode literals, I don't think there is any clear way to have a frozenset literal. I'm not sure what the point is. frozensets are too rarely used to deserve a literal notation. Regards Antoine. From solipsis at pitrou.net Sat Feb 2 15:35:37 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 2 Feb 2013 15:35:37 +0100 Subject: [Python-ideas] frozenset literals References: Message-ID: <20130202153537.1b5fca12@pitrou.net> On Sun, 3 Feb 2013 00:28:55 +1000 Nick Coghlan wrote: > FWIW, I could personally tolerate the introduction of s{} and fs{} > literals. We'd just declare the "s" prefix optional for non-empty sets to > match the current rules. > > Encouraging the use of ast.literal_eval() over the security nightmare that > is eval() would be more than enough justification for me. (As a syntax > change, the idea would still need a PEP, though) If it is enough of a justification, then why not literal support for more useful datatypes (decimal, datetime...) rather than frozenset? Regards Antoine. From ncoghlan at gmail.com Sat Feb 2 15:40:47 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 3 Feb 2013 00:40:47 +1000 Subject: [Python-ideas] frozenset literals In-Reply-To: <510D20B6.5040705@pearwood.info> References: <510D20B6.5040705@pearwood.info> Message-ID: On 3 Feb 2013 00:21, "Steven D'Aprano" wrote: > > On 03/02/13 00:06, Stefan Behnel wrote: >> >> Hua Lu, 02.02.2013 08:38: >>>>>> >>>>>> import ast >>>>>> ast.literal_eval("{ 'key': 'val' }") >>> >>> {'key': 'val'} >>>>>> >>>>>> ast.literal_eval("{ ('key',): 'val' }") >>> >>> {('key',): 'val'} >>>>>> >>>>>> ast.literal_eval("{ frozenset({'key'}): 'val' }") >>> >>> Traceback (most recent call last): >>> File "", line 1, in >>> File "/usr/lib/python3.3/ast.py", line 86, in literal_eval >>> return _convert(node_or_string) >>> File "/usr/lib/python3.3/ast.py", line 63, in _convert >>> in zip(node.keys, node.values)) >>> File "/usr/lib/python3.3/ast.py", line 62, in >>> return dict((_convert(k), _convert(v)) for k, v >>> File "/usr/lib/python3.3/ast.py", line 85, in _convert >>> raise ValueError('malformed node or string: ' + repr(node)) >>> ValueError: malformed node or string:<_ast.Call object at 0x7f865a8c1450> >> >> >> So, why exactly are you using ast.literal_eval() to evaluate a non-literal >> expression? > > > > Stefan, you did ask Hua Lu to show an example of what isn't working. It's just a demonstration of what doesn't work -- you can't create a frozenset using ast.literal_eval. > > I think Hua Lu's original post made it quite clear. He wishes there to be a frozenset literal, because currently there is no way to have ast.literal_eval evaluate something containing a frozenset. Because there is no frozenset literal. > > I think Raymond Hettinger's proposal back in 2008: > > http://mail.python.org/pipermail/python-3000/2008-January/011798.html > > and the following thread is worth reading. Guido even pronounced his agreement: > > http://mail.python.org/pipermail/python-3000/2008-January/011814.html > > but then changed his mind (as did Raymond). So the status quo remains. > > Unfortunately the proposal to use f{ ... } for frozen sets cannot work within the constraints of Python's lexer: > > http://mail.python.org/pipermail/python-3000/2008-January/011838.html To clarify Guido's comment in that post, I'm fairly sure it *can* be made to work, it just won' t be the same way that string prefixes work (because the contents of dict and set displays are not opaque to the compiler the way string contents are). The hypothetical "What if we want to allow expr{} as a general construct?" objection needs to be balanced against the immediate value of a more expressive language subset for use in ast.literal_eval(). Cheers, Nick. > > Unfortunately we're running out of useful, easy to enter symbols for literals. Until such time (Python4000 perhaps, or more likely Python5000) as we can use a rich set of Unicode literals, I don't think there is any clear way to have a frozenset literal. > > > > -- > Steven > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sat Feb 2 15:46:21 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 2 Feb 2013 15:46:21 +0100 Subject: [Python-ideas] frozenset literals References: <510D20B6.5040705@pearwood.info> Message-ID: <20130202154621.1133530a@pitrou.net> On Sun, 3 Feb 2013 00:40:47 +1000 Nick Coghlan wrote: > On 3 Feb 2013 00:21, "Steven D'Aprano" wrote: > > > > On 03/02/13 00:06, Stefan Behnel wrote: > >> > >> Hua Lu, 02.02.2013 08:38: > >>>>>> > >>>>>> import ast > >>>>>> ast.literal_eval("{ 'key': 'val' }") > >>> > >>> {'key': 'val'} > >>>>>> > >>>>>> ast.literal_eval("{ ('key',): 'val' }") > >>> > >>> {('key',): 'val'} > >>>>>> > >>>>>> ast.literal_eval("{ frozenset({'key'}): 'val' }") > >>> > >>> Traceback (most recent call last): > >>> File "", line 1, in > >>> File "/usr/lib/python3.3/ast.py", line 86, in literal_eval > >>> return _convert(node_or_string) > >>> File "/usr/lib/python3.3/ast.py", line 63, in _convert > >>> in zip(node.keys, node.values)) > >>> File "/usr/lib/python3.3/ast.py", line 62, in > >>> return dict((_convert(k), _convert(v)) for k, v > >>> File "/usr/lib/python3.3/ast.py", line 85, in _convert > >>> raise ValueError('malformed node or string: ' + repr(node)) > >>> ValueError: malformed node or string:<_ast.Call object at > 0x7f865a8c1450> > >> > >> > >> So, why exactly are you using ast.literal_eval() to evaluate a > non-literal > >> expression? > > > > > > > > Stefan, you did ask Hua Lu to show an example of what isn't working. It's > just a demonstration of what doesn't work -- you can't create a frozenset > using ast.literal_eval. > > > > I think Hua Lu's original post made it quite clear. He wishes there to be > a frozenset literal, because currently there is no way to have > ast.literal_eval evaluate something containing a frozenset. Because there > is no frozenset literal. > > > > I think Raymond Hettinger's proposal back in 2008: > > > > http://mail.python.org/pipermail/python-3000/2008-January/011798.html > > > > and the following thread is worth reading. Guido even pronounced his > agreement: > > > > http://mail.python.org/pipermail/python-3000/2008-January/011814.html > > > > but then changed his mind (as did Raymond). So the status quo remains. > > > > Unfortunately the proposal to use f{ ... } for frozen sets cannot work > within the constraints of Python's lexer: > > > > http://mail.python.org/pipermail/python-3000/2008-January/011838.html > > To clarify Guido's comment in that post, I'm fairly sure it *can* be made > to work, it just won' t be the same way that string prefixes work (because > the contents of dict and set displays are not opaque to the compiler the > way string contents are). > > The hypothetical "What if we want to allow expr{} as a general construct?" > objection needs to be balanced against the immediate value of a more > expressive language subset for use in ast.literal_eval(). I've just tried: ast.literal_eval() works just fine on a normal set: >>> ast.literal_eval("{1,2,3}") {1, 2, 3} So I'd like to know why people find it important to build a frozenset rather than a normal set. The situation of wanting to use sets as hash keys is very rare in my experience. Regards Antoine. From ncoghlan at gmail.com Sat Feb 2 15:56:57 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 3 Feb 2013 00:56:57 +1000 Subject: [Python-ideas] frozenset literals In-Reply-To: <20130202153537.1b5fca12@pitrou.net> References: <20130202153537.1b5fca12@pitrou.net> Message-ID: On 3 Feb 2013 00:39, "Antoine Pitrou" wrote: > > On Sun, 3 Feb 2013 00:28:55 +1000 > Nick Coghlan wrote: > > FWIW, I could personally tolerate the introduction of s{} and fs{} > > literals. We'd just declare the "s" prefix optional for non-empty sets to > > match the current rules. > > > > Encouraging the use of ast.literal_eval() over the security nightmare that > > is eval() would be more than enough justification for me. (As a syntax > > change, the idea would still need a PEP, though) > > If it is enough of a justification, then why not literal support for > more useful datatypes (decimal, datetime...) rather than frozenset? The difference is that decimal and datetime already have safe "from string" conversion operations. The empty set and frozen sets do not. That said, a decimal literal proposal would be a reasonable follow-up to the incorporation of cdecimal. Datetime is a more difficult prospect, since there are good reasons strptime is as flexible as it is. Regardless, I'm not saying a PEP to support all the builtin container types in ast.literal_eval would necessarily be accepted. I'm merely saying it is *worth writing*. Cheers, Nick. > > Regards > > Antoine. > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sat Feb 2 15:58:11 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 2 Feb 2013 15:58:11 +0100 Subject: [Python-ideas] frozenset literals References: <20130202153537.1b5fca12@pitrou.net> Message-ID: <20130202155811.0ba94896@pitrou.net> On Sun, 3 Feb 2013 00:56:57 +1000 Nick Coghlan wrote: > On 3 Feb 2013 00:39, "Antoine Pitrou" wrote: > > > > On Sun, 3 Feb 2013 00:28:55 +1000 > > Nick Coghlan wrote: > > > FWIW, I could personally tolerate the introduction of s{} and fs{} > > > literals. We'd just declare the "s" prefix optional for non-empty sets > to > > > match the current rules. > > > > > > Encouraging the use of ast.literal_eval() over the security nightmare > that > > > is eval() would be more than enough justification for me. (As a syntax > > > change, the idea would still need a PEP, though) > > > > If it is enough of a justification, then why not literal support for > > more useful datatypes (decimal, datetime...) rather than frozenset? > > The difference is that decimal and datetime already have safe "from string" > conversion operations. The empty set and frozen sets do not. You can call set() or frozenset() on a tuple literal, if you know you want to end up with a set of frozenset. Regards Antoine. From shane at umbrellacode.com Sat Feb 2 16:16:41 2013 From: shane at umbrellacode.com (Shane Green) Date: Sat, 02 Feb 2013 07:16:41 -0800 Subject: [Python-ideas] for line in input with open(path) as input... In-Reply-To: References: <510CFC94.5000502@umbrellacode.com> <510D0AA2.2060805@pearwood.info> Message-ID: <510D2DD9.7020109@umbrellacode.com> Thanks Nick. I definitely see your point about iterwith(); have been thinking about that since someone asked where __exit__() would be invoked. I meant the following as a more compact way of expressing for line in file with open(path) as file: process(line) As a more compact way of expressing with open(path) as file: for line in file: process(line) Not a change to the semantics of for-loops; a point my iterwith() function has confuses greatly, I realize now. I'm not seeing a loss of separation of concerns there. Indentation levels aren't limited, but flatter is better ;-) I saw a bunch of back and forth regarding iteration and context management in the PEP, but didn't notice anything along these lines in particular . I'll have to go back and take a closer look. Nick Coghlan wrote: > > The with statement block is needed to define *when* cleanup happens > (unconditionally at the end of the block). > > The "iterwith" generator is currently pointless, as it results in > nondeterministic cleanup of the context manager, so you may as well > not bother and just rely on the underlying iterable's nondeterministic > cleanup. > > We're never going to add cleanup semantics directly to for loops because: > - separation of concerns is a good design principle > - Indentation levels are not a limited resource (anyone that thinks > they are may be forgetting that factoring out context managers, > iterators and subfunctions gives you more of them, and that judicious > use of early returns and continue statements can avoid wasting them) > - we already considered it when initially designing the with statement > and decided it was a bad idea. > > I forget where that last part is written up. If it's not in PEP 343, > 342, 346 or 340 (the full set of PEPs that led to the current with > statement and contextlib.contextmanager designs), it should be in one > of the threads they reference. > > Cheers, > Nick. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ubershmekel at gmail.com Sat Feb 2 16:47:32 2013 From: ubershmekel at gmail.com (Yuval Greenfield) Date: Sat, 2 Feb 2013 17:47:32 +0200 Subject: [Python-ideas] for line in input with open(path) as input... In-Reply-To: <510D2DD9.7020109@umbrellacode.com> References: <510CFC94.5000502@umbrellacode.com> <510D0AA2.2060805@pearwood.info> <510D2DD9.7020109@umbrellacode.com> Message-ID: On Sat, Feb 2, 2013 at 5:16 PM, Shane Green wrote: > Thanks Nick. I definitely see your point about iterwith(); have been > thinking about that since someone asked where __exit__() would be invoked. > > I meant the following as a more compact way of expressing > > for line in file with open(path) as file: > process(line) > > This is an interesting idea, though a bit too dense for my taste. > Indentation levels aren't limited, but flatter is better ;-) > > I really like Golang's solution (defer) which Nick sort of emulates with ExitStack. http://docs.python.org/3/library/contextlib.html#contextlib.ExitStack If ExitStack ever became a language feature, we could write stuff like: def f(): fhand = local open(path) process(fhand) ghand = local open(path2) process(ghand) # which would be sort of equivalent to def g(): try: fhand = None ghand = None fhand = local open(path) process(fhand) ghand = local open(path2) process(ghand) finally: if fhand is not None: fhand.close() if ghand is not None: ghand.close() Yuval -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Sat Feb 2 19:30:30 2013 From: python at mrabarnett.plus.com (MRAB) Date: Sat, 02 Feb 2013 18:30:30 +0000 Subject: [Python-ideas] frozenset literals In-Reply-To: <20130202152856.0ab823b9@pitrou.net> References: <510D20B6.5040705@pearwood.info> <20130202152856.0ab823b9@pitrou.net> Message-ID: <510D5B46.8080409@mrabarnett.plus.com> On 2013-02-02 14:28, Antoine Pitrou wrote: > On Sun, 03 Feb 2013 01:20:38 +1100 > Steven D'Aprano wrote: >> >> Unfortunately the proposal to use f{ ... } for frozen sets cannot work within the constraints of Python's lexer: >> >> http://mail.python.org/pipermail/python-3000/2008-January/011838.html >> >> Unfortunately we're running out of useful, easy to enter symbols for literals. Until such time (Python4000 perhaps, or more likely Python5000) as we can use a rich set of Unicode literals, I don't think there is any clear way to have a frozenset literal. > > I'm not sure what the point is. frozensets are too rarely used to > deserve a literal notation. > Does it need a special notation as such? Wouldn't an alternative be to support "frozenset(...)" where "..." is parseable by ast.literal_eval()? The same thing could be done with some other built-in classes. From gotoalanlu at gmail.com Sat Feb 2 19:46:17 2013 From: gotoalanlu at gmail.com (Hua Lu) Date: Sat, 2 Feb 2013 12:46:17 -0600 Subject: [Python-ideas] frozenset literals In-Reply-To: <510D5B46.8080409@mrabarnett.plus.com> References: <510D20B6.5040705@pearwood.info> <20130202152856.0ab823b9@pitrou.net> <510D5B46.8080409@mrabarnett.plus.com> Message-ID: Perhaps, but we'd have to be careful with how we introduce those symbols: http://nedbatchelder.com/blog/201206/eval_really_is_dangerous.html You could possibly not allow attribute access but permit those symbols... is there an exploit possible with that much filtering? On Sat, Feb 2, 2013 at 12:30 PM, MRAB wrote: > On 2013-02-02 14:28, Antoine Pitrou wrote: >> >> On Sun, 03 Feb 2013 01:20:38 +1100 >> Steven D'Aprano wrote: >>> >>> >>> Unfortunately the proposal to use f{ ... } for frozen sets cannot work >>> within the constraints of Python's lexer: >>> >>> http://mail.python.org/pipermail/python-3000/2008-January/011838.html >>> >>> Unfortunately we're running out of useful, easy to enter symbols for >>> literals. Until such time (Python4000 perhaps, or more likely Python5000) as >>> we can use a rich set of Unicode literals, I don't think there is any clear >>> way to have a frozenset literal. >> >> >> I'm not sure what the point is. frozensets are too rarely used to >> deserve a literal notation. >> > Does it need a special notation as such? > > Wouldn't an alternative be to support "frozenset(...)" where "..." is > parseable by ast.literal_eval()? > > The same thing could be done with some other built-in classes. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas From python at mrabarnett.plus.com Sat Feb 2 20:07:09 2013 From: python at mrabarnett.plus.com (MRAB) Date: Sat, 02 Feb 2013 19:07:09 +0000 Subject: [Python-ideas] frozenset literals In-Reply-To: References: <510D20B6.5040705@pearwood.info> <20130202152856.0ab823b9@pitrou.net> <510D5B46.8080409@mrabarnett.plus.com> Message-ID: <510D63DD.6020405@mrabarnett.plus.com> On 2013-02-02 18:46, Hua Lu wrote: > On Sat, Feb 2, 2013 at 12:30 PM, MRAB wrote: >> On 2013-02-02 14:28, Antoine Pitrou wrote: >>> On Sun, 03 Feb 2013 01:20:38 +1100 >>> Steven D'Aprano wrote: >>>> >>>> Unfortunately the proposal to use f{ ... } for frozen sets cannot work >>>> within the constraints of Python's lexer: >>>> >>>> http://mail.python.org/pipermail/python-3000/2008-January/011838.html >>>> >>>> Unfortunately we're running out of useful, easy to enter symbols for >>>> literals. Until such time (Python4000 perhaps, or more likely Python5000) as >>>> we can use a rich set of Unicode literals, I don't think there is any clear >>>> way to have a frozenset literal. >>> >>> I'm not sure what the point is. frozensets are too rarely used to >>> deserve a literal notation. >>> >> Does it need a special notation as such? >> >> Wouldn't an alternative be to support "frozenset(...)" where "..." is >> parseable by ast.literal_eval()? >> >> The same thing could be done with some other built-in classes. >> > Perhaps, but we'd have to be careful with how we introduce those symbols: > > http://nedbatchelder.com/blog/201206/eval_really_is_dangerous.html > > You could possibly not allow attribute access but permit those > symbols... is there an exploit possible with that much filtering? > I'm talking about hard-coding "frozenset(...)" and a few other built-in classes to return the appropriate object, so, for example, "set([0, 1, 2])" would be the same as "{0, 1, 2}". From guido at python.org Sat Feb 2 20:25:46 2013 From: guido at python.org (Guido van Rossum) Date: Sat, 2 Feb 2013 11:25:46 -0800 Subject: [Python-ideas] frozenset literals In-Reply-To: <510D63DD.6020405@mrabarnett.plus.com> References: <510D20B6.5040705@pearwood.info> <20130202152856.0ab823b9@pitrou.net> <510D5B46.8080409@mrabarnett.plus.com> <510D63DD.6020405@mrabarnett.plus.com> Message-ID: On Sat, Feb 2, 2013 at 11:07 AM, MRAB wrote: > On 2013-02-02 18:46, Hua Lu wrote: >> >> On Sat, Feb 2, 2013 at 12:30 PM, MRAB wrote: >>> >>> On 2013-02-02 14:28, Antoine Pitrou wrote: >>>> >>>> On Sun, 03 Feb 2013 01:20:38 +1100 >>>> Steven D'Aprano wrote: >>>>> >>>>> >>>>> Unfortunately the proposal to use f{ ... } for frozen sets cannot work >>>>> within the constraints of Python's lexer: >>>>> >>>>> http://mail.python.org/pipermail/python-3000/2008-January/011838.html >>>>> >>>>> Unfortunately we're running out of useful, easy to enter symbols for >>>>> literals. Until such time (Python4000 perhaps, or more likely >>>>> Python5000) as >>>>> we can use a rich set of Unicode literals, I don't think there is any >>>>> clear >>>>> way to have a frozenset literal. >>>> >>>> >>>> I'm not sure what the point is. frozensets are too rarely used to >>>> deserve a literal notation. >>>> >>> Does it need a special notation as such? >>> >>> Wouldn't an alternative be to support "frozenset(...)" where "..." is >>> parseable by ast.literal_eval()? >>> >>> The same thing could be done with some other built-in classes. >>> >> Perhaps, but we'd have to be careful with how we introduce those symbols: >> >> http://nedbatchelder.com/blog/201206/eval_really_is_dangerous.html >> >> You could possibly not allow attribute access but permit those >> symbols... is there an exploit possible with that much filtering? >> > I'm talking about hard-coding "frozenset(...)" and a few other built-in > classes to return the appropriate object, so, for example, "set([0, 1, > 2])" would be the same as "{0, 1, 2}". You're proposing to do this just for literal_eval(), right? But how would you implement it? It seems it would require lots of special cases. Where would you stop? dict(key1=..., key2=...)? list({...})? -- --Guido van Rossum (python.org/~guido) From guido at python.org Sat Feb 2 20:30:46 2013 From: guido at python.org (Guido van Rossum) Date: Sat, 2 Feb 2013 11:30:46 -0800 Subject: [Python-ideas] for line in input with open(path) as input... In-Reply-To: References: <510CFC94.5000502@umbrellacode.com> <510D0AA2.2060805@pearwood.info> <510D2DD9.7020109@umbrellacode.com> Message-ID: None of these proposals have any merit. The last thing we need is more ways to spell the same thing that can already be spelled in several ways, all of which are just fine. Just because you save a line doesn't make your code more readable. -- --Guido van Rossum (python.org/~guido) From raymond.hettinger at gmail.com Sat Feb 2 20:54:45 2013 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Sat, 2 Feb 2013 11:54:45 -0800 Subject: [Python-ideas] frozenset literals In-Reply-To: <510D20B6.5040705@pearwood.info> References: <510D20B6.5040705@pearwood.info> Message-ID: On Feb 2, 2013, at 6:20 AM, Steven D'Aprano wrote: > I think Raymond Hettinger's proposal back in 2008: > > http://mail.python.org/pipermail/python-3000/2008-January/011798.html > > and the following thread is worth reading. Guido even pronounced his agreement: > > http://mail.python.org/pipermail/python-3000/2008-January/011814.html > > but then changed his mind (as did Raymond). So the status quo remains. > > Unfortunately the proposal to use f{ ... } for frozen sets cannot work within the constraints of Python's lexer: > > http://mail.python.org/pipermail/python-3000/2008-January/011838.html > > Unfortunately we're running out of useful, easy to enter symbols for literals. Until such time (Python4000 perhaps, or more likely Python5000) as we can use a rich set of Unicode literals, I don't think there is any clear way to have a frozenset l Wow, that was a nice recap of the historical discussions. Thank you :-) FWIW, if a workable frozenset literal were proposed, I would have no objections. Their original use case was be useable as elements of other sets (for sets of sets) or as dictionary keys (to implement graph structures). Later, it became clear that it would be useful for code-optimization to treat frozensets as constants that are built when a function is defined, rather than later when it is invoked. In the future, frozen sets may obtain other desirable properties. For example, I'm still evaluating whether to store the key/hash entries in a dense array and storing only indicies in a collision table. For regular sets, the dense array needs to over-allocate in order to grow efficiently. In contrast, a frozenset could free the excess memory and become very compact (about a third of the current size). If that ever comes to fruition, then a frozenset literal would be nice because of its compactness as compared to regular sets. Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben at bendarnell.com Sat Feb 2 21:10:37 2013 From: ben at bendarnell.com (Ben Darnell) Date: Sat, 2 Feb 2013 15:10:37 -0500 Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface Message-ID: The event loop interface in PEP 3156 has an extensibility problem. It seems likely that it will have a method like listen_udp() by the time it's done, but suppose that doesn't make it into the first official release. Third-party event loop implementations may want to provide UDP support as an extension, but the most consistent way to provide that extension is by adding new methods on the event loop object, where various extensions risk conflicting with each other or with new methods that become standardized later. The PEP specifies the add_reader family of methods in part so that core protocol implementations can be shared across all EventLoop implementations that support these methods. However, if those transports are implemented in a common base class (like Twisted's PosixReactorBase), there is no way for third-party transports to take advantage of a similar structure. I'd like to make it possible for transports to be developed independent of a particular EventLoop implementation in a way that is consistent with the way the core transports work. (This is a bigger concern for tulip than it is for twisted because twisted can update PosixReactorBase more frequently than the stdlib can change) I propose turning the interface around so that transport creation uses static functions that take the event loop as an argument, with a double-dispatch mechanism to allow the event loop to provide the actual implementation when it can: def create_connection(protocol_factory, host, port, event_loop=None): if event_loop is None: event_loop = get_event_loop() # Note the use of a fully-qualified name in the registry impl = event_loop.get_implementation('tulip.create_connection') return impl(protocol_factory, host, port) New third-party transports could provide fallbacks for event loops that don't have their own implementations: if impl is None: # These supports_*() functions are placeholders for a to-be-determined introspection interface. if supports_fd_interface(event_loop): return posix_udp_implementation(*args) elif supports_iocp_interface(event_loop): return iocp_udp_implementation(*args) else: raise Exception("This transport is not supported on this event loop") Or they could plug into the event loop's implementation registry: LibUVEventLoop.register_implementation('mymodule.listen_udp', libuv_udp_implementation) This does introduce a little magic (is there any precedent for this kind of multiple-dispatch in the standard library?), but I like the way it keeps the event loop interface from getting too big and monolithic. Third-party transports can avoid naming conflicts without looking fundamentally different from standard ones, and there's a clean path from doing something that's platform-specific (e.g. with add_reader and friends) to supporting multiple event loops to full standardization. -Ben -------------- next part -------------- An HTML attachment was scrubbed... URL: From markus at unterwaditzer.net Sun Feb 3 00:52:17 2013 From: markus at unterwaditzer.net (Markus Unterwaditzer) Date: Sun, 03 Feb 2013 00:52:17 +0100 Subject: [Python-ideas] frozenset literals In-Reply-To: References: Message-ID: <2bb9748e-9998-4800-9522-9c296de7623b@email.android.com> Hua Lu wrote: >This is similar to another proposal: >http://mail.python.org/pipermail/python-3000/2008-January/011798.html > >Anyway, I was using ast.literal_eval and attempted to use >frozenset({...}) >as a key in a dictionary, which failed, because frozenset isn't a >literal >(though putting frozenset in the environment would be a security risk). >I >am currently working around this with tuples, but I'd like a literal >for >representing frozensets as well. I also use frozensets elsewhere in the >code in ways similar to Raymond's original suggestion. > >Perhaps something like f{...} for declaring frozenset( comprehension)? >literals? > > >------------------------------------------------------------------------ > >_______________________________________________ >Python-ideas mailing list >Python-ideas at python.org >http://mail.python.org/mailman/listinfo/python-ideas What about {1, 2, 3}.frozen() ? While not exactly new syntax, i think it would be a good short way to create a frozenset. -- Markus From greg.ewing at canterbury.ac.nz Sun Feb 3 02:27:39 2013 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 03 Feb 2013 14:27:39 +1300 Subject: [Python-ideas] frozenset literals In-Reply-To: References: <510D20B6.5040705@pearwood.info> <20130202152856.0ab823b9@pitrou.net> <510D5B46.8080409@mrabarnett.plus.com> <510D63DD.6020405@mrabarnett.plus.com> Message-ID: <510DBD0B.7040406@canterbury.ac.nz> Guido van Rossum wrote: > You're proposing to do this just for literal_eval(), right? But how > would you implement it? It seems it would require lots of special > cases. Where would you stop? dict(key1=..., key2=...)? list({...})? I think you just do frozenset() and stop there until someone asks for something else. dict(key1=..., key2=...) and list({...}) are not needed, because they can already be spelled as literals. -- Greg From steve at pearwood.info Sun Feb 3 03:18:20 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 03 Feb 2013 13:18:20 +1100 Subject: [Python-ideas] frozenset literals In-Reply-To: <510D5B46.8080409@mrabarnett.plus.com> References: <510D20B6.5040705@pearwood.info> <20130202152856.0ab823b9@pitrou.net> <510D5B46.8080409@mrabarnett.plus.com> Message-ID: <510DC8EC.4010001@pearwood.info> On 03/02/13 05:30, MRAB wrote: > On 2013-02-02 14:28, Antoine Pitrou wrote: >> On Sun, 03 Feb 2013 01:20:38 +1100 >> Steven D'Aprano wrote: >>> >>> Unfortunately the proposal to use f{ ... } for frozen sets cannot work within the constraints of Python's lexer: >>> >>> http://mail.python.org/pipermail/python-3000/2008-January/011838.html >>> >>> Unfortunately we're running out of useful, easy to enter symbols for literals. Until such time (Python4000 perhaps, or more likely Python5000) as we can use a rich set of Unicode literals, I don't think there is any clear way to have a frozenset literal. >> >> I'm not sure what the point is. frozensets are too rarely used to >> deserve a literal notation. >> > Does it need a special notation as such? > > Wouldn't an alternative be to support "frozenset(...)" where "..." is > parseable by ast.literal_eval()? > > The same thing could be done with some other built-in classes. I think that having literal_eval support non-literals is a bad, bad idea. Let's just not go there. It will surely end in tears. However, I think that having something in between the strictness of literal_eval and the dangerous "anything goes" power of eval is a good idea. For a long time now I've toyed with an engine for building expression evaluators. Something that understands operators, function calls, etc, and you can tell it what names to accept. My main motivation is for evaluating mathematical expressions like: 5x^3 - 2x + log(1/y) + n!/?? I have some old, broken Pascal code that half does what I want, and I keep intending to revisit it some day in Python. In my copious spare time. In any case, I think something like that belongs in a third-party module, at least at first. -- Steven From rosuav at gmail.com Sun Feb 3 03:22:03 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 3 Feb 2013 13:22:03 +1100 Subject: [Python-ideas] frozenset literals In-Reply-To: <510DC8EC.4010001@pearwood.info> References: <510D20B6.5040705@pearwood.info> <20130202152856.0ab823b9@pitrou.net> <510D5B46.8080409@mrabarnett.plus.com> <510DC8EC.4010001@pearwood.info> Message-ID: On Sun, Feb 3, 2013 at 1:18 PM, Steven D'Aprano wrote: > I think that having literal_eval support non-literals is a bad, bad idea. > Let's just not go there. It will surely end in tears. > > However, I think that having something in between the strictness of > literal_eval and the dangerous "anything goes" power of eval is a good idea. > For a long time now I've toyed with an engine for building expression > evaluators. Something that understands operators, function calls, etc, and > you can tell it what names to accept. My main motivation is for evaluating > mathematical expressions like: > > 5x^3 - 2x + log(1/y) + n!/?? This sounds like a good idea, especially if there can be some way to enforce that these names may ONLY be called - you can't piggyback on log to get other functionality with log.__globals__ etc. That would cover frozenset quite happily. ChrisA From python at mrabarnett.plus.com Sun Feb 3 04:09:33 2013 From: python at mrabarnett.plus.com (MRAB) Date: Sun, 03 Feb 2013 03:09:33 +0000 Subject: [Python-ideas] frozenset literals In-Reply-To: <510DC8EC.4010001@pearwood.info> References: <510D20B6.5040705@pearwood.info> <20130202152856.0ab823b9@pitrou.net> <510D5B46.8080409@mrabarnett.plus.com> <510DC8EC.4010001@pearwood.info> Message-ID: <510DD4ED.9070300@mrabarnett.plus.com> On 2013-02-03 02:18, Steven D'Aprano wrote: > On 03/02/13 05:30, MRAB wrote: >> On 2013-02-02 14:28, Antoine Pitrou wrote: >>> On Sun, 03 Feb 2013 01:20:38 +1100 >>> Steven D'Aprano wrote: >>>> >>>> Unfortunately the proposal to use f{ ... } for frozen sets cannot work within the constraints of Python's lexer: >>>> >>>> http://mail.python.org/pipermail/python-3000/2008-January/011838.html >>>> >>>> Unfortunately we're running out of useful, easy to enter symbols for literals. Until such time (Python4000 perhaps, or more likely Python5000) as we can use a rich set of Unicode literals, I don't think there is any clear way to have a frozenset literal. >>> >>> I'm not sure what the point is. frozensets are too rarely used to >>> deserve a literal notation. >>> >> Does it need a special notation as such? >> >> Wouldn't an alternative be to support "frozenset(...)" where "..." is >> parseable by ast.literal_eval()? >> >> The same thing could be done with some other built-in classes. > > > I think that having literal_eval support non-literals is a bad, bad idea. Let's just not go there. It will surely end in tears. > It needn't go beyond a currently-supported literal passed as an argument to frozenset, etc, for a very limited number of built-in classes. A literal list or a literal set passed to frozenset would be treated as a literal frozenset. > However, I think that having something in between the strictness of literal_eval and the dangerous "anything goes" power of eval is a good idea. For a long time now I've toyed with an engine for building expression evaluators. Something that understands operators, function calls, etc, and you can tell it what names to accept. My main motivation is for evaluating mathematical expressions like: > > 5x^3 - 2x + log(1/y) + n!/?? > > I have some old, broken Pascal code that half does what I want, and I keep intending to revisit it some day in Python. In my copious spare time. > > In any case, I think something like that belongs in a third-party module, at least at first. > > > From ncoghlan at gmail.com Sun Feb 3 05:26:29 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 3 Feb 2013 14:26:29 +1000 Subject: [Python-ideas] for line in input with open(path) as input... In-Reply-To: References: <510CFC94.5000502@umbrellacode.com> <510D0AA2.2060805@pearwood.info> <510D2DD9.7020109@umbrellacode.com> Message-ID: On Sun, Feb 3, 2013 at 1:47 AM, Yuval Greenfield wrote: > I really like Golang's solution (defer) which Nick sort of emulates with > ExitStack. > > http://docs.python.org/3/library/contextlib.html#contextlib.ExitStack > > If ExitStack ever became a language feature Why would it ever become a language feature? It works just fine as a context manager, and the need for it isn't frequent enough to justify special syntax. > we could write stuff like: > > > def f(): > fhand = local open(path) > process(fhand) > ghand = local open(path2) > process(ghand) Why would you leave fhand open longer than necessary? The above would be better written as: def f(): with open(path) as fhand: process(fhand) with open(path2) as ghand: process(ghand) If you need both files open at the same time, you can use a nested context manager: def f(): with open(path) as fhand: with open(path2) as ghand: process(fhand, ghand) Or the nesting behaviour built into with statements themselves: def f(): with open(path) as fhand, open(path2) as ghand: process(fhand, ghand) It's only when the number of paths you need to open is dynamic that ExitStack comes into play (this is actually very close to the example in ExitStack's docstring, as handling a variable number of simultaneously open files was the use case that highlighted the fatal flaw in the way the old contextlib.nested design handled context managers that acquired the resource in __init__ rather than __enter__): def f(*paths): with contextlib.ExitStack() as stack: files = [stack.enter_context(open(path)) for path in paths] process(files) Function and class definitions control name scope (amongst other things), with statements control deterministic cleanup, loops control iteration. That's what I mean by "separation of concerns" in relation to these aspects of the language design and it's a *good* thing (and one of the key reasons with statements behave like PEP 343, rather than being closer to Guido's original looping idea that is described in PEP 340). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sun Feb 3 05:36:53 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 3 Feb 2013 14:36:53 +1000 Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface In-Reply-To: References: Message-ID: On Sun, Feb 3, 2013 at 6:10 AM, Ben Darnell wrote: > The event loop interface in PEP 3156 has an extensibility problem. It seems > likely that it will have a method like listen_udp() by the time it's done, > but suppose that doesn't make it into the first official release. > Third-party event loop implementations may want to provide UDP support as an > extension, but the most consistent way to provide that extension is by > adding new methods on the event loop object, where various extensions risk > conflicting with each other or with new methods that become standardized > later. The general idea of using factory functions for transport creation, even for the "core transports" (like sockets, SSL and pipes) sounds good to me. I've expressed some concerns previously about the breadth of the event loop class API, and this would go a long way towards alleviating them. As far as precedents for the proposed dispatch/registration mechanism goes, probably the closest we currently have is the codecs registry. A "transport registry" (using Python dottedname identifiers as the naming scheme) sounds promising to me. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sun Feb 3 05:50:45 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 3 Feb 2013 14:50:45 +1000 Subject: [Python-ideas] frozenset literals In-Reply-To: <510DD4ED.9070300@mrabarnett.plus.com> References: <510D20B6.5040705@pearwood.info> <20130202152856.0ab823b9@pitrou.net> <510D5B46.8080409@mrabarnett.plus.com> <510DC8EC.4010001@pearwood.info> <510DD4ED.9070300@mrabarnett.plus.com> Message-ID: On Sun, Feb 3, 2013 at 1:09 PM, MRAB wrote: > On 2013-02-03 02:18, Steven D'Aprano wrote: >> >> On 03/02/13 05:30, MRAB wrote: >>> >>> On 2013-02-02 14:28, Antoine Pitrou wrote: >>>> >>>> On Sun, 03 Feb 2013 01:20:38 +1100 >>>> Steven D'Aprano wrote: >>>>> >>>>> >>>>> Unfortunately the proposal to use f{ ... } for frozen sets cannot work >>>>> within the constraints of Python's lexer: >>>>> >>>>> http://mail.python.org/pipermail/python-3000/2008-January/011838.html >>>>> >>>>> Unfortunately we're running out of useful, easy to enter symbols for >>>>> literals. Until such time (Python4000 perhaps, or more likely Python5000) as >>>>> we can use a rich set of Unicode literals, I don't think there is any clear >>>>> way to have a frozenset literal. >>>> >>>> >>>> I'm not sure what the point is. frozensets are too rarely used to >>>> deserve a literal notation. >>>> >>> Does it need a special notation as such? >>> >>> Wouldn't an alternative be to support "frozenset(...)" where "..." is >>> parseable by ast.literal_eval()? >>> >>> The same thing could be done with some other built-in classes. >> >> >> >> I think that having literal_eval support non-literals is a bad, bad idea. >> Let's just not go there. It will surely end in tears. >> > It needn't go beyond a currently-supported literal passed as an > argument to frozenset, etc, for a very limited number of built-in > classes. A literal list or a literal set passed to frozenset would be > treated as a literal frozenset. This is why we need a PEP or a PyPI module. It's certainly not clear to me that special casing in ast.literal_eval (or a new "ast.limited_eval") is a superior solution to s{} and fs{} syntax for creating the empty set and frozen sets. (And, as Raymond notes, there are other compile-time benefits in terms of constant-caching when it comes to dedicated syntax) On the other hand, a "limited_eval" style solution might be easier to extend to other builtins like range, reversed and enum, as well as to container comprehensions and generator expressions. It also has the virtue of being possible to write as a PyPI module, and made available for *current* Python versions, rather than only being available in Python 3.4+. It's certainly a space worth exploring, even though the best way to improve the status quo isn't immediately obvious. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From shane at umbrellacode.com Sun Feb 3 07:17:56 2013 From: shane at umbrellacode.com (Shane Green) Date: Sat, 02 Feb 2013 22:17:56 -0800 Subject: [Python-ideas] for line in input with open(path) as input... In-Reply-To: References: <510CFC94.5000502@umbrellacode.com> <510D0AA2.2060805@pearwood.info> <510D2DD9.7020109@umbrellacode.com> Message-ID: <510E0114.1070809@umbrellacode.com> Okay, Nick, thanks for iterating exactly what separation of concerns you meant: it clearly identifies where my thinking was going a bit awry. My thinking was more along the lines of, "with" controls deterministic cleanup around a block of code, can't that block be an existing for loop? The "with" statement still controls deterministic cleanup; its __enter__() still necessarily precedes evaluation of the for loop, and its __exit__() still immediately follows evaluation of the for loop. But, there's not much to gain from the idea, at best, so it's a bit of a waste of time, I'm afraid... 02/02/2013 22:13:02 > Nick Coghlan > February 2, 2013 8:26 PM > On Sun, Feb 3, 2013 at 1:47 AM, Yuval Greenfield wrote: >> I really like Golang's solution (defer) which Nick sort of emulates with >> ExitStack. >> >> http://docs.python.org/3/library/contextlib.html#contextlib.ExitStack >> >> If ExitStack ever became a language feature > > Why would it ever become a language feature? It works just fine as a > context manager, and the need for it isn't frequent enough to justify > special syntax. > >> we could write stuff like: >> >> >> def f(): >> fhand = local open(path) >> process(fhand) >> ghand = local open(path2) >> process(ghand) > > Why would you leave fhand open longer than necessary? The above would > be better written as: > > def f(): > with open(path) as fhand: > process(fhand) > with open(path2) as ghand: > process(ghand) > > If you need both files open at the same time, you can use a nested > context manager: > > def f(): > with open(path) as fhand: > with open(path2) as ghand: > process(fhand, ghand) > > Or the nesting behaviour built into with statements themselves: > > def f(): > with open(path) as fhand, open(path2) as ghand: > process(fhand, ghand) > > It's only when the number of paths you need to open is dynamic that > ExitStack comes into play (this is actually very close to the example > in ExitStack's docstring, as handling a variable number of > simultaneously open files was the use case that highlighted the fatal > flaw in the way the old contextlib.nested design handled context > managers that acquired the resource in __init__ rather than > __enter__): > > def f(*paths): > with contextlib.ExitStack() as stack: > files = [stack.enter_context(open(path)) for path in paths] > process(files) > > Function and class definitions control name scope (amongst other > things), with statements control deterministic cleanup, loops control > iteration. That's what I mean by "separation of concerns" in relation > to these aspects of the language design and it's a *good* thing (and > one of the key reasons with statements behave like PEP 343, rather > than being closer to Guido's original looping idea that is described > in PEP 340). > > Cheers, > Nick. > > Yuval Greenfield > February 2, 2013 7:47 AM > On Sat, Feb 2, 2013 at 5:16 PM, Shane Green > wrote: > > Thanks Nick. I definitely see your point about iterwith(); have > been thinking about that since someone asked where __exit__() > would be invoked. > > I meant the following as a more compact way of expressing > > for line in file with open(path) as file: > process(line) > > > > This is an interesting idea, though a bit too dense for my taste. > > > Indentation levels aren't limited, but flatter is better ;-) > > > I really like Golang's solution (defer) which Nick sort of emulates > with ExitStack. > > http://docs.python.org/3/library/contextlib.html#contextlib.ExitStack > > If ExitStack ever became a language feature, we could write stuff like: > > > def f(): > fhand = local open(path) > process(fhand) > ghand = local open(path2) > process(ghand) > > # which would be sort of equivalent to > > def g(): > try: > fhand = None > ghand = None > fhand = local open(path) > process(fhand) > ghand = local open(path2) > process(ghand) > finally: > if fhand is not None: > fhand.close() > if ghand is not None: > ghand.close() > > > > Yuval > Shane Green > February 2, 2013 7:16 AM > Thanks Nick. I definitely see your point about iterwith(); have been > thinking about that since someone asked where __exit__() would be > invoked. > > I meant the following as a more compact way of expressing > > for line in file with open(path) as file: > process(line) > > As a more compact way of expressing > > with open(path) as file: > for line in file: > process(line) > > Not a change to the semantics of for-loops; a point my iterwith() > function has confuses greatly, I realize now. I'm not seeing a loss of > separation of concerns there. > > Indentation levels aren't limited, but flatter is better ;-) > > I saw a bunch of back and forth regarding iteration and context > management in the PEP, but didn't notice anything along these lines in > particular . I'll have to go back and take a closer look. > > > > Nick Coghlan > February 2, 2013 6:18 AM > > The with statement block is needed to define *when* cleanup happens > (unconditionally at the end of the block). > > The "iterwith" generator is currently pointless, as it results in > nondeterministic cleanup of the context manager, so you may as well > not bother and just rely on the underlying iterable's nondeterministic > cleanup. > > We're never going to add cleanup semantics directly to for loops because: > - separation of concerns is a good design principle > - Indentation levels are not a limited resource (anyone that thinks > they are may be forgetting that factoring out context managers, > iterators and subfunctions gives you more of them, and that judicious > use of early returns and continue statements can avoid wasting them) > - we already considered it when initially designing the with statement > and decided it was a bad idea. > > I forget where that last part is written up. If it's not in PEP 343, > 342, 346 or 340 (the full set of PEPs that led to the current with > statement and contextlib.contextmanager designs), it should be in one > of the threads they reference. > > Cheers, > Nick. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > Chris Angelico > February 2, 2013 5:52 AM > > According to the OP's posted code, as soon as the iterable runs out. > Not sure what happens if you don't exhaust it but I'm sure generator > functions have already solved that, too. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: postbox-contact.jpg Type: image/jpeg Size: 1215 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: postbox-contact.jpg Type: image/jpeg Size: 1203 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: compose-unknown-contact.jpg Type: image/jpeg Size: 770 bytes Desc: not available URL: From ubershmekel at gmail.com Sun Feb 3 09:38:33 2013 From: ubershmekel at gmail.com (Yuval Greenfield) Date: Sun, 3 Feb 2013 10:38:33 +0200 Subject: [Python-ideas] for line in input with open(path) as input... In-Reply-To: References: <510CFC94.5000502@umbrellacode.com> <510D0AA2.2060805@pearwood.info> <510D2DD9.7020109@umbrellacode.com> Message-ID: On Sun, Feb 3, 2013 at 6:26 AM, Nick Coghlan wrote: > If you need both files open at the same time, you can use a nested > context manager: > > def f(): > with open(path) as fhand: > with open(path2) as ghand: > process(fhand, ghand) > > Or the nesting behaviour built into with statements themselves: > > def f(): > with open(path) as fhand, open(path2) as ghand: > process(fhand, ghand) > > This is indeed what I was looking for. The "with" statement does give a lot of control, though I do dislike the double indentation and also dislike opening 2 files on one line. Sucks to be me. > Function and class definitions control name scope (amongst other > things), with statements control deterministic cleanup, loops control > iteration. That's what I mean by "separation of concerns" in relation > to these aspects of the language design and it's a *good* thing (and > one of the key reasons with statements behave like PEP 343, rather > than being closer to Guido's original looping idea that is described > in PEP 340). > > Yes, separation of concerns is indeed a good thing and as Guido mentioned, there are already too many ways to do this. Thanks for the enlightenment, Yuval -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sun Feb 3 10:04:55 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 3 Feb 2013 10:04:55 +0100 Subject: [Python-ideas] frozenset literals References: <510D20B6.5040705@pearwood.info> <20130202152856.0ab823b9@pitrou.net> <510D5B46.8080409@mrabarnett.plus.com> <510DC8EC.4010001@pearwood.info> <510DD4ED.9070300@mrabarnett.plus.com> Message-ID: <20130203100455.5657a9d7@pitrou.net> On Sun, 3 Feb 2013 14:50:45 +1000 Nick Coghlan wrote: > > This is why we need a PEP or a PyPI module. It's certainly not clear > to me that special casing in ast.literal_eval (or a new > "ast.limited_eval") is a superior solution to s{} and fs{} syntax for > creating the empty set and frozen sets. Well, it's superior because it doesn't need new syntax. > (And, as Raymond notes, there > are other compile-time benefits in terms of constant-caching when it > comes to dedicated syntax) Which is a rather dubious benefit when it comes to a little-used datatype. If you want to avoid the cost of instantiating a frozenset in every loop iteration, just hoist it manually out of the loop. > On the other hand, a "limited_eval" style solution might be easier to > extend to other builtins like range, reversed and enum, as well as to > container comprehensions and generator expressions. It also has the > virtue of being possible to write as a PyPI module, and made available > for *current* Python versions, rather than only being available in > Python 3.4+. Agreed. Regards Antoine. From yorik.sar at gmail.com Sun Feb 3 10:23:03 2013 From: yorik.sar at gmail.com (Yuriy Taraday) Date: Sun, 3 Feb 2013 13:23:03 +0400 Subject: [Python-ideas] frozenset literals In-Reply-To: References: Message-ID: On Sat, Feb 2, 2013 at 11:38 AM, Hua Lu wrote: > >>> import ast > >>> ast.literal_eval("{ 'key': 'val' }") > {'key': 'val'} > >>> ast.literal_eval("{ ('key',): 'val' }") > {('key',): 'val'} > >>> ast.literal_eval("{ frozenset({'key'}): 'val' }") > Traceback (most recent call last): > File "", line 1, in > File "/usr/lib/python3.3/ast.py", line 86, in literal_eval > return _convert(node_or_string) > File "/usr/lib/python3.3/ast.py", line 63, in _convert > in zip(node.keys, node.values)) > File "/usr/lib/python3.3/ast.py", line 62, in > return dict((_convert(k), _convert(v)) for k, v > File "/usr/lib/python3.3/ast.py", line 85, in _convert > raise ValueError('malformed node or string: ' + repr(node)) > ValueError: malformed node or string: <_ast.Call object at 0x7f865a8c1450> > It looks like we can special-case usage of set literal as a key for a dict or a member for another set and create a frozenset constant instead. So that adict[{1, 2, 3}] should be interpreted as adict[frozenset([1, 2, 3])] and { {'b', 'a', 'r'}, 'foo' } as { frozenset('bar'), 'foo' } This will provide a minimal change to the interpreter while making it possible to use any literal-parsing with frozensets while keeping method calls out of literal_eval. -- Kind regards, Yuriy. -------------- next part -------------- An HTML attachment was scrubbed... URL: From eliben at gmail.com Sun Feb 3 15:54:01 2013 From: eliben at gmail.com (Eli Bendersky) Date: Sun, 3 Feb 2013 06:54:01 -0800 Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface In-Reply-To: References: Message-ID: On Sat, Feb 2, 2013 at 8:36 PM, Nick Coghlan wrote: > On Sun, Feb 3, 2013 at 6:10 AM, Ben Darnell wrote: > > The event loop interface in PEP 3156 has an extensibility problem. It > seems > > likely that it will have a method like listen_udp() by the time it's > done, > > but suppose that doesn't make it into the first official release. > > Third-party event loop implementations may want to provide UDP support > as an > > extension, but the most consistent way to provide that extension is by > > adding new methods on the event loop object, where various extensions > risk > > conflicting with each other or with new methods that become standardized > > later. > > The general idea of using factory functions for transport creation, > even for the "core transports" (like sockets, SSL and pipes) sounds > good to me. I've expressed some concerns previously about the breadth > of the event loop class API, and this would go a long way towards > alleviating them. > > +1 Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sun Feb 3 18:30:45 2013 From: guido at python.org (Guido van Rossum) Date: Sun, 3 Feb 2013 09:30:45 -0800 Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface In-Reply-To: References: Message-ID: Ben: I wrote a long reply, it is inline below. However it's possible that I am not seeing the use case right. Your proposal is written in very general terms; perhaps you can come up with a more specific example to defend it further? The UDP example somehow doesn't seem very compelling to me. On Sat, Feb 2, 2013 at 12:10 PM, Ben Darnell wrote: > The event loop interface in PEP 3156 has an extensibility problem. It seems > likely that it will have a method like listen_udp() by the time it's done, > but suppose that doesn't make it into the first official release. > Third-party event loop implementations may want to provide UDP support as an > extension, but the most consistent way to provide that extension is by > adding new methods on the event loop object, where various extensions risk > conflicting with each other or with new methods that become standardized > later. This may be based on a misunderstanding. If a specific event loop implementation wants to offer a new transport, there is no reason to add it as a method to the event loop. The app that wants to use that transport has to import that event loop too, so the app might as well call a module-level function that's specific to that event loop, in order to instantiate the new transport. We may have to point this out in the PEP, since it is likely that implementers wanting to offer new features will think of adding new methods to their event loop first. But that's a precious namespace (since it's shared by all event loop implementations), whereas their own implementation's module namespace is less precious. > The PEP specifies the add_reader family of methods in part so that core > protocol implementations can be shared across all EventLoop implementations > that support these methods. However, if those transports are implemented in > a common base class (like Twisted's PosixReactorBase), there is no way for > third-party transports to take advantage of a similar structure. I'm not sure I understand this (the "there is no way" part). Is the problem that that base class is private, or that the add_reader methods may not be there, or what? > I'd like > to make it possible for transports to be developed independent of a > particular EventLoop implementation in a way that is consistent with the way > the core transports work. Hm, now we're talking about something else. Transports implemented *independently* from an event loop implementation should not assume more than the standardized API. This feels rather limiting unless we're talking about transports built on top of other protocols (e.g. the mythical TCP-over-HTTP transport :-). My expectation is that most transport implementations (as opposed to protocols) are probably tied closely to an event loop implementation (and note that the various Tulip event loop implementations using select, poll, epoll, kqueue, are a single implementation for this purpose -- OTOH the IocpEventLoop (in the iocp branch) is a different implementation and, indeed, has different transport implementations! > (This is a bigger concern for tulip than it is > for twisted because twisted can update PosixReactorBase more frequently than > the stdlib can change) I think what you're really getting at here is that there is no 3rd significant party cottage industry creating new transports. Transports in practice are all part of the Twisted distribution, so development is only gated by backwards compatibility requirements with user apps (APIs, once offered, must remain supported and stable), not by compatibilities with older versions of the rest of the framework (a new transport introduced in Twisted 12.1 doesn't have to work with Twisted 12.0). > I propose turning the interface around so that transport creation uses > static functions that take the event loop as an argument, with a > double-dispatch mechanism to allow the event loop to provide the actual > implementation when it can: > > def create_connection(protocol_factory, host, port, event_loop=None): > if event_loop is None: > event_loop = get_event_loop() > # Note the use of a fully-qualified name in the registry > impl = event_loop.get_implementation('tulip.create_connection') > return impl(protocol_factory, host, port) Hm. I don't see what this adds. It still always gets the protocol from the event loop so moving this standardized method out of the event loop class doesn't seem to buy anything. The implementation-dependent work is just moved into get_implementation(). I also don't see why we need a registry. > New third-party transports could provide fallbacks for event loops that > don't have their own implementations: > > if impl is None: > # These supports_*() functions are placeholders for a to-be-determined > # introspection interface. > if supports_fd_interface(event_loop): > return posix_udp_implementation(*args) > elif supports_iocp_interface(event_loop): > return iocp_udp_implementation(*args) > else: > raise Exception("This transport is not supported on this event loop") It seems you want each transport implementation to provide its own create_connection() function, right? There's nothing wrong with that, and I don't see that just because 3rd party transports will be instantiated through a module-level function (in a specific module) that means that the standard transports specified by the PEP (standard in semantics, not in implementation!) can't be instantiated through methods on the event loop. Perhaps you are placing a higher value on consistency between standard and non-standard transports? To me, it is actually positive to be inconsistent here, so readers are made fully aware that a non-standard transport is being used. (The idea of introspection functions is fine, however.) > Or they could plug into the event loop's implementation registry: > > LibUVEventLoop.register_implementation('mymodule.listen_udp', > libuv_udp_implementation) Yeah, but wouldn't this likely be a private affair between UV's event loop and UV's UDP transport, which are being distributed together? Users who wish to use UDP (assuming PEP 3156 ends up not specifying UDP support) are required to depend on a non-standard feature of their event loop, you can't hide that with a registry. (Note that a UDP transport has a different API than a TCP transport, and requires the protocol to implement different methods as well.) > This does introduce a little magic (is there any precedent for this kind of > multiple-dispatch in the standard library?), but I like the way it keeps the > event loop interface from getting too big and monolithic. Third-party > transports can avoid naming conflicts without looking fundamentally > different from standard ones, and there's a clean path from doing something > that's platform-specific (e.g. with add_reader and friends) to supporting > multiple event loops to full standardization. I actually consider it a good thing that when a concept is standardized, the "name" of the API changes. When we adopt a 3rd party module in the stdlib we typically give it a new name too, to avoid confusion about which version is meant (since inevitably the 3rd party has more variability than the version adopted into the stdlib). With all that said, if a particular event loop implementation prefers to add non-standard methods to their event loop object, and then starts lobbying for its adoption in the standard, I can't stop them, and they may even have a good shot at getting adopted in the next Python version. But they should be aware of the risk they run, that the next version of the stdlib might expose a different API under their chose name. It will be easier for their users to transition if they choose a way to spell their extension that is *not* likely to be standardized, e.g. a function in their own module, or an event loop method name with a custom prefix like uv_listen_udp(). I'm looking forward to explanations of why I am preventing the developing of 3rd party transports with this response.... -- --Guido van Rossum (python.org/~guido) From ben at bendarnell.com Sun Feb 3 20:55:05 2013 From: ben at bendarnell.com (Ben Darnell) Date: Sun, 3 Feb 2013 14:55:05 -0500 Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface In-Reply-To: References: Message-ID: On Sun, Feb 3, 2013 at 12:30 PM, Guido van Rossum wrote: > Ben: I wrote a long reply, it is inline below. However it's possible > that I am not seeing the use case right. Your proposal is written in > very general terms; perhaps you can come up with a more specific > example to defend it further? The UDP example somehow doesn't seem > very compelling to me. > UDP is a real-life example from tornado - we don't have any built-in support for UDP, but people who need it have been able to build it without touching tornado itself. The same argument would apply to pipes or any number of other (admittedly much more esoteric) network protocols. I'll elaborate on the UDP example below. > > On Sat, Feb 2, 2013 at 12:10 PM, Ben Darnell wrote: > > The event loop interface in PEP 3156 has an extensibility problem. It > seems > > likely that it will have a method like listen_udp() by the time it's > done, > > but suppose that doesn't make it into the first official release. > > Third-party event loop implementations may want to provide UDP support > as an > > extension, but the most consistent way to provide that extension is by > > adding new methods on the event loop object, where various extensions > risk > > conflicting with each other or with new methods that become standardized > > later. > > This may be based on a misunderstanding. If a specific event loop > implementation wants to offer a new transport, there is no reason to > add it as a method to the event loop. The app that wants to use that > transport has to import that event loop too, so the app might as well > call a module-level function that's specific to that event loop, in > order to instantiate the new transport. > > We may have to point this out in the PEP, since it is likely that > implementers wanting to offer new features will think of adding new > methods to their event loop first. But that's a precious namespace > (since it's shared by all event loop implementations), whereas their > own implementation's module namespace is less precious. > Right. Third-party extensions to the event loop interface are inherently problematic, so we'll have to provide them in some other way. I'm proposing a pattern for that "some other way" and then realizing that I like it even for first-party interfaces. > > > The PEP specifies the add_reader family of methods in part so that core > > protocol implementations can be shared across all EventLoop > implementations > > that support these methods. However, if those transports are > implemented in > > a common base class (like Twisted's PosixReactorBase), there is no way > for > > third-party transports to take advantage of a similar structure. > > I'm not sure I understand this (the "there is no way" part). Is the > problem that that base class is private, or that the add_reader > methods may not be there, or what? > Suppose twisted did not have UDP support built in. Most reactor implementations subclass PosixReactorBase (with IOCPReactor as the notable exception). Twisted can add UDP support and implement listenUDP in PosixReactorBase and IOCPReactor, and suddenly most reactors (even third-party ones like TornadoReactor) support UDP for free. Those that don't (a hypothetical LibUVReactor?) can implement it themselves and interoperate with everything else. If a third party wanted to add UDP support separately from twisted's release schedule, they can't do with an interface that is generically usable across all reactors. They could make a static function listenUDP() that works with any IReactorFDSet, and maybe special-case IOCPReactor, but then there'd be no way for a third-party LibUVReactor to participate. > > > I'd like > > to make it possible for transports to be developed independent of a > > particular EventLoop implementation in a way that is consistent with the > way > > the core transports work. > > Hm, now we're talking about something else. Transports implemented > *independently* from an event loop implementation should not assume > more than the standardized API. This feels rather limiting unless > we're talking about transports built on top of other protocols (e.g. > the mythical TCP-over-HTTP transport :-). > My expectation is that most transport implementations (as opposed to > protocols) are probably tied closely to an event loop implementation > (and note that the various Tulip event loop implementations using > select, poll, epoll, kqueue, are a single implementation for this > purpose -- OTOH the IocpEventLoop (in the iocp branch) is a different > implementation and, indeed, has different transport implementations! > add_reader is not very limiting except for its platform-specificity. It's possible to have a generic protocol across all posixy event loops and then special-case the small number of interesting non-posixy ones (or maybe there is some other class of methods that could be standardized for other platforms? Is there some set of methods analogous to add_reader that multiple IOCP-based loops could share?) > > > (This is a bigger concern for tulip than it is > > for twisted because twisted can update PosixReactorBase more frequently > than > > the stdlib can change) > > I think what you're really getting at here is that there is no 3rd > significant party cottage industry creating new transports. Transports > in practice are all part of the Twisted distribution, so development > is only gated by backwards compatibility requirements with user apps > (APIs, once offered, must remain supported and stable), not by > compatibilities with older versions of the rest of the framework (a > new transport introduced in Twisted 12.1 doesn't have to work with > Twisted 12.0). > I was thinking more about release schedules. Twisted has several releases a year, but the standard library moves much more slowly. > > > I propose turning the interface around so that transport creation uses > > static functions that take the event loop as an argument, with a > > double-dispatch mechanism to allow the event loop to provide the actual > > implementation when it can: > > > > def create_connection(protocol_factory, host, port, event_loop=None): > > if event_loop is None: > > event_loop = get_event_loop() > > # Note the use of a fully-qualified name in the registry > > impl = event_loop.get_implementation('tulip.create_connection') > > return impl(protocol_factory, host, port) > > Hm. I don't see what this adds. It still always gets the protocol from > the event loop so moving this standardized method out of the event > loop class doesn't seem to buy anything. The implementation-dependent > work is just moved into get_implementation(). I also don't see why we > need a registry. > This version doesn't change much, it's mainly to set the stage for the following variations. However, it does have a few nice properties - it keeps the (public) event loop interface small and manageable, and callers don't need to touch actual event loop objects unless they want to have more than one. From a stylistic perspective I like this style of interface more than using dozens of methods on the event loop object itself (even if those dozens of methods are still there but hidden as an implementation detail). > > > New third-party transports could provide fallbacks for event loops that > > don't have their own implementations: > > > > if impl is None: > > # These supports_*() functions are placeholders for a > to-be-determined > > # introspection interface. > > if supports_fd_interface(event_loop): > > return posix_udp_implementation(*args) > > elif supports_iocp_interface(event_loop): > > return iocp_udp_implementation(*args) > > else: > > raise Exception("This transport is not supported on this event > loop") > > It seems you want each transport implementation to provide its own > create_connection() function, right? There's nothing wrong with that, > and I don't see that just because 3rd party transports will be > instantiated through a module-level function (in a specific module) > that means that the standard transports specified by the PEP (standard > in semantics, not in implementation!) can't be instantiated through > methods on the event loop. > > Perhaps you are placing a higher value on consistency between standard > and non-standard transports? To me, it is actually positive to be > inconsistent here, so readers are made fully aware that a non-standard > transport is being used. > When third-party modules get absorbed into the standard library, it's often possible to support both just by trying different imports until one works (unittest.mock vs mock, json vs simplejson, etc). Sometimes a module's interface gets cleaned up and rearranged in the process, but that seems to be less common. It would be nice if a third-party transport could get standardized and the only thing callers would need to change is their imports. However, this is a minor concern; as I wrote up this design I realized I liked it for first-party work even if there were no third-party modules to be consistent with. > > (The idea of introspection functions is fine, however.) > > > Or they could plug into the event loop's implementation registry: > > > > LibUVEventLoop.register_implementation('mymodule.listen_udp', > > libuv_udp_implementation) > > Yeah, but wouldn't this likely be a private affair between UV's event > loop and UV's UDP transport, which are being distributed together? > But libuv doesn't necessarily contain the transport creation function. The idea is that someone can propose a transport interface in a third-party module (mymodule.listen_udp in this example), implement it themselves for some event loop implementations, and other event loops can declare themselves compatible with it. (And in an admittedly far-fetched scenario, if there were two third-party UDP interfaces and LibUVEventLoop implemented one of them, yet another party could build a bridge between the two, and then they'd plug it in with register_implementation) > Users who wish to use UDP (assuming PEP 3156 ends up not specifying > UDP support) are required to depend on a non-standard feature of their > event loop, you can't hide that with a registry. (Note that a UDP > transport has a different API than a TCP transport, and requires the > protocol to implement different methods as well.) > Yes, third-party transports will be non-standard, but in practice the add_reader family makes it easy to get broad coverage in a quasi-standard way, and the registry makes it possible to fill in the gaps. > > > This does introduce a little magic (is there any precedent for this kind > of > > multiple-dispatch in the standard library?), but I like the way it keeps > the > > event loop interface from getting too big and monolithic. Third-party > > transports can avoid naming conflicts without looking fundamentally > > different from standard ones, and there's a clean path from doing > something > > that's platform-specific (e.g. with add_reader and friends) to supporting > > multiple event loops to full standardization. > > I actually consider it a good thing that when a concept is > standardized, the "name" of the API changes. When we adopt a 3rd party > module in the stdlib we typically give it a new name too, to avoid > confusion about which version is meant (since inevitably the 3rd party > has more variability than the version adopted into the stdlib). > The *module* gets a new name (and that is indeed a good thing), but the functions and classes within (usually) stay the same. > > With all that said, if a particular event loop implementation prefers > to add non-standard methods to their event loop object, and then > starts lobbying for its adoption in the standard, I can't stop them, > and they may even have a good shot at getting adopted in the next > Python version. But they should be aware of the risk they run, that > the next version of the stdlib might expose a different API under > their chose name. It will be easier for their users to transition if > they choose a way to spell their extension that is *not* likely to be > standardized, e.g. a function in their own module, or an event loop > method name with a custom prefix like uv_listen_udp(). > Agreed. > > I'm looking forward to explanations of why I am preventing the > developing of 3rd party transports with this response.... > I don't think the status quo prevents the development of third-party transports, but it does implicitly encourage two bad habits: A) adding methods to the event loop, inviting name collisions, or B) just building on add_reader and friends without thinking about non-posix platforms. Of course, no one expects a groundswell of third-party development at the event loop and transport level, so this could just be so much overengineering, but I like it from a stylistic perspective even without the third-party benefits. -Ben > > -- > --Guido van Rossum (python.org/~guido) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gotoalanlu at gmail.com Sun Feb 3 21:45:14 2013 From: gotoalanlu at gmail.com (Hua Lu) Date: Sun, 3 Feb 2013 14:45:14 -0600 Subject: [Python-ideas] frozenset literals In-Reply-To: References: Message-ID: > It looks like we can special-case usage of set literal as a key for a dict > or a member for another set and create a frozenset constant instead. > > So that adict[{1, 2, 3}] should be interpreted as adict[frozenset([1, 2, > 3])] and { {'b', 'a', 'r'}, 'foo' } as { frozenset('bar'), 'foo' } This could interfere with the behavior of types overriding the index operator. Also, it's a special case. > This will provide a minimal change to the interpreter while making it > possible to use any literal-parsing with frozensets while keeping method > calls out of literal_eval. Why shouldn't we change the interpreter when it could make the language better? Honestly, I can't think of an element which will require special syntax like symbol{...}. Furthermore, it could be really confusable with current syntax. Example: for fn { bar(foo) for foo in baz }: What if bar was a higher order function and that syntax meant something like "make a generator applying each thing in the set to an original fn before introducing fn in the scope" when you really meant `for fn in {...}`? Python 3 has already made some calls more explicitly a function (e.g. print) so having that as syntactic sugar for that would also be weird. There doesn't seem to be any _use_ for syntax like symbol{...}, so I feel for example s{...}, f{...}, and d{...} would be a natural extension to Python insofar as b'...' and u'...' are in the language, _even though_ sets and maps feel more general mathematically speaking (so the need for special use syntax _may_ be inappropriate (but people write math in languages ;)). From abarnert at yahoo.com Sun Feb 3 22:35:19 2013 From: abarnert at yahoo.com (Andrew Barnert) Date: Sun, 3 Feb 2013 13:35:19 -0800 Subject: [Python-ideas] for line in input with open(path) as input... In-Reply-To: <510D2DD9.7020109@umbrellacode.com> References: <510CFC94.5000502@umbrellacode.com> <510D0AA2.2060805@pearwood.info> <510D2DD9.7020109@umbrellacode.com> Message-ID: <8B2CB8DF-31AF-4290-AA4A-E2EC27D98F22@yahoo.com> On Feb 2, 2013, at 7:16, Shane Green wrote: > Thanks Nick. I definitely see your point about iterwith(); have been thinking about that since someone asked where __exit__() would be If iterwith calls close when the iterator is exhausted, it's useful. In that case, close on __exit__ is only a fallback in cases where you don't exhaust it. If it only has close on __exit__, it's useless, because its the same as not doing anything. A few months ago I proposed and unproposed a similar extension to generator expression syntax. People suggested multiple versions of an iterwith style function, and I think everyone (including me) agreed that this was a better answer even if you ignore the obvious huge benefit that it doesn't require changing the language. While your idea isn't identical to mine (in a generator expression, the with would have to come before the for, which doesn't feel as natural as your postfix, and it also breaks the lifting of the outermost iterable, which isn't an issue in the for statement), I think the same thing ends up being true in your case. for line in iterwith(open(path)): for line in f with open(path) as f: I think the first one is more readable. It also doesn't make you repeat the name of an otherwise-unnecessary variable twice. And it doesn't expose that variable to the loop body. If you _want_ to use f, i think you want the explicit with statement for clarity. The only real advantage to the second is that it immediately closes the file on break, instead of only doing so on normal exit or return. Sometimes that's important, but again, I think in most of those cases you'll want the explicit with scope for clarity. From timothy.c.delaney at gmail.com Sun Feb 3 22:39:57 2013 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Mon, 4 Feb 2013 08:39:57 +1100 Subject: [Python-ideas] constant/enum type in stdlib In-Reply-To: References: <51085AAB.6090303@canterbury.ac.nz> <5108A87D.9000207@canterbury.ac.nz> <20130130082639.0b28d7eb@pitrou.net> <20130201103656.0b24c40d@anarchist.wooz.org> Message-ID: On 2 February 2013 07:28, Tim Delaney wrote: > On 2 February 2013 02:36, Barry Warsaw wrote: > >> On Feb 01, 2013, at 03:18 PM, Tim Delaney wrote: >> >> >Last version (for now). I'm really interested in people's opinions on >> this. >> >For this version I've taken some inspiration from flufl.enum (but there >> >remains the major difference that these enums subclass int). >> >> Why not package it up and put it in PyPI? Better there than sitting in an >> email thread of some mailing list full of crazy people. :) >> > > Yep - I intend to. Was just hacking away initially to see if what I wanted > to achieve was feasible using Michael's metaclass as a base. It's in a > Mercurial repo so I'll put it up on BitBucket in the next day or two and > look at cleaning it up (documentation!) to put on PyPI. > Public repository on BitBucket: https://bitbucket.org/magao/enum Feel free to raise issues there, clone and make pull requests, etc. Tim Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Sun Feb 3 22:43:48 2013 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 04 Feb 2013 10:43:48 +1300 Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface In-Reply-To: References: Message-ID: <510EDA14.1030905@canterbury.ac.nz> > On Sat, Feb 2, 2013 at 8:36 PM, Nick Coghlan > wrote: > > The general idea of using factory functions for transport creation, > even for the "core transports" (like sockets, SSL and pipes) sounds > good to me. I've expressed some concerns previously about the breadth > of the event loop class API, and this would go a long way towards > alleviating them. How would this work, exactly? The implementation of these functions will depend on the platform and event loop being used, so there still needs to be some kind of dispatching mechanism based on the current event loop. -- Greg From timothy.c.delaney at gmail.com Mon Feb 4 00:07:01 2013 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Mon, 4 Feb 2013 10:07:01 +1100 Subject: [Python-ideas] constant/enum type in stdlib In-Reply-To: References: <51085AAB.6090303@canterbury.ac.nz> <5108A87D.9000207@canterbury.ac.nz> <20130130082639.0b28d7eb@pitrou.net> <20130201103656.0b24c40d@anarchist.wooz.org> Message-ID: On 4 February 2013 08:39, Tim Delaney wrote: > Public repository on BitBucket: https://bitbucket.org/magao/enum > > Feel free to raise issues there, clone and make pull requests, etc. > As Eli has noted in the issues, no comments or anything except the unit tests yet. Lots of magic. Needs significant cleanup. And I'll note again in case it's not clear - this currently *only* works with Python 3.3. Tim Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Feb 4 00:19:18 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 4 Feb 2013 09:19:18 +1000 Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface In-Reply-To: <510EDA14.1030905@canterbury.ac.nz> References: <510EDA14.1030905@canterbury.ac.nz> Message-ID: On 4 Feb 2013 07:45, "Greg Ewing" wrote: >> >> On Sat, Feb 2, 2013 at 8:36 PM, Nick Coghlan > wrote: >> >> The general idea of using factory functions for transport creation, >> even for the "core transports" (like sockets, SSL and pipes) sounds >> good to me. I've expressed some concerns previously about the breadth >> of the event loop class API, and this would go a long way towards >> alleviating them. > > > How would this work, exactly? The implementation of these > functions will depend on the platform and event loop being > used, so there still needs to be some kind of dispatching > mechanism based on the current event loop. Ben covered that in his original post: the event loop has a registry of transport factories, keyed by the factory name (e.g. "mycoolmodule.myneattransport". The factory functions check the event loop registry for an implementation and use it if they find it. Otherwise, they fall back to a non-optimised implementation based on the standard event loop API, or they throw an error indicating that the transport doesn't support the current event loop. It's a really elegant design that: - provides a consistent user experience between "first party" and "third party" transport implementations, rather than switching arbitrarily from "event loop method" to "module level function" based on an implementation detail - allows an optimised transport implementation to be added to an event loop without requiring application code to change in order to take advantage of it and without even requiring cooperation from the event loop developers - the transport creation API also extends cleanly to the creation of protocol stacks A per event loop implementation transport registry makes sense for the same reason the codec registry makes sense. The only differences are that: - the different transports may offer different APIs - the registry isn't global, but specific to each event loop implementation. Cheers, Nick. > > -- > Greg > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas -------------- next part -------------- An HTML attachment was scrubbed... URL: From jbvsmo at gmail.com Mon Feb 4 00:53:26 2013 From: jbvsmo at gmail.com (=?ISO-8859-1?Q?Jo=E3o_Bernardo?=) Date: Sun, 3 Feb 2013 21:53:26 -0200 Subject: [Python-ideas] constant/enum type in stdlib In-Reply-To: References: <51085AAB.6090303@canterbury.ac.nz> <5108A87D.9000207@canterbury.ac.nz> <20130130082639.0b28d7eb@pitrou.net> <20130201103656.0b24c40d@anarchist.wooz.org> Message-ID: Hi, about this enum/const thing, The use case I like more is a class where you know all the instances and not just a sequence of names. Particularly It would be nice to have custom attributes and methods besides the value and the name. I have my own implementation with a basic api somewhat borrowed from flufl.enum (plus a lot of other stuff), but with this kind of support: https://github.com/jbvsmo/makeobj I couldn't find the best way to express enums with the current python syntax, so I also wrote a simple regex-parsed language to fit objects with an arbitrary level of complexity. I think, enumeration per se is not much more useful than just a bunch of integers... Having this kind of control IMO is. Although Java is not a good example of anything, they have a similar feature. What do you people think? Jo?o Bernardo 2013/2/3 Tim Delaney > On 4 February 2013 08:39, Tim Delaney wrote: > >> Public repository on BitBucket: https://bitbucket.org/magao/enum >> >> Feel free to raise issues there, clone and make pull requests, etc. >> > > As Eli has noted in the issues, no comments or anything except the unit > tests yet. Lots of magic. Needs significant cleanup. > > And I'll note again in case it's not clear - this currently *only* works > with Python 3.3. > > Tim Delaney > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From timothy.c.delaney at gmail.com Mon Feb 4 01:17:29 2013 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Mon, 4 Feb 2013 11:17:29 +1100 Subject: [Python-ideas] constant/enum type in stdlib In-Reply-To: References: <51085AAB.6090303@canterbury.ac.nz> <5108A87D.9000207@canterbury.ac.nz> <20130130082639.0b28d7eb@pitrou.net> <20130201103656.0b24c40d@anarchist.wooz.org> Message-ID: On 4 February 2013 10:53, Jo?o Bernardo wrote: > Hi, about this enum/const thing, The use case I like more is a class where > you know all the > instances and not just a sequence of names. > Particularly It would be nice to have custom attributes and methods > besides the value and the name. > > I have my own implementation with a basic api somewhat borrowed from > flufl.enum (plus a lot of other stuff), > but with this kind of support: https://github.com/jbvsmo/makeobj > I considered it, and in fact you could almost do it with my implementation by using a custom subclass of EnumValue (except trying it has just exposed a design flaw with the whole _EnumProxy bit). Works if you create the enum in the same module as EnumValues, fails otherwise. Going to have to have a rethink. Tim Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: From desnacked at riseup.net Mon Feb 4 01:18:35 2013 From: desnacked at riseup.net (desnacked at riseup.net) Date: Sun, 3 Feb 2013 16:18:35 -0800 Subject: [Python-ideas] Specification of procedures to store cryptographic secrets Message-ID: <7cf4246409c5b449716dd1c2aec287b5.squirrel@fruiteater.riseup.net> Some hours ago I sent an email to python-crypto asking how to securely wipe cryptographic secrets from memory: http://mail.python.org/pipermail/python-crypto/2013-February/001170.html Antoine said that cryptographic secret wiping could be achieved if one uses bytearrays carefully and then overwrites their contents after use. I agree that this sounds reasonable, but I think it would be even better if that was a documented property of bytearrays. If that property of bytearrays was specified in the Python standards, it would be easier for people who write cryptographic applications and libraries to use bytearrays correctly, and it would also guarantee that this property won't change in future versions of Python. Furthermore, it would help authors of cryptographic libraries to design their APIs and internal functions in a way that would allow the secure erasure of sensitive data. Would this make sense or am I asking too much from Python? From jbvsmo at gmail.com Mon Feb 4 01:25:55 2013 From: jbvsmo at gmail.com (=?ISO-8859-1?Q?Jo=E3o_Bernardo?=) Date: Sun, 3 Feb 2013 22:25:55 -0200 Subject: [Python-ideas] constant/enum type in stdlib In-Reply-To: References: <51085AAB.6090303@canterbury.ac.nz> <5108A87D.9000207@canterbury.ac.nz> <20130130082639.0b28d7eb@pitrou.net> <20130201103656.0b24c40d@anarchist.wooz.org> Message-ID: 2013/2/3 Tim Delaney > On 4 February 2013 10:53, Jo?o Bernardo wrote: > >> Hi, about this enum/const thing, The use case I like more is a class >> where you know all the >> instances and not just a sequence of names. >> Particularly It would be nice to have custom attributes and methods >> besides the value and the name. >> >> I have my own implementation with a basic api somewhat borrowed from >> flufl.enum (plus a lot of other stuff), >> but with this kind of support: https://github.com/jbvsmo/makeobj >> > > I considered it, and in fact you could almost do it with my implementation > by using a custom subclass of EnumValue (except trying it has just exposed > a design flaw with the whole _EnumProxy bit). Works if you create the enum > in the same module as EnumValues, fails otherwise. Going to have to have a > rethink. > > For attributes, it would probably be easy to do, but for methods you will probably need a new _EnumProxy subclass for each class. I did this with a metaclass factory. Jo?o Bernardo -------------- next part -------------- An HTML attachment was scrubbed... URL: From dustin at v.igoro.us Mon Feb 4 01:30:09 2013 From: dustin at v.igoro.us (Dustin J. Mitchell) Date: Sun, 3 Feb 2013 19:30:09 -0500 Subject: [Python-ideas] Specification of procedures to store cryptographic secrets In-Reply-To: <7cf4246409c5b449716dd1c2aec287b5.squirrel@fruiteater.riseup.net> References: <7cf4246409c5b449716dd1c2aec287b5.squirrel@fruiteater.riseup.net> Message-ID: On Sun, Feb 3, 2013 at 7:18 PM, wrote: > If that property of bytearrays was specified in the Python standards, it > would be easier for people who write cryptographic applications and > libraries to use bytearrays correctly, and it would also guarantee that > this property won't change in future versions of Python. Furthermore, it > would help authors of cryptographic libraries to design their APIs and > internal functions in a way that would allow the secure erasure of > sensitive data. It would similarly be helpful to add low-level support for "pinning" such memory so that it is not written to backing store. While that can be done with the mmap module, the details are tricky. I don't think that this belongs in the Python core, though. Rather, I think that this should be implemented in a module which can be used in conjunction with bytearrays, mmap, and any other necessary pieces of the core and stdlib. In fact, such a thing might already exist - I haven't looked (it's really not within my area of interest) Putting such a thing in the stdlib might achieve the guarantee you suggest, but it might not. It really just shifts responsibility for ensuring good cryptographic programming onto people who spend their time implementing programming languages. Dustin From christian at python.org Mon Feb 4 02:11:00 2013 From: christian at python.org (Christian Heimes) Date: Mon, 04 Feb 2013 02:11:00 +0100 Subject: [Python-ideas] Specification of procedures to store cryptographic secrets In-Reply-To: <7cf4246409c5b449716dd1c2aec287b5.squirrel@fruiteater.riseup.net> References: <7cf4246409c5b449716dd1c2aec287b5.squirrel@fruiteater.riseup.net> Message-ID: > Some hours ago I sent an email to python-crypto asking how to securely > wipe cryptographic secrets from memory: > http://mail.python.org/pipermail/python-crypto/2013-February/001170.html > > Antoine said that cryptographic secret wiping could be achieved if one > uses bytearrays carefully and then overwrites their contents after use. I > agree that this sounds reasonable, but I think it would be even better if > that was a documented property of bytearrays. That might work if you never ever resize a bytearray during its life cycle. A resize op calls realloc() which may copy the data to a new memory region. The old region isn't zeroed. The approach only takes care of the object itself on the heap. Some function may store data on the stack or make a temporary copy to another memory location on the heap. You have to compensate for that. libtomcrypt has a function burn_stack() that allocates and overwrites memory on the stack with a recursive function call. Christian From greg at krypto.org Mon Feb 4 02:54:39 2013 From: greg at krypto.org (Gregory P. Smith) Date: Sun, 3 Feb 2013 17:54:39 -0800 Subject: [Python-ideas] Specification of procedures to store cryptographic secrets In-Reply-To: References: <7cf4246409c5b449716dd1c2aec287b5.squirrel@fruiteater.riseup.net> Message-ID: Correct. this isn't something that belongs in the core python language and types. something needing memory-pinning and secure wiping should be implemented as a special type (c extension module) for use with the c extension libraries that need those properties. as soon as anything enters python's own types or values ever make it into python code in any way, no guarantees can ever be made as to how many copies were made and scattered around the process's own address space. assume "many". Python doesn't implement any sort of chain of custody for data internally. On Sun, Feb 3, 2013 at 5:11 PM, Christian Heimes wrote: > > Some hours ago I sent an email to python-crypto asking how to securely > > wipe cryptographic secrets from memory: > > http://mail.python.org/pipermail/python-crypto/2013-February/001170.html > > > > Antoine said that cryptographic secret wiping could be achieved if one > > uses bytearrays carefully and then overwrites their contents after use. I > > agree that this sounds reasonable, but I think it would be even better if > > that was a documented property of bytearrays. > > That might work if you never ever resize a bytearray during its life > cycle. A resize op calls realloc() which may copy the data to a new > memory region. The old region isn't zeroed. > > The approach only takes care of the object itself on the heap. Some > function may store data on the stack or make a temporary copy to another > memory location on the heap. You have to compensate for that. > libtomcrypt has a function burn_stack() that allocates and overwrites > memory on the stack with a recursive function call. > > Christian > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > -------------- next part -------------- An HTML attachment was scrubbed... URL: From christian at python.org Mon Feb 4 03:12:38 2013 From: christian at python.org (Christian Heimes) Date: Mon, 04 Feb 2013 03:12:38 +0100 Subject: [Python-ideas] Specification of procedures to store cryptographic secrets In-Reply-To: References: <7cf4246409c5b449716dd1c2aec287b5.squirrel@fruiteater.riseup.net> Message-ID: <510F1916.4040307@python.org> Am 04.02.2013 02:54, schrieb Gregory P. Smith: > Correct. this isn't something that belongs in the core python language > and types. something needing memory-pinning and secure wiping should be > implemented as a special type (c extension module) for use with the c > extension libraries that need those properties. as soon as anything > enters python's own types or values ever make it into python code in any > way, no guarantees can ever be made as to how many copies were made and > scattered around the process's own address space. assume "many". > > Python doesn't implement any sort of chain of custody for data internally. I agree! A custom type came into my mind, too. Data wiping is merely a small part of the general issue. A confident and secure container for secrets must do more. For example it has to prevent the memory page from getting swapped to disk with mlock(2). Lot's of bad things can happen when you look at L1/L2/L3 CPU cache, hyper threading and virtualization. All that stuff makes it hard to conceal secrets. On the bright side attacks rarely crack cryptography. In most cases it's easier, faster and less costly to do social engineering. Humans are lazy, ignorant and bribable. Christian From tjreedy at udel.edu Mon Feb 4 03:31:20 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 03 Feb 2013 21:31:20 -0500 Subject: [Python-ideas] Specification of procedures to store cryptographic secrets In-Reply-To: <7cf4246409c5b449716dd1c2aec287b5.squirrel@fruiteater.riseup.net> References: <7cf4246409c5b449716dd1c2aec287b5.squirrel@fruiteater.riseup.net> Message-ID: On 2/3/2013 7:18 PM, desnacked at riseup.net wrote: > Some hours ago I sent an email to python-crypto asking how to securely > wipe cryptographic secrets from memory: > http://mail.python.org/pipermail/python-crypto/2013-February/001170.html > > Antoine said that cryptographic secret wiping could be achieved if one > uses bytearrays carefully and then overwrites their contents after use. I I presume he meant with CPython with its non-compacting gc on current major OSes. Perhaps the system also needs to be unloaded enough that the memory is not written to disk. Or the secret is written and erased before that would happen. > agree that this sounds reasonable, but I think it would be even better if > that was a documented property of bytearrays. I do not think such a low-level special-case property would be appropriate. Python is a high-level languages for manipulating fairly abstract objects defined by interface and behavior. The reference manual defining the language intentionally says almost nothing about the hardware and memory of an implementation. This is partly why Python is relatively easy to read and mentally execute in a human brain. One usually does not need to mentally simulate a linear byte memory. > If that property of bytearrays was specified in the Python standards, it > would be easier for people who write cryptographic applications and > libraries to use bytearrays correctly, and it would also guarantee that > this property won't change in future versions of Python. This would mean that Python could not run on hardware that made the guarantee impossible. What if a future OS ran directly off an SSD, either dispensing with current DRAM, or using it as the outer cache layer? My understanding is that SSDs run independently with their own os and that external access is to logical rather than physical memory. What if a farther future system had a write-once, read-many, never-erase petabyte or exabyte 3d cube memory, with the SSD only serving as an index? Of course, it is possible that security concern will figure into future designs. The statement 'del x' only means "break the association between the name 'x' and the object currently associated with 'x'". If that is the last link to the object, it becomes inaccessible from Python and *eligible* to be physically deleted. What what happens in concrete hardware is explicitly not Python's concern. From 3.1. Objects, values and types: "Objects are never explicitly destroyed; however, when they become unreachable they may be garbage-collected. An implementation is allowed to postpone garbage collection or omit it altogether ..." > Furthermore, it > would help authors of cryptographic libraries to design their APIs and > internal functions in a way that would allow the secure erasure of > sensitive data. I agree with Dustin that you need a 3rd-party crytobytes module. It could be specific to OS and hardware, keep up with changes, and refuse to run if the required guarantees cannot be met. > Would this make sense or am I asking too much from Python? To me, it makes perfect sense for you to want a cryptobytes class that does exactly what you want it to do. And, again to me, you are asking too much for such to be part of the stdlib. Whether you are asking too much of any particular OS is beyond my knowledge. If the OS can provide the guarantees, a 3rd party Python wrapping should be possible. -- Terry Jan Reedy From eliben at gmail.com Mon Feb 4 05:51:35 2013 From: eliben at gmail.com (Eli Bendersky) Date: Sun, 3 Feb 2013 20:51:35 -0800 Subject: [Python-ideas] constant/enum type in stdlib In-Reply-To: References: <51085AAB.6090303@canterbury.ac.nz> <5108A87D.9000207@canterbury.ac.nz> <20130130082639.0b28d7eb@pitrou.net> <20130201103656.0b24c40d@anarchist.wooz.org> Message-ID: On Sun, Feb 3, 2013 at 3:53 PM, Jo?o Bernardo wrote: > Hi, about this enum/const thing, The use case I like more is a class where > you know all the > instances and not just a sequence of names. > Particularly It would be nice to have custom attributes and methods > besides the value and the name. > > I have my own implementation with a basic api somewhat borrowed from > flufl.enum (plus a lot of other stuff), > but with this kind of support: https://github.com/jbvsmo/makeobj > > I couldn't find the best way to express enums with the current python > syntax, so I also wrote a simple > regex-parsed language to fit objects with an arbitrary level of > complexity. I think, enumeration per se > is not much more useful than just a bunch of integers... Having this kind > of control IMO is. > > Although Java is not a good example of anything, they have a similar > feature. What do you people think? > > Personally, I disagree with the "more features is better" approach. Features have a cost - they complicate the implementation which makes it fragile, harder to maintain and harder to understand. Even more importantly, they make *user* code harder to understand. Therefore it's IMHO best to decide on a basic functionality that brings most of the benefits, and then think about how to make the implementation *simpler*. All I really want from an enum is what I have in C and lack in Python - a nice, minimally type-safe way to name special constants. Having this in hand, I want the simplest and cleanest syntax possible, not more features. Tim's implementation strikes a good balance - the syntax is as minimal as can be in a library implementation, and it provides all the basic features well. It adds some more, possibly at the cost of complexity, which may or may not be good. This is why I think that a PEP weighing features for/against inclusion is a logical next step. Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Mon Feb 4 06:25:45 2013 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 04 Feb 2013 18:25:45 +1300 Subject: [Python-ideas] Specification of procedures to store cryptographic secrets In-Reply-To: <7cf4246409c5b449716dd1c2aec287b5.squirrel@fruiteater.riseup.net> References: <7cf4246409c5b449716dd1c2aec287b5.squirrel@fruiteater.riseup.net> Message-ID: <510F4659.4050607@canterbury.ac.nz> desnacked at riseup.net wrote: > Antoine said that cryptographic secret wiping could be achieved if one > uses bytearrays carefully and then overwrites their contents after use. I > agree that this sounds reasonable, but I think it would be even better if > that was a documented property of bytearrays. I think to fully guarantee that you would need a promise from the OS that overwriting a particular piece of your virtual address space removes all evidence of that data from swap space, etc. I don't know whether any current OSes provide that kind of guarantee. -- Greg From mal at egenix.com Mon Feb 4 12:21:20 2013 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 04 Feb 2013 12:21:20 +0100 Subject: [Python-ideas] Specification of procedures to store cryptographic secrets In-Reply-To: <7cf4246409c5b449716dd1c2aec287b5.squirrel@fruiteater.riseup.net> References: <7cf4246409c5b449716dd1c2aec287b5.squirrel@fruiteater.riseup.net> Message-ID: <510F99B0.3000600@egenix.com> On 04.02.2013 01:18, desnacked at riseup.net wrote: > Some hours ago I sent an email to python-crypto asking how to securely > wipe cryptographic secrets from memory: > http://mail.python.org/pipermail/python-crypto/2013-February/001170.html > > Antoine said that cryptographic secret wiping could be achieved if one > uses bytearrays carefully and then overwrites their contents after use. I > agree that this sounds reasonable, but I think it would be even better if > that was a documented property of bytearrays. > > If that property of bytearrays was specified in the Python standards, it > would be easier for people who write cryptographic applications and > libraries to use bytearrays correctly, and it would also guarantee that > this property won't change in future versions of Python. Furthermore, it > would help authors of cryptographic libraries to design their APIs and > internal functions in a way that would allow the secure erasure of > sensitive data. > > Would this make sense or am I asking too much from Python? I don't think there's any safe way to store crypto information in memory. You'd have to use a dedicated hardware crypto device to avoid leaking the keys (think memory reallocation, the OS swapping memory to disk, your code running on a VM, etc.). See e.g. http://c0decstuff.blogspot.de/2011/01/in-memory-extraction-of-ssl-private.html for an example on how to do this intentionally. Not even OpenSSL tries to address this, so I think it's asking a bit much from Python ;-) That said, adding a little more security to a custom blob type would certainly not hurt :-) Here's some inspiration for locking and cleaning memory: http://c0decstuff.blogspot.de/2011/01/in-memory-extraction-of-ssl-private.html (pages 36ff) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 04 2013) >>> Python Projects, Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From guido at python.org Mon Feb 4 20:02:57 2013 From: guido at python.org (Guido van Rossum) Date: Mon, 4 Feb 2013 11:02:57 -0800 Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface In-Reply-To: References: Message-ID: I'm going to try and snip as much as I can to get the the heart of this... On Sun, Feb 3, 2013 at 11:55 AM, Ben Darnell wrote: > UDP is a real-life example from tornado - we don't have any built-in > support for UDP, but people who need it have been able to build it without > touching tornado itself. The same argument would apply to pipes or any > number of other (admittedly much more esoteric) network protocols. I'll > elaborate on the UDP example below. Hm. UDP is relatively easy because it uses sockets. Pipes are harder -- testing for the presence of add_reader (etc. -- I will leave this off from now on) isn't enough, because select on Windows does not support pipes. Thinking about what you could mean by "more esoteric protocols", there's really not much at the level of TCP and UDP that comes to mind. UNIX-domain sockets, and perhaps the (root-only) protocol for sniffing packets (raw sockets?). A new feature just landed in Tulip (I still have to update PEP 3156) where you can pass a pre-constructed socket object to create_connection() and start_serving(), which will make it a little easier to support esoteric ways of setting up the socket; however, create_connection() is still limited to sockets that implement a byte stream, because of the way the transport/protocol API works. > Right. Third-party extensions to the event loop interface are inherently > problematic, so we'll have to provide them in some other way. I'm proposing > a pattern for that "some other way" and then realizing that I like it even > for first-party interfaces. Glad that is out of the way. But I'm still skeptical -- first, as I explained before, I am actually in favor of using different styles for 1st and 3rd party interfaces, so the status of the interface used is obvious to the reader (and the coder, in case they are copy-pasting recipes :-); second, I don't expect there will be too many opportunities to put the pattern at work. Note that I'm only talking about 3rd party *interfaces* -- if a 3rd party module implements a 1st party interface (e.g. the stream transport/protocol interface specified in PEP 3156) it can just use the event loop create_connection method (assuming it is also implementing a new event loop -- otherwise what would be the point of the 3rd party code?). And note that even UDP requires a different interface between transport and protocol -- e.g. the protocol method called should be packet_received() rather than data_received(), and the protocol should probably not implement write/writelines but send and send_multiple. And the signatures of these methods will be different because (depending on how you use UDP) you have to have a parameter for the peer address. And yet, implementing UDP as pure 3rd party code using add_reader is simple, as long as the event loop supports add_reader. You just can't use create_connection or start_serving -- but those are really just convenience methods that are easily reimplemented. (We could refactor the standard implementations to have more reusable parts, but we'd run into the same problem as with add_reader -- while most UNIXy event loops will easily support such refactorings, that's not the case with event loops based on IOCP, other other libraries that don't naturally offer add_reader functionality. (Not sure if that's the case for libuv.) All this makes me skeptical that a single API should be used to register "transports". At the very least you will need different registries for each distinct transport/protocol interface; in addition, custom transports (even if they implement the same transport/protocol interface) may have different constructor arguments (e.g. consider plain TCP vs. SSL in Tulip). > Suppose twisted did not have UDP support built in. Most reactor > implementations subclass PosixReactorBase (with IOCPReactor as the notable > exception). Twisted can add UDP support and implement listenUDP in > PosixReactorBase and IOCPReactor, and suddenly most reactors (even > third-party ones like TornadoReactor) support UDP for free. If you say so. I don't know enough about Twisted's internals to verify this claim. Depending on how things were factored I could easily imagining something in PosixReactorBase making the assumption of a stream protocol somewhere. In a stream protocol like TCP, it is safe to collapse two consecutive sends into one, and to split one send into multiples. But not for datagram protocols like UDP. In an ideal world, knowledge of all this is completely left out of the reactor. But, in a hypothetical world where Twisted only supported streams, who knows whether that is done? > Those that > don't (a hypothetical LibUVReactor?) can implement it themselves and > interoperate with everything else. In practice I suspect that the number of 3rd party event loop implementations that support add_reader and let a different 3rd party's UDP implementation succeed will be vanishingly small. Even smaller if you don't count the ones that are essentially clones or subclasses of Tulip's UNIX support. > If a third party wanted to add UDP support separately from twisted's release > schedule, they can't do with an interface that is generically usable across > all reactors. They could make a static function listenUDP() that works with > any IReactorFDSet, and maybe special-case IOCPReactor, but then there'd be > no way for a third-party LibUVReactor to participate. This sounds unavoidable no matter how you refactor the interface. There are potentially event loop implementations that don't use socket objects at all. (And yes, those will have to reject the 'sock' argument to create_connection and start_serving; and I have to change start_serving's return type to be something other than a socket.) When you implement a new 3rd party transport, you are pretty much inevitably limiting yourself to a subset of event loops. That subset won't be empty, and it will be sufficient for your purpose, but the ideal of portability across all (or even most) event loops, including ones that haven't been written yet, is unattainable. I certainly haven't seen an indication that your proposed registry will address this. > add_reader is not very limiting except for its platform-specificity. It's > possible to have a generic protocol across all posixy event loops and then > special-case the small number of interesting non-posixy ones (or maybe there > is some other class of methods that could be standardized for other > platforms? Is there some set of methods analogous to add_reader that > multiple IOCP-based loops could share?) Not exactly analogous -- the whole point of IOCP is that it is not "ready-based" but "completion-based". The sock_recv (etc.) methods on the event loop are my attempt to suggest a way for other completion-based event loops to open themselves up for new transport implementations, but this is much more limiting than add_reader -- e.g. I suspect that IOCP will let you read from a named pipe, but you must use a different library call than for receiving from a socket; even receiving a packet from UDP will require a different method. This issue doesn't exist in the same way for add_reader, because the system call to do the read is not made by the event loop, it is made by the transport. > This version doesn't change much, it's mainly to set the stage for the > following variations. However, it does have a few nice properties - it > keeps the (public) event loop interface small and manageable, and callers > don't need to touch actual event loop objects unless they want to have more > than one. Not quite -- the call_soon(), call_later() etc. functionality is also exposed as event loop methods. > From a stylistic perspective I like this style of interface more > than using dozens of methods on the event loop object itself (even if those > dozens of methods are still there but hidden as an implementation detail). You can't argue about style. :-) > When third-party modules get absorbed into the standard library, it's often > possible to support both just by trying different imports until one works > (unittest.mock vs mock, json vs simplejson, etc). Sometimes a module's > interface gets cleaned up and rearranged in the process, but that seems to > be less common. It would be nice if a third-party transport could get > standardized and the only thing callers would need to change is their > imports. However, this is a minor concern; as I wrote up this design I > realized I liked it for first-party work even if there were no third-party > modules to be consistent with. You can't argue about style. :-) > But libuv doesn't necessarily contain the transport creation function. But a libuv-based PEP 3156-compliant event loop implementation must. > The > idea is that someone can propose a transport interface in a third-party > module (mymodule.listen_udp in this example), implement it themselves for > some event loop implementations, and other event loops can declare > themselves compatible with it. Aha! This is the executive summary of your proposal, or at least your goal for it. This is hypothesizing rather a lot of goodwill and coordination between different 3rd party developers. And the registry offered by the event loop comes down to not much more than a dictionary with keys that follow a certain convention (e.g. fully-qualified package+module name plus some identifier for the feature) and nothing can be said about what the items stored in the registry are (since a packet transport and a stream transport are not interchangeable, and even two stream transports may not be). Given that for each 3rd party transport the details of how to implement a compatible version of it will vary hugely, both depending on what the transport is trying to do and how the event loop works, I expect that the market for this registry will be rather small. And when a particular 3rd party transport wants to enable other 3rd party events to support them, they can implement their own registry, which the other 3rd party could then plug into. (But see below.) > (And in an admittedly far-fetched scenario, if there were two third-party > UDP interfaces and LibUVEventLoop implemented one of them, yet another party > could build a bridge between the two, and then they'd plug it in with > register_implementation) I do see one argument in favor of having a standard registry on the event loop, even if it's just a dict with register/lookup APIs and a naming convention, and no semantics assigned to the items registered. That argument is to make 3rd party transport implementers aware of the possibility that some other 3rd party might want to offer a compatible implementation aimed at an event loop that's not supported natively by the (former) 3rd party transport. And I could even be convinced that the standard protocols should use this registry so that the source code serves as an example of best practices. Still, it's a pretty weak argument IMO -- I don't expect there to be a significant cottage industry cranking out 3rd party protocol implementations, assuming we add UDP to PEP 3156, which is my plan. And I don't think that create_connection() should be used to create UDP connections -- the signature of the protocol factory passed in would be quite different, for starters, and the set of options needed to configure the transport is also different (assuming we want to support both connected and connection-less UDP). > I don't think the status quo prevents the development of third-party > transports, but it does implicitly encourage two bad habits: A) adding > methods to the event loop, inviting name collisions, or B) just building on > add_reader and friends without thinking about non-posix platforms. Of > course, no one expects a groundswell of third-party development at the event > loop and transport level, so this could just be so much overengineering, but > I like it from a stylistic perspective even without the third-party > benefits. Yeah, so that's the rub: I'm not so keen on adding extra machinery to the PEP that I don't expect to be used much. I have more important fish to fry (such as adding UDP :-). And adding a registry one whole Python release cycle later (e.g. in 3.5, assuming PEP 3156 is standardized and included in 3.4) doesn't strike me as such a bad thing -- I don't think we're painting ourselves into much of a corner by not having a registry right from the start. -- --Guido van Rossum (python.org/~guido) From ericsnowcurrently at gmail.com Tue Feb 5 00:57:05 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Mon, 4 Feb 2013 16:57:05 -0700 Subject: [Python-ideas] Extend module objects to support properties In-Reply-To: References: <51087225.3040801@hastings.org> <51094D8D.606@hastings.org> Message-ID: On Thu, Jan 31, 2013 at 5:50 PM, Nick Coghlan wrote: > On Fri, Feb 1, 2013 at 5:56 AM, Eric Snow wrote: >> ... or making sys an >> instance of another type, as someone suggested. > > sys is already special cased so heavily in the interpreter > initialization, making it a little more special really wouldn't bother > me much :) One benefit of this is that sys.modules could be made a [carefully done] property that wraps interp->modules, making sys.modules truly replaceable, which would simplify a number of different places where we've had to work around it. -eric From ben at bendarnell.com Tue Feb 5 05:50:20 2013 From: ben at bendarnell.com (Ben Darnell) Date: Mon, 4 Feb 2013 23:50:20 -0500 Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface In-Reply-To: References: Message-ID: On Mon, Feb 4, 2013 at 2:02 PM, Guido van Rossum wrote: > I'm going to try and snip as much as I can to get the the heart of this... > > Me too... > > Thinking about what you could mean by "more esoteric protocols", > there's really not much at the level of TCP and UDP that comes to > mind. UNIX-domain sockets, and perhaps the (root-only) protocol for > sniffing packets (raw sockets?). > Looking at twisted as an example, the things that are supported across both PosixReactorBase and IOCPReactor are TCP, UDP, SSL, multicast, and subprocesses. Unix domain sockets aren't that interesting here since any system that supports them will (presumably?) support add_reader and things will just work. > > A new feature just landed in Tulip (I still have to update PEP 3156) > where you can pass a pre-constructed socket object to > create_connection() and start_serving(), which will make it a little > easier to support esoteric ways of setting up the socket; however, > create_connection() is still limited to sockets that implement a byte > stream, because of the way the transport/protocol API works. > Cool. That's definitely useful (especially on the server side), but we'll still need a separate interface for datagrams. > > > Right. Third-party extensions to the event loop interface are inherently > > problematic, so we'll have to provide them in some other way. I'm > proposing > > a pattern for that "some other way" and then realizing that I like it > even > > for first-party interfaces. > > Glad that is out of the way. But I'm still skeptical -- first, as I > explained before, I am actually in favor of using different styles for > 1st and 3rd party interfaces, so the status of the interface used is > obvious to the reader (and the coder, in case they are copy-pasting > recipes :-); second, I don't expect there will be too many > opportunities to put the pattern at work. > You can't argue about style. :-) > > The > > idea is that someone can propose a transport interface in a third-party > > module (mymodule.listen_udp in this example), implement it themselves for > > some event loop implementations, and other event loops can declare > > themselves compatible with it. > > Aha! This is the executive summary of your proposal, or at least your > goal for it. > > This is hypothesizing rather a lot of goodwill and coordination > between different 3rd party developers. Yeah, history is unfortunately not very supportive of the idea that developers of asynchronous frameworks will coordinate on this kind of thing :) > And the registry offered by > the event loop comes down to not much more than a dictionary with keys > that follow a certain convention (e.g. fully-qualified package+module > name plus some identifier for the feature) and nothing can be said > about what the items stored in the registry are (since a packet > transport and a stream transport are not interchangeable, and even two > stream transports may not be). > > Given that for each 3rd party transport the details of how to > implement a compatible version of it will vary hugely, both depending > on what the transport is trying to do and how the event loop works, I > expect that the market for this registry will be rather small. And > when a particular 3rd party transport wants to enable other 3rd party > events to support them, they can implement their own registry, which > the other 3rd party could then plug into. (But see below.) > True. > > > (And in an admittedly far-fetched scenario, if there were two third-party > > UDP interfaces and LibUVEventLoop implemented one of them, yet another > party > > could build a bridge between the two, and then they'd plug it in with > > register_implementation) > > I do see one argument in favor of having a standard registry on the > event loop, even if it's just a dict with register/lookup APIs and a > naming convention, and no semantics assigned to the items registered. > That argument is to make 3rd party transport implementers aware of the > possibility that some other 3rd party might want to offer a compatible > implementation aimed at an event loop that's not supported natively by > the (former) 3rd party transport. And I could even be convinced that > the standard protocols should use this registry so that the source > code serves as an example of best practices. > > Still, it's a pretty weak argument IMO -- I don't expect there to be a > significant cottage industry cranking out 3rd party protocol > implementations, assuming we add UDP to PEP 3156, which is my plan. > Yeah, as long as we get TCP, UDP, SSL, and pipes (at least for subprocesses), I'm hard pressed to imagine anything else that would be in so much demand it would need to be supported by all event loops. > And I don't think that create_connection() should be used to create > UDP connections -- the signature of the protocol factory passed in > would be quite different, for starters, and the set of options needed > to configure the transport is also different (assuming we want to > support both connected and connection-less UDP). > Of course. (I may have been unclear somewhere along the way, but it was never my intention for create_connection to work for both TCP and UDP) > > > I don't think the status quo prevents the development of third-party > > transports, but it does implicitly encourage two bad habits: A) adding > > methods to the event loop, inviting name collisions, or B) just building > on > > add_reader and friends without thinking about non-posix platforms. Of > > course, no one expects a groundswell of third-party development at the > event > > loop and transport level, so this could just be so much overengineering, > but > > I like it from a stylistic perspective even without the third-party > > benefits. > > Yeah, so that's the rub: I'm not so keen on adding extra machinery to > the PEP that I don't expect to be used much. I have more important > fish to fry (such as adding UDP :-). And adding a registry one whole > Python release cycle later (e.g. in 3.5, assuming PEP 3156 is > standardized and included in 3.4) doesn't strike me as such a bad > thing -- I don't think we're painting ourselves into much of a corner > by not having a registry right from the start. > Fair enough. I may use this pattern when/if I retrofit Tornado to be IOCP-friendly since we currently create transports with static functions, but for Tulip let's wait and see if the problem this proposal is trying to solve ever materializes. -Ben > > -- > --Guido van Rossum (python.org/~guido) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From saghul at gmail.com Tue Feb 5 15:48:43 2013 From: saghul at gmail.com (=?ISO-8859-1?Q?Sa=FAl_Ibarra_Corretg=E9?=) Date: Tue, 05 Feb 2013 15:48:43 +0100 Subject: [Python-ideas] PEP 3156 / Tulip: Remove 'when' from Handler Message-ID: <51111BCB.6070609@gmail.com> Hi, While working on some changes for rose (an eventloop for tulip based on libuv) I realized the Handler class is a bit too coupled to the event loop implementation currently present in Tulip. pyuv (and pyev, and probably others) provide timers already, so there is no need to implement them "manually" and keep a heap with them. This basically means that Handler objects will always have the 'when' value set to None, and the comparison functions don't really apply, moreover __eq__ would even be incorrect in its current form. IMHO it would be good to have a base Handler class which doesn't take time into account and then a Timer class which does. Thoughts? Also, I see that the PEP does mention the possibility of providing a __call__ method on the Handler class itself, which would basically call the callback and suppress exceptions (I guess by logging them). Was this ever discussed? Sounds like a good idea to me :-) Regards, -- Sa?l Ibarra Corretg? http://saghul.net/blog | http://about.me/saghul From guido at python.org Tue Feb 5 16:49:43 2013 From: guido at python.org (Guido van Rossum) Date: Tue, 5 Feb 2013 07:49:43 -0800 Subject: [Python-ideas] PEP 3156 / Tulip: Remove 'when' from Handler In-Reply-To: <51111BCB.6070609@gmail.com> References: <51111BCB.6070609@gmail.com> Message-ID: I think both suggestions are good ones. The when field has bothered me for a while, although it did not come to the surface. Adding the call/catch/log logic on the Handler seems nice too, but I think I would rather not use handler() -- I would rather use handler.call(). I.e. make the method name 'call', not '__call__'. (The latter almost always makes APIs more confusing IMO.) Can you contribute code? --Guido On Tue, Feb 5, 2013 at 6:48 AM, Sa?l Ibarra Corretg? wrote: > Hi, > > While working on some changes for rose (an eventloop for tulip based on > libuv) I realized the Handler class is a bit too coupled to the event loop > implementation currently present in Tulip. > > pyuv (and pyev, and probably others) provide timers already, so there is no > need to implement them "manually" and keep a heap with them. This basically > means that Handler objects will always have the 'when' value set to None, > and the comparison functions don't really apply, moreover __eq__ would even > be incorrect in its current form. > > IMHO it would be good to have a base Handler class which doesn't take time > into account and then a Timer class which does. Thoughts? > > Also, I see that the PEP does mention the possibility of providing a > __call__ method on the Handler class itself, which would basically call the > callback and suppress exceptions (I guess by logging them). Was this ever > discussed? Sounds like a good idea to me :-) > > > Regards, > > -- > Sa?l Ibarra Corretg? > http://saghul.net/blog | http://about.me/saghul > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas -- --Guido van Rossum (python.org/~guido) From saghul at gmail.com Tue Feb 5 17:20:15 2013 From: saghul at gmail.com (=?ISO-8859-1?Q?Sa=FAl_Ibarra_Corretg=E9?=) Date: Tue, 05 Feb 2013 17:20:15 +0100 Subject: [Python-ideas] PEP 3156 / Tulip: Remove 'when' from Handler In-Reply-To: References: <51111BCB.6070609@gmail.com> Message-ID: <5111313F.6010708@gmail.com> Guido van Rossum wrote: > I think both suggestions are good ones. The when field has bothered me > for a while, although it did not come to the surface. Adding the > call/catch/log logic on the Handler seems nice too, but I think I > would rather not use handler() -- I would rather use handler.call(). > I.e. make the method name 'call', not '__call__'. (The latter almost > always makes APIs more confusing IMO.) > > Can you contribute code? > Sure, I've never used the codereview thing nor mercurial , but I guess this a good time to start :-) -- Sa?l Ibarra Corretg? http://saghul.net/blog | http://about.me/saghul From guido at python.org Tue Feb 5 17:53:05 2013 From: guido at python.org (Guido van Rossum) Date: Tue, 5 Feb 2013 08:53:05 -0800 Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface In-Reply-To: References: Message-ID: On Mon, Feb 4, 2013 at 8:50 PM, Ben Darnell wrote: > Fair enough. I may use this pattern when/if I retrofit Tornado to be > IOCP-friendly since we currently create transports with static functions, > but for Tulip let's wait and see if the problem this proposal is trying to > solve ever materializes. Great! -- --Guido van Rossum (python.org/~guido) From p.f.moore at gmail.com Tue Feb 5 21:32:24 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 5 Feb 2013 20:32:24 +0000 Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface In-Reply-To: References: Message-ID: On 5 February 2013 04:50, Ben Darnell wrote: >> Still, it's a pretty weak argument IMO -- I don't expect there to be a >> significant cottage industry cranking out 3rd party protocol >> implementations, assuming we add UDP to PEP 3156, which is my plan. > > Yeah, as long as we get TCP, UDP, SSL, and pipes (at least for > subprocesses), I'm hard pressed to imagine anything else that would be in so > much demand it would need to be supported by all event loops. There are two thing that come to *my* mind whenever this sort of debate comes up. I'll freely admit that they are 100% theoretical in terms of my actual requirements, but neither is particular implausible. - Synchronisation primitlives like Windows event objects - wanting to integrate code to be run when an event is set into an event loop seems relatively reasonable. - GUI input events - I don't know about Unix, but Windows GUI events are a separate notification stream from network or pipe data, and it's very plausible that someone would want to integrate GUI events into an async app. Twisted, for instance, has GUI event loop integration facilities, I believe. (I know that "write your own event loop" is always a solution. But I'm not sure that doesn't set the bar a bit too high - it depends on how easy it is to reuse and extend the implementation of the standard event loops, which is something I'm not really clear on yet). Paul From guido at python.org Tue Feb 5 22:28:11 2013 From: guido at python.org (Guido van Rossum) Date: Tue, 5 Feb 2013 13:28:11 -0800 Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface In-Reply-To: References: Message-ID: On Tue, Feb 5, 2013 at 12:32 PM, Paul Moore wrote: > On 5 February 2013 04:50, Ben Darnell wrote: >>> Still, it's a pretty weak argument IMO -- I don't expect there to be a >>> significant cottage industry cranking out 3rd party protocol >>> implementations, assuming we add UDP to PEP 3156, which is my plan. >> >> Yeah, as long as we get TCP, UDP, SSL, and pipes (at least for >> subprocesses), I'm hard pressed to imagine anything else that would be in so >> much demand it would need to be supported by all event loops. > > There are two thing that come to *my* mind whenever this sort of > debate comes up. I'll freely admit that they are 100% theoretical in > terms of my actual requirements, but neither is particular > implausible. > > - Synchronisation primitlives like Windows event objects - wanting to > integrate code to be run when an event is set into an event loop seems > relatively reasonable. Since this is about interacting with the threading world, you can always wrap that in a concurrent.futures.Future, and then wrap that in eventloop.wrap_future(). > - GUI input events - I don't know about Unix, but Windows GUI events > are a separate notification stream from network or pipe data, and it's > very plausible that someone would want to integrate GUI events into an > async app. Twisted, for instance, has GUI event loop integration > facilities, I believe. That's way too large a topic to try to anticipate without thorough research. And it's not very likely that you can do this in a portable way either -- the best you can probably hope for is have a PEP 3156-compliant event loop that lets you use portable async networking APIs (transports and protocols) while also letting you write platform-*specific* GUI code. The best route here will probably be the PEP 3156 bridge that Twisted is going to develop once the PEP and Twisted's Python 3 port stabilize. (Unfortunately, wxPython is not ported to Python 3.) > (I know that "write your own event loop" is always a solution. But I'm > not sure that doesn't set the bar a bit too high - it depends on how > easy it is to reuse and extend the implementation of the standard > event loops, which is something I'm not really clear on yet). Check the most recent changes that landed in Tulip. The EventLoop class has been refactored to bits and pieces, with exactly that purpose in mind. (The PEP is a bit behind ATM, and I just broke the ProactorEventLoop's ability to act as a server, but I'm on that.) -- --Guido van Rossum (python.org/~guido) From p.f.moore at gmail.com Tue Feb 5 23:08:02 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 5 Feb 2013 22:08:02 +0000 Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface In-Reply-To: References: Message-ID: Thanks for the responses. I think you're right on both counts - in particular, I'd forgotten the ability to wrap futures for the event case. On 5 February 2013 21:28, Guido van Rossum wrote: > Check the most recent changes that landed in Tulip. The EventLoop > class has been refactored to bits and pieces, with exactly that > purpose in mind. (The PEP is a bit behind ATM, and I just broke the > ProactorEventLoop's ability to act as a server, but I'm on that.) I'll try to get a chance to look. I haven't really checked the code for a while, as I've been busy. Did you ever look at my subprocess code? It was in a bitbucket repo rather than reitveld, my apologies if that was a problem, I can investigate how to use reitveld if needed. Paul From saghul at gmail.com Tue Feb 5 23:35:19 2013 From: saghul at gmail.com (=?ISO-8859-1?Q?Sa=FAl_Ibarra_Corretg=E9?=) Date: Tue, 05 Feb 2013 23:35:19 +0100 Subject: [Python-ideas] libuv based eventloop for tulip experiment In-Reply-To: <51070056.8020006@gmail.com> References: <51070056.8020006@gmail.com> Message-ID: <51118927.3060506@gmail.com> Hi again, I just updated rose [0] to match latest changes in Tulip API and remove Tulip itself from the code, now a proper EventLoopPolicy is defined, which will in turn use the pyuv-based event loop. Regards, [0]: https://github.com/saghul/rose -- Sa?l Ibarra Corretg? http://saghul.net/blog | http://about.me/saghul From guido at python.org Tue Feb 5 23:39:30 2013 From: guido at python.org (Guido van Rossum) Date: Tue, 5 Feb 2013 14:39:30 -0800 Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface In-Reply-To: References: Message-ID: On Tue, Feb 5, 2013 at 2:08 PM, Paul Moore wrote: > Did you ever look at my subprocess > code? It was in a bitbucket repo rather than reitveld, my apologies if > that was a problem, I can investigate how to use reitveld if needed. I glanced at it, but didn't really review it in depth. Do you think it's ready for integration? I do like the idea of using as much from the subprocess module as possible. We also need to look at supporting the same API on Windows (since subprocess supports it). I have access to a Windows box now. If you think you have something that's ready to integrate (even if just UNIX), please do use Rietveld (codereview.appspot.com). The best strategy is to leave your changes uncommitted in a current checkout of Tulip, and run the upload.py script that you can download here: https://codereview.appspot.com/static/upload.py and which is documented here: http://code.google.com/p/rietveld/wiki/UploadPyUsage (You are already a PSF contributed, right?) -- --Guido van Rossum (python.org/~guido) From guido at python.org Tue Feb 5 23:58:02 2013 From: guido at python.org (Guido van Rossum) Date: Tue, 5 Feb 2013 14:58:02 -0800 Subject: [Python-ideas] libuv based eventloop for tulip experiment In-Reply-To: <51118927.3060506@gmail.com> References: <51070056.8020006@gmail.com> <51118927.3060506@gmail.com> Message-ID: On Tue, Feb 5, 2013 at 2:35 PM, Sa?l Ibarra Corretg? wrote: > I just updated rose [0] to match latest changes in Tulip API and remove > Tulip itself from the code, now a proper EventLoopPolicy is defined, which > will in turn use the pyuv-based event loop. Neat! I wonder if you could see how many of Tulip's own tests pass with your event loop implementation? > [0]: https://github.com/saghul/rose > > > -- > Sa?l Ibarra Corretg? > http://saghul.net/blog | http://about.me/saghul > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas -- --Guido van Rossum (python.org/~guido) From fafhrd91 at gmail.com Wed Feb 6 00:10:58 2013 From: fafhrd91 at gmail.com (Nikolay Kim) Date: Tue, 5 Feb 2013 15:10:58 -0800 Subject: [Python-ideas] libuv based eventloop for tulip experiment In-Reply-To: References: <51070056.8020006@gmail.com> <51118927.3060506@gmail.com> Message-ID: I'll try to run gunicorn worker as well On Feb 5, 2013, at 2:58 PM, Guido van Rossum wrote: > On Tue, Feb 5, 2013 at 2:35 PM, Sa?l Ibarra Corretg? wrote: >> I just updated rose [0] to match latest changes in Tulip API and remove >> Tulip itself from the code, now a proper EventLoopPolicy is defined, which >> will in turn use the pyuv-based event loop. > > Neat! > > I wonder if you could see how many of Tulip's own tests pass with your > event loop implementation? > >> [0]: https://github.com/saghul/rose >> >> >> -- >> Sa?l Ibarra Corretg? >> http://saghul.net/blog | http://about.me/saghul >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> http://mail.python.org/mailman/listinfo/python-ideas > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas From fafhrd91 at gmail.com Wed Feb 6 00:32:02 2013 From: fafhrd91 at gmail.com (Nikolay Kim) Date: Tue, 5 Feb 2013 15:32:02 -0800 Subject: [Python-ideas] libuv based eventloop for tulip experiment In-Reply-To: References: <51070056.8020006@gmail.com> <51118927.3060506@gmail.com> Message-ID: <381AA99C-E174-44BD-A1B1-92C1DB678960@gmail.com> On Feb 5, 2013, at 2:58 PM, Guido van Rossum wrote: > On Tue, Feb 5, 2013 at 2:35 PM, Sa?l Ibarra Corretg? wrote: >> I just updated rose [0] to match latest changes in Tulip API and remove >> Tulip itself from the code, now a proper EventLoopPolicy is defined, which >> will in turn use the pyuv-based event loop. > > Neat! > > I wonder if you could see how many of Tulip's own tests pass with your > event loop implementation? > two tests are failing. i think thats very good result. ====================================================================== FAIL: test_start_serving_cant_bind (tulip.events_test.UVEventLoopTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/unittest/mock.py", line 1088, in patched return func(*args, **keywargs) File "/Users/nikolay/dev/tulip/src/tulip/tulip/events_test.py", line 497, in test_start_serving_cant_bind self.assertRaises(Err, self.event_loop.run_until_complete, fut) AssertionError: Err not raised by run_until_complete ====================================================================== FAIL: test_baseexception_during_cancel (tulip.tasks_test.TaskTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/nikolay/dev/tulip/src/tulip/tulip/tasks_test.py", line 451, in test_baseexception_during_cancel self.assertRaises(BaseException, self.event_loop.run_once) AssertionError: BaseException not raised by run_once >> [0]: https://github.com/saghul/rose >> >> >> -- >> Sa?l Ibarra Corretg? >> http://saghul.net/blog | http://about.me/saghul >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> http://mail.python.org/mailman/listinfo/python-ideas > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas From p.f.moore at gmail.com Wed Feb 6 00:56:18 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 5 Feb 2013 23:56:18 +0000 Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface In-Reply-To: References: Message-ID: On 5 February 2013 22:39, Guido van Rossum wrote: > On Tue, Feb 5, 2013 at 2:08 PM, Paul Moore wrote: >> Did you ever look at my subprocess >> code? It was in a bitbucket repo rather than reitveld, my apologies if >> that was a problem, I can investigate how to use reitveld if needed. > > I glanced at it, but didn't really review it in depth. Do you think > it's ready for integration? I do like the idea of using as much from > the subprocess module as possible. We also need to look at supporting > the same API on Windows (since subprocess supports it). I have access > to a Windows box now. Functionally, I'm happy with the patch (on Unix, it needs something for Windows but I don't know IOCP very well, so I'm not sure how much I can do there, ironically). I'm a bit concerned with the usability of the API in the coroutine style (it's fine using callbacks, which I'm comfortable with, but I'm struggling to get my head round coroutines, so I've no intuition as to whether it feels natural in that style). > If you think you have something that's ready to integrate (even if > just UNIX), please do use Rietveld (codereview.appspot.com). The best > strategy is to > leave your changes uncommitted in a current checkout of Tulip, and run > the upload.py script that you > can download here: https://codereview.appspot.com/static/upload.py > and which is documented here: > http://code.google.com/p/rietveld/wiki/UploadPyUsage OK, I'll give it a try - probably not for a week or two, as I'm away a lot at the moment... > (You are already a PSF contributed, right?) TBH, I'm not entirely sure. I *should* be, as there's code in Python from me, but I don't recall ever sending in a form. I'll send a new form in just to be certain. Paul. From saghul at gmail.com Wed Feb 6 00:56:51 2013 From: saghul at gmail.com (=?ISO-8859-1?Q?Sa=FAl_Ibarra_Corretg=E9?=) Date: Wed, 06 Feb 2013 00:56:51 +0100 Subject: [Python-ideas] libuv based eventloop for tulip experiment In-Reply-To: <381AA99C-E174-44BD-A1B1-92C1DB678960@gmail.com> References: <51070056.8020006@gmail.com> <51118927.3060506@gmail.com> <381AA99C-E174-44BD-A1B1-92C1DB678960@gmail.com> Message-ID: <51119C43.9060604@gmail.com> Nikolay Kim wrote: > On Feb 5, 2013, at 2:58 PM, Guido van Rossum wrote: > >> On Tue, Feb 5, 2013 at 2:35 PM, Sa?l Ibarra Corretg? wrote: >>> I just updated rose [0] to match latest changes in Tulip API and remove >>> Tulip itself from the code, now a proper EventLoopPolicy is defined, which >>> will in turn use the pyuv-based event loop. >> Neat! >> >> I wonder if you could see how many of Tulip's own tests pass with your >> event loop implementation? >> Hum, they all pass for me on OSX. Did you use the runtests.py script from rose? > > two tests are failing. i think thats very good result. > > ====================================================================== > FAIL: test_start_serving_cant_bind (tulip.events_test.UVEventLoopTests) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/unittest/mock.py", line 1088, in patched > return func(*args, **keywargs) > File "/Users/nikolay/dev/tulip/src/tulip/tulip/events_test.py", line 497, in test_start_serving_cant_bind > self.assertRaises(Err, self.event_loop.run_until_complete, fut) > AssertionError: Err not raised by run_until_complete > I overwrote this test in rose/events_test.py > ====================================================================== > FAIL: test_baseexception_during_cancel (tulip.tasks_test.TaskTests) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/Users/nikolay/dev/tulip/src/tulip/tulip/tasks_test.py", line 451, in test_baseexception_during_cancel > self.assertRaises(BaseException, self.event_loop.run_once) > AssertionError: BaseException not raised by run_once > Hum, I have not seen this one. -- Sa?l Ibarra Corretg? http://saghul.net/blog | http://about.me/saghul From fafhrd91 at gmail.com Wed Feb 6 01:30:37 2013 From: fafhrd91 at gmail.com (Nikolay Kim) Date: Tue, 5 Feb 2013 16:30:37 -0800 Subject: [Python-ideas] libuv based eventloop for tulip experiment In-Reply-To: <51119C43.9060604@gmail.com> References: <51070056.8020006@gmail.com> <51118927.3060506@gmail.com> <381AA99C-E174-44BD-A1B1-92C1DB678960@gmail.com> <51119C43.9060604@gmail.com> Message-ID: <051B29A4-9023-4778-9B0B-976E1633676C@gmail.com> i used uv_events as default event loop and ran all tulip tests with it. but those failures are not related to uv_events anyway I'd inherit uv_events.EventLoop from base_events.BaseEventLoop. On Feb 5, 2013, at 3:56 PM, Sa?l Ibarra Corretg? wrote: > Nikolay Kim wrote: >> On Feb 5, 2013, at 2:58 PM, Guido van Rossum wrote: >> >>> On Tue, Feb 5, 2013 at 2:35 PM, Sa?l Ibarra Corretg? wrote: >>>> I just updated rose [0] to match latest changes in Tulip API and remove >>>> Tulip itself from the code, now a proper EventLoopPolicy is defined, which >>>> will in turn use the pyuv-based event loop. >>> Neat! >>> >>> I wonder if you could see how many of Tulip's own tests pass with your >>> event loop implementation? >>> > > Hum, they all pass for me on OSX. Did you use the runtests.py script from rose? > >> >> two tests are failing. i think thats very good result. >> >> ====================================================================== >> FAIL: test_start_serving_cant_bind (tulip.events_test.UVEventLoopTests) >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/unittest/mock.py", line 1088, in patched >> return func(*args, **keywargs) >> File "/Users/nikolay/dev/tulip/src/tulip/tulip/events_test.py", line 497, in test_start_serving_cant_bind >> self.assertRaises(Err, self.event_loop.run_until_complete, fut) >> AssertionError: Err not raised by run_until_complete >> > > I overwrote this test in rose/events_test.py > >> ====================================================================== >> FAIL: test_baseexception_during_cancel (tulip.tasks_test.TaskTests) >> ---------------------------------------------------------------------- >> Traceback (most recent call last): >> File "/Users/nikolay/dev/tulip/src/tulip/tulip/tasks_test.py", line 451, in test_baseexception_during_cancel >> self.assertRaises(BaseException, self.event_loop.run_once) >> AssertionError: BaseException not raised by run_once >> > > Hum, I have not seen this one. > > > -- > Sa?l Ibarra Corretg? > http://saghul.net/blog | http://about.me/saghul From tjreedy at udel.edu Wed Feb 6 01:30:16 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 05 Feb 2013 19:30:16 -0500 Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface In-Reply-To: References: Message-ID: On 2/5/2013 6:56 PM, Paul Moore wrote: > On 5 February 2013 22:39, Guido van Rossum wrote: >> (You are already a PSF contributed, right?) > > TBH, I'm not entirely sure. I *should* be, as there's code in Python > from me, but I don't recall ever sending in a form. I'll send a new > form in just to be certain. This is now recorded in user records on the tracker. As for you, Paul, http://bugs.python.org/user301 indicates that there is no current record. If you ever did send a paper form, it got lost a few years ago like mine and many others. You can now send a scan by email. http://www.python.org/psf/contrib/ People with contributor agreements are also indicated by '*' after their name in issue messages. Other past or prospective contributors who are not sure of their recorded contributor status can check either way and send a form if needed. -- Terry Jan Reedy From saghul at gmail.com Wed Feb 6 01:45:39 2013 From: saghul at gmail.com (=?ISO-8859-1?Q?Sa=FAl_Ibarra_Corretg=E9?=) Date: Wed, 06 Feb 2013 01:45:39 +0100 Subject: [Python-ideas] libuv based eventloop for tulip experiment In-Reply-To: <051B29A4-9023-4778-9B0B-976E1633676C@gmail.com> References: <51070056.8020006@gmail.com> <51118927.3060506@gmail.com> <381AA99C-E174-44BD-A1B1-92C1DB678960@gmail.com> <51119C43.9060604@gmail.com> <051B29A4-9023-4778-9B0B-976E1633676C@gmail.com> Message-ID: <5111A7B3.7050504@gmail.com> Nikolay Kim wrote: > i used uv_events as default event loop and ran all tulip tests with it. but those failures are not related to uv_events anyway > I'd inherit uv_events.EventLoop from base_events.BaseEventLoop. > Why? There is not much code that can be reused, since timers are not needed and the _run_once function is also not applicable. -- Sa?l Ibarra Corretg? http://saghul.net/blog | http://about.me/saghul From guido at python.org Wed Feb 6 02:04:16 2013 From: guido at python.org (Guido van Rossum) Date: Tue, 5 Feb 2013 17:04:16 -0800 Subject: [Python-ideas] libuv based eventloop for tulip experiment In-Reply-To: <5111A7B3.7050504@gmail.com> References: <51070056.8020006@gmail.com> <51118927.3060506@gmail.com> <381AA99C-E174-44BD-A1B1-92C1DB678960@gmail.com> <51119C43.9060604@gmail.com> <051B29A4-9023-4778-9B0B-976E1633676C@gmail.com> <5111A7B3.7050504@gmail.com> Message-ID: On Tue, Feb 5, 2013 at 4:45 PM, Sa?l Ibarra Corretg? wrote: > Nikolay Kim wrote: >> >> i used uv_events as default event loop and ran all tulip tests with it. >> but those failures are not related to uv_events anyway >> I'd inherit uv_events.EventLoop from base_events.BaseEventLoop. >> > > Why? There is not much code that can be reused, since timers are not needed > and the _run_once function is also not applicable. Because the test mocks base_events.socket. :-) That's not a great reason, I agree. Perhaps you and Nikolay can work on a better way to do the mocking so it doesn't rely on base_events? -- --Guido van Rossum (python.org/~guido) From ubershmekel at gmail.com Wed Feb 6 20:19:10 2013 From: ubershmekel at gmail.com (Yuval Greenfield) Date: Wed, 6 Feb 2013 21:19:10 +0200 Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface In-Reply-To: References: Message-ID: Any others wanting to toy with tulip on windows can have the _overlapped.pyd I painfully made for python3.3-32bit. https://www.dropbox.com/s/20ljdyafpe25ekh/_overlapped.pyd BTW I needed to require windows vista for it to compile... #define _WIN32_WINNT 0x0600 #define NTDDI_VERSION 0x06000000 Yuval Greenfield -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Wed Feb 6 20:29:56 2013 From: guido at python.org (Guido van Rossum) Date: Wed, 6 Feb 2013 11:29:56 -0800 Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface In-Reply-To: References: Message-ID: Great! Are you keeping that up to date? Richard is checking changes at a furious pace. :-) On Wed, Feb 6, 2013 at 11:19 AM, Yuval Greenfield wrote: > Any others wanting to toy with tulip on windows can have the _overlapped.pyd > I painfully made for python3.3-32bit. > > https://www.dropbox.com/s/20ljdyafpe25ekh/_overlapped.pyd > > BTW I needed to require windows vista for it to compile... > > #define _WIN32_WINNT 0x0600 > #define NTDDI_VERSION 0x06000000 > > > Yuval Greenfield > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > -- --Guido van Rossum (python.org/~guido) From ubershmekel at gmail.com Wed Feb 6 22:23:13 2013 From: ubershmekel at gmail.com (Yuval Greenfield) Date: Wed, 6 Feb 2013 23:23:13 +0200 Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface In-Reply-To: References: Message-ID: On Wed, Feb 6, 2013 at 9:29 PM, Guido van Rossum wrote: > Great! Are you keeping that up to date? Richard is checking changes at > a furious pace. :-) > > I updated the file so the same link has the new pyd that is up to date as of http://code.google.com/p/tulip/source/detail?r=65c456e2c20ece3adabc6d5f37d78957e9aeaf18- Fix return type of SetFromWindowsErr(). E:\Dropbox\dev\python\tulip>c:\python33\python runtests.py ....s......s............s.............sss......ss.....sss...s..........sss...... .................................................... ---------------------------------------------------------------------- Ran 132 tests in 8.513s OK (skipped=15) Strangely, the first time I run the tests I get a big pile of output exceptions though the tests do pass. E.g. .........sss......ss.ERROR:root:Exception in task Traceback (most recent call last): File "E:\Dropbox\dev\python\tulip\tulip\tasks.py", line 96, in _step result = coro.send(value) File "E:\Dropbox\dev\python\tulip\tulip\base_events.py", line 235, in create_c onnection raise exceptions[0] File "E:\Dropbox\dev\python\tulip\tulip\base_events.py", line 226, in create_c onnection yield self.sock_connect(sock, address) File "c:\python33\lib\unittest\mock.py", line 846, in __call__ return _mock_self._mock_call(*args, **kwargs) File "c:\python33\lib\unittest\mock.py", line 901, in _mock_call raise effect OSError .ERROR:root:Exception in task [...] Indeed, it's hard to keep up. -------------- next part -------------- An HTML attachment was scrubbed... URL: From shibturn at gmail.com Wed Feb 6 22:48:37 2013 From: shibturn at gmail.com (Richard Oudkerk) Date: Wed, 06 Feb 2013 21:48:37 +0000 Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface In-Reply-To: References: Message-ID: On 06/02/2013 9:23pm, Yuval Greenfield wrote: > Strangely, the first time I run the tests I get a big pile of output > exceptions though the tests do pass. E.g. > > .........sss......ss.ERROR:root:Exception in task > Traceback (most recent call last): > File "E:\Dropbox\dev\python\tulip\tulip\tasks.py", line 96, in _step > result = coro.send(value) > File "E:\Dropbox\dev\python\tulip\tulip\base_events.py", line 235, in > create_c > onnection > raise exceptions[0] > File "E:\Dropbox\dev\python\tulip\tulip\base_events.py", line 226, in > create_c > onnection > yield self.sock_connect(sock, address) > File "c:\python33\lib\unittest\mock.py", line 846, in __call__ > return _mock_self._mock_call(*args, **kwargs) > File "c:\python33\lib\unittest\mock.py", line 901, in _mock_call > raise effect > OSError > .ERROR:root:Exception in task > [...] I see this sometimes too. It seems that these are expected errors caused by using self.assertRaises(...). Why these errors are logged only sometimes I don't understand. -- Richard From p.f.moore at gmail.com Wed Feb 6 23:20:17 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 6 Feb 2013 22:20:17 +0000 Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface In-Reply-To: References: Message-ID: On 6 February 2013 21:48, Richard Oudkerk wrote: > On 06/02/2013 9:23pm, Yuval Greenfield wrote: >> >> Strangely, the first time I run the tests I get a big pile of output >> exceptions though the tests do pass. E.g. I just tried a build on Windows 7 64-bit with Python 3.3 (i.e., not a 3.4 checkout). I got a lot of these types of error as well. I also got a couple of genuine ones. One was a build problem - PY_ULONG_MAX doesn't exist. I changed it to ULONG_MAX which got the compile to work - but it may not be correct, I guess. The other was: ERROR: test_sock_accept (tulip.events_test.ProactorEventLoopTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Work\Scratch\tulip\tulip\events_test.py", line 309, in test_sock_accept conn, addr = self.event_loop.run_until_complete(f) File "C:\Work\Scratch\tulip\tulip\base_events.py", line 104, in run_until_complete return future.result() # May raise future.exception(). File "C:\Work\Scratch\tulip\tulip\futures.py", line 148, in result raise self._exception File "C:\Work\Scratch\tulip\tulip\windows_events.py", line 132, in _poll value = callback() File "C:\Work\Scratch\tulip\tulip\windows_events.py", line 83, in finish_accept listener.fileno()) OSError: [WinError 10014] The system detected an invalid pointer address in attempting to use a pointer argument in a call Interestingly, I saw the same OSError occurring in some of the "ERROR:root:Accept failed" logging stuff. I don't know if that's of any use... Paul From amauryfa at gmail.com Wed Feb 6 23:37:06 2013 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Wed, 6 Feb 2013 23:37:06 +0100 Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface In-Reply-To: References: Message-ID: 2013/2/5 Guido van Rossum > > - GUI input events - I don't know about Unix, but Windows GUI events > > are a separate notification stream from network or pipe data, and it's > > very plausible that someone would want to integrate GUI events into an > > async app. Twisted, for instance, has GUI event loop integration > > facilities, I believe. > > That's way too large a topic to try to anticipate without thorough > research. And it's not very likely that you can do this in a portable > way either -- the best you can probably hope for is have a PEP > 3156-compliant event loop that lets you use portable async networking > APIs (transports and protocols) while also letting you write > platform-*specific* GUI code. The best route here will probably be the > PEP 3156 bridge that Twisted is going to develop once the PEP and > Twisted's Python 3 port stabilize. (Unfortunately, wxPython is not > ported to Python 3.) That's not exactly true: http://wiki.wxpython.org/ProjectPhoenix#Links It's still a work in progress, but the basic interface is unlikely to change. -- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Wed Feb 6 23:43:54 2013 From: guido at python.org (Guido van Rossum) Date: Wed, 6 Feb 2013 14:43:54 -0800 Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface In-Reply-To: References: Message-ID: On Wed, Feb 6, 2013 at 1:48 PM, Richard Oudkerk wrote: > On 06/02/2013 9:23pm, Yuval Greenfield wrote: >> >> Strangely, the first time I run the tests I get a big pile of output >> exceptions though the tests do pass. E.g. >> >> .........sss......ss.ERROR:root:Exception in task >> Traceback (most recent call last): >> File "E:\Dropbox\dev\python\tulip\tulip\tasks.py", line 96, in _step >> result = coro.send(value) >> File "E:\Dropbox\dev\python\tulip\tulip\base_events.py", line 235, in >> create_c >> onnection >> raise exceptions[0] >> File "E:\Dropbox\dev\python\tulip\tulip\base_events.py", line 226, in >> create_c >> onnection >> yield self.sock_connect(sock, address) >> File "c:\python33\lib\unittest\mock.py", line 846, in __call__ >> return _mock_self._mock_call(*args, **kwargs) >> File "c:\python33\lib\unittest\mock.py", line 901, in _mock_call >> raise effect >> OSError >> .ERROR:root:Exception in task >> [...] > > > I see this sometimes too. > > It seems that these are expected errors caused by using > self.assertRaises(...). Why these errors are logged only sometimes I don't > understand. Me neither. :-( The message "Exception in task" means that it is a task that raises an exception. I used to ignore these; now I log them always, but ideally they should only be logged when whoever waits for the Task doesn't catch them (or, better, when nobody waits for the task). I tried to implement that part but I couldn't get it to work (yet) -- I will have to get back to this at some point, because accurate exception logging (never silent, but not too spammy either) is very important for a good user experience. But it remains a mystery why they sometimes show and not other times. It suggests there's some indeterminate timing in some tests. If it happens only the first time when the tests are run this usually points to a timing behavior that's different when the source code is parsed as opposed to read from a .pyc file. -- --Guido van Rossum (python.org/~guido) From guido at python.org Wed Feb 6 23:44:41 2013 From: guido at python.org (Guido van Rossum) Date: Wed, 6 Feb 2013 14:44:41 -0800 Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface In-Reply-To: References: Message-ID: On Wed, Feb 6, 2013 at 2:37 PM, Amaury Forgeot d'Arc wrote: > 2013/2/5 Guido van Rossum >> (Unfortunately, wxPython is not >> ported to Python 3.) > That's not exactly true: http://wiki.wxpython.org/ProjectPhoenix#Links > It's still a work in progress, but the basic interface is unlikely to > change. Glad to hear it! -- --Guido van Rossum (python.org/~guido) From shibturn at gmail.com Thu Feb 7 00:58:03 2013 From: shibturn at gmail.com (Richard Oudkerk) Date: Wed, 06 Feb 2013 23:58:03 +0000 Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface In-Reply-To: References: Message-ID: On 06/02/2013 10:43pm, Guido van Rossum wrote: > Me neither.:-( > > The message "Exception in task" means that it is a task that raises an > exception. I used to ignore these; now I log them always, but ideally > they should only be logged when whoever waits for the Task doesn't > catch them (or, better, when nobody waits for the task). I tried to > implement that part but I couldn't get it to work (yet) -- I will have > to get back to this at some point, because accurate exception logging > (never silent, but not too spammy either) is very important for a good > user experience. > > But it remains a mystery why they sometimes show and not other times. > It suggests there's some indeterminate timing in some tests. If it > happens only the first time when the tests are run this usually points > to a timing behavior that's different when the source code is parsed > as opposed to read from a .pyc file. Commenting out the one use of suppress_log_errors() makes the "expected errors" appear on Linux too. But I would have thought that that would only effect the test which uses suppress_log_errors(). diff -r 65c456e2c20e tulip/events_test.py --- a/tulip/events_test.py Wed Feb 06 19:08:14 2013 +0000 +++ b/tulip/events_test.py Wed Feb 06 23:51:32 2013 +0000 @@ -508,7 +508,7 @@ self.assertFalse(sock.close.called) def test_accept_connection_exception(self): - self.suppress_log_errors() + #self.suppress_log_errors() sock = unittest.mock.Mock() sock.accept.side_effect = socket.error -- Richard From guido at python.org Thu Feb 7 01:11:37 2013 From: guido at python.org (Guido van Rossum) Date: Wed, 6 Feb 2013 16:11:37 -0800 Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface In-Reply-To: References: Message-ID: On Wed, Feb 6, 2013 at 3:58 PM, Richard Oudkerk wrote: > On 06/02/2013 10:43pm, Guido van Rossum wrote: >> >> Me neither.:-( >> >> The message "Exception in task" means that it is a task that raises an >> exception. I used to ignore these; now I log them always, but ideally >> they should only be logged when whoever waits for the Task doesn't >> catch them (or, better, when nobody waits for the task). I tried to >> implement that part but I couldn't get it to work (yet) -- I will have >> to get back to this at some point, because accurate exception logging >> (never silent, but not too spammy either) is very important for a good >> user experience. >> >> But it remains a mystery why they sometimes show and not other times. >> It suggests there's some indeterminate timing in some tests. If it >> happens only the first time when the tests are run this usually points >> to a timing behavior that's different when the source code is parsed >> as opposed to read from a .pyc file. > > > Commenting out the one use of suppress_log_errors() makes the "expected > errors" appear on Linux too. But I would have thought that that would only > effect the test which uses suppress_log_errors(). > > diff -r 65c456e2c20e tulip/events_test.py > --- a/tulip/events_test.py Wed Feb 06 19:08:14 2013 +0000 > +++ b/tulip/events_test.py Wed Feb 06 23:51:32 2013 +0000 > @@ -508,7 +508,7 @@ > self.assertFalse(sock.close.called) > > def test_accept_connection_exception(self): > - self.suppress_log_errors() > + #self.suppress_log_errors() > > sock = unittest.mock.Mock() > sock.accept.side_effect = socket.error Good catch! What's going on is that the super().tearDown() call was missing from EventLoopTestsMixin. I'll fix that -- then we'll have to separately add suppress_log_errors() calls to various tests (I'll do that at a slower pace). -- --Guido van Rossum (python.org/~guido) From guido at python.org Thu Feb 7 01:21:25 2013 From: guido at python.org (Guido van Rossum) Date: Wed, 6 Feb 2013 16:21:25 -0800 Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface In-Reply-To: References: Message-ID: On Wed, Feb 6, 2013 at 4:11 PM, Guido van Rossum wrote: > On Wed, Feb 6, 2013 at 3:58 PM, Richard Oudkerk wrote: >> On 06/02/2013 10:43pm, Guido van Rossum wrote: >>> >>> Me neither.:-( >>> >>> The message "Exception in task" means that it is a task that raises an >>> exception. I used to ignore these; now I log them always, but ideally >>> they should only be logged when whoever waits for the Task doesn't >>> catch them (or, better, when nobody waits for the task). I tried to >>> implement that part but I couldn't get it to work (yet) -- I will have >>> to get back to this at some point, because accurate exception logging >>> (never silent, but not too spammy either) is very important for a good >>> user experience. >>> >>> But it remains a mystery why they sometimes show and not other times. >>> It suggests there's some indeterminate timing in some tests. If it >>> happens only the first time when the tests are run this usually points >>> to a timing behavior that's different when the source code is parsed >>> as opposed to read from a .pyc file. >> >> >> Commenting out the one use of suppress_log_errors() makes the "expected >> errors" appear on Linux too. But I would have thought that that would only >> effect the test which uses suppress_log_errors(). >> >> diff -r 65c456e2c20e tulip/events_test.py >> --- a/tulip/events_test.py Wed Feb 06 19:08:14 2013 +0000 >> +++ b/tulip/events_test.py Wed Feb 06 23:51:32 2013 +0000 >> @@ -508,7 +508,7 @@ >> self.assertFalse(sock.close.called) >> >> def test_accept_connection_exception(self): >> - self.suppress_log_errors() >> + #self.suppress_log_errors() >> >> sock = unittest.mock.Mock() >> sock.accept.side_effect = socket.error > > Good catch! What's going on is that the super().tearDown() call was > missing from EventLoopTestsMixin. I'll fix that -- then we'll have to > separately add suppress_log_errors() calls to various tests (I'll do > that at a slower pace). Should all be fixed now. (Please check on Windows, I can't check it right now.) -- --Guido van Rossum (python.org/~guido) From ubershmekel at gmail.com Thu Feb 7 01:32:00 2013 From: ubershmekel at gmail.com (Yuval Greenfield) Date: Thu, 7 Feb 2013 02:32:00 +0200 Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface In-Reply-To: References: Message-ID: On Thu, Feb 7, 2013 at 2:21 AM, Guido van Rossum wrote: > Should all be fixed now. (Please check on Windows, I can't check it right > now.) > > Removed the pyc's and ran on windows a few times. There was no strange exception output. Seems to be fixed. Yuval -------------- next part -------------- An HTML attachment was scrubbed... URL: From shibturn at gmail.com Thu Feb 7 01:33:38 2013 From: shibturn at gmail.com (Richard Oudkerk) Date: Thu, 07 Feb 2013 00:33:38 +0000 Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface In-Reply-To: References: Message-ID: On 06/02/2013 10:20pm, Paul Moore wrote: > I just tried a build on Windows 7 64-bit with Python 3.3 (i.e., not a > 3.4 checkout). I got a lot of these types of error as well. I also got > a couple of genuine ones. One was a build problem - PY_ULONG_MAX > doesn't exist. I changed it to ULONG_MAX which got the compile to work > - but it may not be correct, I guess. The other was: > > ERROR: test_sock_accept (tulip.events_test.ProactorEventLoopTests) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Work\Scratch\tulip\tulip\events_test.py", line 309, in > test_sock_accept > conn, addr = self.event_loop.run_until_complete(f) > File "C:\Work\Scratch\tulip\tulip\base_events.py", line 104, in > run_until_complete > return future.result() # May raise future.exception(). > File "C:\Work\Scratch\tulip\tulip\futures.py", line 148, in result > raise self._exception > File "C:\Work\Scratch\tulip\tulip\windows_events.py", line 132, in _poll > value = callback() > File "C:\Work\Scratch\tulip\tulip\windows_events.py", line 83, in > finish_accept > listener.fileno()) > OSError: [WinError 10014] The system detected an invalid pointer > address in attempting to use a pointer argument in a call > > Interestingly, I saw the same OSError occurring in some of the > "ERROR:root:Accept failed" logging stuff. > > I don't know if that's of any use... Thanks. It should be fixed now, but I have not tested it on a 64 bit build. (My 64 bit setup seems to be all screwed up.) -- Richard From p.f.moore at gmail.com Thu Feb 7 11:13:39 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 7 Feb 2013 10:13:39 +0000 Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface In-Reply-To: References: Message-ID: I can test it when I get home... On Thursday, 7 February 2013, Richard Oudkerk wrote: > On 06/02/2013 10:20pm, Paul Moore wrote: > >> I just tried a build on Windows 7 64-bit with Python 3.3 (i.e., not a >> 3.4 checkout). I got a lot of these types of error as well. I also got >> a couple of genuine ones. One was a build problem - PY_ULONG_MAX >> doesn't exist. I changed it to ULONG_MAX which got the compile to work >> - but it may not be correct, I guess. The other was: >> >> ERROR: test_sock_accept (tulip.events_test.**ProactorEventLoopTests) >> ------------------------------**------------------------------** >> ---------- >> Traceback (most recent call last): >> File "C:\Work\Scratch\tulip\tulip\**events_test.py", line 309, in >> test_sock_accept >> conn, addr = self.event_loop.run_until_**complete(f) >> File "C:\Work\Scratch\tulip\tulip\**base_events.py", line 104, in >> run_until_complete >> return future.result() # May raise future.exception(). >> File "C:\Work\Scratch\tulip\tulip\**futures.py", line 148, in result >> raise self._exception >> File "C:\Work\Scratch\tulip\tulip\**windows_events.py", line 132, in >> _poll >> value = callback() >> File "C:\Work\Scratch\tulip\tulip\**windows_events.py", line 83, in >> finish_accept >> listener.fileno()) >> OSError: [WinError 10014] The system detected an invalid pointer >> address in attempting to use a pointer argument in a call >> >> Interestingly, I saw the same OSError occurring in some of the >> "ERROR:root:Accept failed" logging stuff. >> >> I don't know if that's of any use... >> > > Thanks. > > It should be fixed now, but I have not tested it on a 64 bit build. (My > 64 bit setup seems to be all screwed up.) > > -- > Richard > > ______________________________**_________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/**mailman/listinfo/python-ideas > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Thu Feb 7 18:44:54 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 7 Feb 2013 17:44:54 +0000 Subject: [Python-ideas] PEP 3156/Tulip: Extensible EventLoop interface In-Reply-To: References: Message-ID: On 7 February 2013 10:13, Paul Moore wrote: > On Thursday, 7 February 2013, Richard Oudkerk wrote: >> >> It should be fixed now, but I have not tested it on a 64 bit build. (My >> 64 bit setup seems to be all screwed up.) > > I can test it when I get home... Yep, works fine now on 64-bit Windows 7. Ran 132 tests in 9.048s OK (skipped=15) No errors, no spurious logging of exceptions. Thanks! Paul From fafhrd91 at gmail.com Fri Feb 8 01:18:14 2013 From: fafhrd91 at gmail.com (Nikolay Kim) Date: Thu, 7 Feb 2013 16:18:14 -0800 Subject: [Python-ideas] PEP 3156 / Tulip: Synchronization primitives Message-ID: I think Tulip should have synchronization primitives by default. Here is my motivation: 1. it is more convenient to use locking primitive with existing semantics rather than tulip magic. crawl.py example could use semaphore instead of tasks.wait with timeout 0. 2. while it seems easy to implement semaphore with tulip, it still requires deep understanding of tulips control flow. i had 2 not very obvious bugs in my semaphore implementation: a) release() could be called from different co-routines during same scheduling step. b) bug with acquire and release during same scheduling step. example: 1. task tries to acquire locked semaphore 2. other task releases semaphore in result - semaphore gets acquired 2 times at the end of scheduling step, because actual semaphore acquisition happens in "call_soon" and at this stage release() call is already released semaphore but first waiter does not acquire it yet. Thoughts? my implementation: https://codereview.appspot.com/download/issue7230045_15001.diff From saghul at gmail.com Fri Feb 8 01:24:35 2013 From: saghul at gmail.com (=?ISO-8859-1?Q?Sa=FAl_Ibarra_Corretg=E9?=) Date: Fri, 08 Feb 2013 01:24:35 +0100 Subject: [Python-ideas] libuv based eventloop for tulip experiment In-Reply-To: References: <51070056.8020006@gmail.com> <51118927.3060506@gmail.com> <381AA99C-E174-44BD-A1B1-92C1DB678960@gmail.com> <51119C43.9060604@gmail.com> <051B29A4-9023-4778-9B0B-976E1633676C@gmail.com> <5111A7B3.7050504@gmail.com> Message-ID: <511445C3.8080601@gmail.com> Guido van Rossum wrote: > On Tue, Feb 5, 2013 at 4:45 PM, Sa?l Ibarra Corretg? wrote: >> Nikolay Kim wrote: >>> i used uv_events as default event loop and ran all tulip tests with it. >>> but those failures are not related to uv_events anyway >>> I'd inherit uv_events.EventLoop from base_events.BaseEventLoop. >>> >> Why? There is not much code that can be reused, since timers are not needed >> and the _run_once function is also not applicable. > > Because the test mocks base_events.socket. :-) That's not a great > reason, I agree. Perhaps you and Nikolay can work on a better way to > do the mocking so it doesn't rely on base_events? > I see. I modified it to inherit from BaseEventLoop now. I found a bug in the way rose executes handler callbacks (https://github.com/saghul/rose/commit/94e984cf756c3d0730acf804c18cab682cffd6d6) I need to think how to implement this, probably saving the first base exception, stopping the processing and reraising it... -- Sa?l Ibarra Corretg? http://saghul.net/blog | http://about.me/saghul From gotoalanlu at gmail.com Fri Feb 8 03:34:30 2013 From: gotoalanlu at gmail.com (Hua Lu) Date: Thu, 7 Feb 2013 20:34:30 -0600 Subject: [Python-ideas] frozenset literals In-Reply-To: References: Message-ID: Hey all, I have a simple hack around this problem for the time being. It involves adding a parameter to ast.literal_eval. See this diff: $ diff ast.py.bak ast.py 38c38 < def literal_eval(node_or_string): --- > def literal_eval(node_or_string, use_frozensets=False): 49a50 > set_t = frozenset if use_frozensets else set 60c61 < return set(map(_convert, node.elts)) --- > return set_t(map(_convert, node.elts)) Use is as follows: >>> import ast >>> ast.literal_eval("{1,2,3}") {1, 2, 3} >>> ast.literal_eval("{1,2,3}", use_frozensets=True) frozenset({1, 2, 3}) >>> ast.literal_eval("{{1,2,3}: 'foo'}", use_frozensets=True) {frozenset({1, 2, 3}): 'foo'} Regards, Alan From solipsis at pitrou.net Fri Feb 8 10:06:11 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 8 Feb 2013 10:06:11 +0100 Subject: [Python-ideas] PEP 3156 / Tulip: Synchronization primitives References: Message-ID: <20130208100611.1d7c53fd@pitrou.net> Le Thu, 7 Feb 2013 16:18:14 -0800, Nikolay Kim a ?crit : > I think Tulip should have synchronization primitives by default. > Here is my motivation: > > 1. it is more convenient to use locking primitive with existing > semantics rather than tulip magic. crawl.py example could use > semaphore instead of tasks.wait with timeout 0. 2. while it seems > easy to implement semaphore with tulip, it still requires deep > understanding of tulips control flow. i had 2 not very obvious bugs > in my semaphore implementation: > > a) release() could be called from different co-routines during > same scheduling step. b) bug with acquire and release during same > scheduling step. example: > 1. task tries to acquire locked semaphore > 2. other task releases semaphore > > in result - semaphore gets acquired 2 times at the end of > scheduling step, because actual semaphore acquisition happens in > "call_soon" and at this stage release() call is already released > semaphore but first waiter does not acquire it yet. I think Twisted has DeferredSemaphore for the same reasons. Regards Antoine. From guido at python.org Fri Feb 8 18:44:33 2013 From: guido at python.org (Guido van Rossum) Date: Fri, 8 Feb 2013 09:44:33 -0800 Subject: [Python-ideas] PEP 3156 / Tulip: Synchronization primitives In-Reply-To: <20130208100611.1d7c53fd@pitrou.net> References: <20130208100611.1d7c53fd@pitrou.net> Message-ID: On Fri, Feb 8, 2013 at 1:06 AM, Antoine Pitrou wrote: > Le Thu, 7 Feb 2013 16:18:14 -0800, > Nikolay Kim a ?crit : >> I think Tulip should have synchronization primitives by default. >> Here is my motivation: >> >> 1. it is more convenient to use locking primitive with existing >> semantics rather than tulip magic. crawl.py example could use >> semaphore instead of tasks.wait with timeout 0. 2. while it seems >> easy to implement semaphore with tulip, it still requires deep >> understanding of tulips control flow. i had 2 not very obvious bugs >> in my semaphore implementation: >> >> a) release() could be called from different co-routines during >> same scheduling step. b) bug with acquire and release during same >> scheduling step. example: >> 1. task tries to acquire locked semaphore >> 2. other task releases semaphore >> >> in result - semaphore gets acquired 2 times at the end of >> scheduling step, because actual semaphore acquisition happens in >> "call_soon" and at this stage release() call is already released >> semaphore but first waiter does not acquire it yet. > > I think Twisted has DeferredSemaphore for the same reasons. So, is anyone against adding Nikolay's locks implementation to PEP 3156? It's a pretty simple API, and looks similar to the locks in threading.py except for a sprinkling of "yield from". It does not depend on Tulip internals. -- --Guido van Rossum (python.org/~guido) From ncoghlan at gmail.com Sat Feb 9 04:26:56 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 9 Feb 2013 13:26:56 +1000 Subject: [Python-ideas] PEP 3156 / Tulip: Synchronization primitives In-Reply-To: References: <20130208100611.1d7c53fd@pitrou.net> Message-ID: On Sat, Feb 9, 2013 at 3:44 AM, Guido van Rossum wrote: > On Fri, Feb 8, 2013 at 1:06 AM, Antoine Pitrou wrote: >> I think Twisted has DeferredSemaphore for the same reasons. > > So, is anyone against adding Nikolay's locks implementation to PEP > 3156? It's a pretty simple API, and looks similar to the locks in > threading.py except for a sprinkling of "yield from". It does not > depend on Tulip internals. Common need + easy to get the implementation details wrong = sounds like a good candidate for inclusion to me. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From timothy.c.delaney at gmail.com Sun Feb 10 23:43:45 2013 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Mon, 11 Feb 2013 09:43:45 +1100 Subject: [Python-ideas] constant/enum type in stdlib In-Reply-To: References: <51085AAB.6090303@canterbury.ac.nz> <5108A87D.9000207@canterbury.ac.nz> <20130130082639.0b28d7eb@pitrou.net> <20130201103656.0b24c40d@anarchist.wooz.org> Message-ID: On 4 February 2013 11:17, Tim Delaney wrote: > On 4 February 2013 10:53, Jo?o Bernardo wrote: > >> Hi, about this enum/const thing, The use case I like more is a class >> where you know all the >> instances and not just a sequence of names. >> Particularly It would be nice to have custom attributes and methods >> besides the value and the name. >> >> I have my own implementation with a basic api somewhat borrowed from >> flufl.enum (plus a lot of other stuff), >> but with this kind of support: https://github.com/jbvsmo/makeobj >> > > I considered it, and in fact you could almost do it with my implementation > by using a custom subclass of EnumValue (except trying it has just exposed > a design flaw with the whole _EnumProxy bit). Works if you create the enum > in the same module as EnumValues, fails otherwise. Going to have to have a > rethink. > Fixed the _EnumProxy issue (but it's a kludge - I've used sys._getframe() - there's probably a better way). I've also made it so that you can override the metaclass to return a subclass of EnumValue. Now you can do something like: Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:57:17) [MSC v.1600 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> from enum import Enum, EnumValue, EnumMeta >>> >>> class MyEnumValue1(EnumValue): ... pass ... >>> class MyEnumMeta1(EnumMeta): ... @classmethod ... def _create_value(cls, key, value): ... return MyEnumValue1(key, value) ... >>> class MyEnum1(Enum, metaclass=MyEnumMeta1): ... VALUE1, ... VALUE2 ... >>> class MyEnumValue2(EnumValue): ... pass ... >>> class MyEnumMeta2(MyEnumMeta1): ... @classmethod ... def _create_value(cls, key, value): ... return MyEnumValue2(key, value) ... >>> class MyEnum2(MyEnum1, metaclass=MyEnumMeta2): ... VALUE3, ... VALUE4 ... >>> print(repr(MyEnum1)) , }> >>> print(repr(MyEnum2)) , , , }> >>> Tim Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: From timothy.c.delaney at gmail.com Mon Feb 11 07:59:55 2013 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Mon, 11 Feb 2013 17:59:55 +1100 Subject: [Python-ideas] constant/enum type in stdlib In-Reply-To: References: <51085AAB.6090303@canterbury.ac.nz> <5108A87D.9000207@canterbury.ac.nz> <20130130082639.0b28d7eb@pitrou.net> <20130201103656.0b24c40d@anarchist.wooz.org> Message-ID: On 11 February 2013 09:43, Tim Delaney wrote: > I've also made it so that you can override the metaclass to return a > subclass of EnumValue. > Expanded on this a bit to simplify things. Now to use a different type you can just go: Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:57:17) [MSC v.1600 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> from enum import Enum, EnumValue >>> >>> class MyEnumValue1(EnumValue): ... pass ... >>> class MyEnum1(Enum, metaclass=Enum.subtype(MyEnumValue1)): ... VALUE1, ... VALUE2 ... >>> class MyEnumValue2(EnumValue): ... pass ... >>> class MyEnum2(MyEnum1, metaclass=MyEnum1.subtype(MyEnumValue2)): ... VALUE3, ... VALUE4 ... >>> print(repr(MyEnum1)) , }> >>> print(repr(MyEnum2)) , , , }> The parameter passed to subtype() can be any callable that takes 2 parameters (key, value) and returns an instance of EnumValue. Unfortunately, I can't see any way to avoid specifying the base class twice (once as the base class, once for the metaclass=). Similarly, passing the 'value_type' parameter to Enum.make() creates an appropriate metaclass from the base class: Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:57:17) [MSC v.1600 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> from enum import Enum, EnumValue >>> class MyEnumValue1(EnumValue): ... pass ... >>> MyEnum1 = Enum.make('MyEnum1', ('VALUE1', 'VALUE2'), value_type=MyEnumValue1) >>> print(repr(MyEnum1)) , }> Whilst the original "use a different metaclass to produce values of a different type" was possibly feature-creep, these modifications at least make it pretty painless to use. Overall, I'm pretty happy with where this is now, except for the use of sys._getframe(). Tim Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: From saghul at gmail.com Mon Feb 11 13:21:44 2013 From: saghul at gmail.com (=?ISO-8859-1?Q?Sa=FAl_Ibarra_Corretg=E9?=) Date: Mon, 11 Feb 2013 13:21:44 +0100 Subject: [Python-ideas] PEP 3156 / Tulip: stopping the loop while in run_until_complete Message-ID: <5118E258.90705@gmail.com> Hi, While analyzing some code I came across the following situation: - User calls run_until_complete for some Task - The task itself calls loop.stop() at some point - run_until_complete raises TimeoutError Here is a very simple example: https://gist.github.com/saghul/4754117 Something seems a bit off here. While one could argue that stopping the loop while run_until_complete is ongoing, getting a TimeoutError doesn't feel right. I think we should detect that the loop was stopped but the future is not done and raise something like NotCompleted or FutureNotCompleted. Thoughts? -- Sa?l Ibarra Corretg? http://saghul.net/blog | http://about.me/saghul From eliben at gmail.com Mon Feb 11 14:28:45 2013 From: eliben at gmail.com (Eli Bendersky) Date: Mon, 11 Feb 2013 05:28:45 -0800 Subject: [Python-ideas] constant/enum type in stdlib In-Reply-To: References: <51085AAB.6090303@canterbury.ac.nz> <5108A87D.9000207@canterbury.ac.nz> <20130130082639.0b28d7eb@pitrou.net> <20130201103656.0b24c40d@anarchist.wooz.org> Message-ID: On Sun, Feb 10, 2013 at 2:43 PM, Tim Delaney wrote: > On 4 February 2013 11:17, Tim Delaney wrote: > >> On 4 February 2013 10:53, Jo?o Bernardo wrote: >> >>> Hi, about this enum/const thing, The use case I like more is a class >>> where you know all the >>> instances and not just a sequence of names. >>> Particularly It would be nice to have custom attributes and methods >>> besides the value and the name. >>> >>> I have my own implementation with a basic api somewhat borrowed from >>> flufl.enum (plus a lot of other stuff), >>> but with this kind of support: https://github.com/jbvsmo/makeobj >>> >> >> I considered it, and in fact you could almost do it with my >> implementation by using a custom subclass of EnumValue (except trying it >> has just exposed a design flaw with the whole _EnumProxy bit). Works if you >> create the enum in the same module as EnumValues, fails otherwise. Going to >> have to have a rethink. >> > > Fixed the _EnumProxy issue (but it's a kludge - I've used sys._getframe() > - there's probably a better way). I've also made it so that you can > override the metaclass to return a subclass of EnumValue. > > Now you can do something like: > > Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:57:17) [MSC v.1600 64 > bit (AMD64)] on win32 > Type "help", "copyright", "credits" or "license" for more information. > >>> from enum import Enum, EnumValue, EnumMeta > >>> > >>> class MyEnumValue1(EnumValue): > ... pass > ... > >>> class MyEnumMeta1(EnumMeta): > ... @classmethod > ... def _create_value(cls, key, value): > ... return MyEnumValue1(key, value) > ... > >>> class MyEnum1(Enum, metaclass=MyEnumMeta1): > ... VALUE1, > ... VALUE2 > ... > >>> class MyEnumValue2(EnumValue): > ... pass > ... > >>> class MyEnumMeta2(MyEnumMeta1): > ... @classmethod > ... def _create_value(cls, key, value): > ... return MyEnumValue2(key, value) > ... > >>> class MyEnum2(MyEnum1, metaclass=MyEnumMeta2): > ... VALUE3, > ... VALUE4 > ... > >>> print(repr(MyEnum1)) > , > }> > >>> print(repr(MyEnum2)) > , > , , > }> > >>> > Can you elaborate on the utility of this feature? What realistic use cases do you see for it? I think that at this point it's important to weigh all benefits of features vs. implementation complexity, and there's absolutely no need to support every feature every other enum implementation has. I want to stress again that the most important characteristic of your implementation is the clean syntax which means that enums are so easy to define they don't really need special Python syntax and a library feature can do. However, there's a big leap from this to defining custom metaclasses for enums. Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Feb 11 17:06:16 2013 From: guido at python.org (Guido van Rossum) Date: Mon, 11 Feb 2013 08:06:16 -0800 Subject: [Python-ideas] PEP 3156 / Tulip: stopping the loop while in run_until_complete In-Reply-To: <5118E258.90705@gmail.com> References: <5118E258.90705@gmail.com> Message-ID: Yeah, I think the base implementation of run_until_complete is wrong. I think I didn't have stop() when I coded that. Could you change it to something that just registers a lambda calling stop() on the Future, and see if that makes your test case behave better? (If so, please add a unit test and submit for code review.) On Mon, Feb 11, 2013 at 4:21 AM, Sa?l Ibarra Corretg? wrote: > Hi, > > While analyzing some code I came across the following situation: > > - User calls run_until_complete for some Task > - The task itself calls loop.stop() at some point > - run_until_complete raises TimeoutError > > Here is a very simple example: https://gist.github.com/saghul/4754117 > > Something seems a bit off here. While one could argue that stopping the loop > while run_until_complete is ongoing, getting a TimeoutError doesn't feel > right. I think we should detect that the loop was stopped but the future is > not done and raise something like NotCompleted or FutureNotCompleted. > > Thoughts? > > -- > Sa?l Ibarra Corretg? > http://saghul.net/blog | http://about.me/saghul > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas -- --Guido van Rossum (python.org/~guido) From saghul at gmail.com Mon Feb 11 19:18:18 2013 From: saghul at gmail.com (=?ISO-8859-1?Q?Sa=FAl_Ibarra_Corretg=E9?=) Date: Mon, 11 Feb 2013 19:18:18 +0100 Subject: [Python-ideas] PEP 3156 / Tulip: stopping the loop while in run_until_complete In-Reply-To: References: <5118E258.90705@gmail.com> Message-ID: <511935EA.3040201@gmail.com> Guido van Rossum wrote: > Yeah, I think the base implementation of run_until_complete is wrong. > I think I didn't have stop() when I coded that. > > Could you change it to something that just registers a lambda calling > stop() on the Future, and see if that makes your test case behave > better? (If so, please add a unit test and submit for code review.) > Hum, I guess I'm missing something, but futures don't have a stop() method. The code already calls loop.stop() when the future is done. What I had in mind was to add a _called to Handler and expose it with a readonly property, so then we could check if handler.called and raise TimeoutError or not. As for raising another exception, maybe it's not such a good idea, since caller can always check future.done() later anyway. -- Sa?l Ibarra Corretg? http://saghul.net/blog | http://about.me/saghul From guido at python.org Mon Feb 11 19:26:58 2013 From: guido at python.org (Guido van Rossum) Date: Mon, 11 Feb 2013 10:26:58 -0800 Subject: [Python-ideas] PEP 3156 / Tulip: stopping the loop while in run_until_complete In-Reply-To: <511935EA.3040201@gmail.com> References: <5118E258.90705@gmail.com> <511935EA.3040201@gmail.com> Message-ID: On Mon, Feb 11, 2013 at 10:18 AM, Sa?l Ibarra Corretg? wrote: > Guido van Rossum wrote: > >> Yeah, I think the base implementation of run_until_complete is wrong. >> I think I didn't have stop() when I coded that. >> >> Could you change it to something that just registers a lambda calling >> stop() on the Future, and see if that makes your test case behave >> better? (If so, please add a unit test and submit for code review.) >> >> > Hum, I guess I'm missing something, but futures don't have a stop() > method. The code already calls loop.stop() when the future is done. > Oops, I read your post (and the code) too fast. > What I had in mind was to add a _called to Handler and expose it with a > readonly property, so then we could check if handler.called and raise > TimeoutError or not. As for raising another exception, maybe it's not such > a good idea, since caller can always check future.done() later anyway. > I'm not so keen on that. But you were probably calling run_until_complete() without an explicit timeout. In that case, it should not call run() but run_forever(), and it should never raise TimeoutError. You could still get a TimeoutError if a timeout was given; in that case, I think you can fix this case by passing call_later() a helper function that sets a nonlocal variable which you then inspect. You could also use a different mechanism, e.g. call cancel() on a Future when the timeout occurs. (But I think that might be less reliable, since I think Tasks can catch cancellations.) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From saghul at gmail.com Mon Feb 11 20:29:28 2013 From: saghul at gmail.com (=?UTF-8?Q?Sa=C3=BAl_Ibarra_Corretg=C3=A9?=) Date: Mon, 11 Feb 2013 20:29:28 +0100 Subject: [Python-ideas] PEP 3156 / Tulip: stopping the loop while in run_until_complete In-Reply-To: References: <5118E258.90705@gmail.com> <511935EA.3040201@gmail.com> Message-ID: > But you were probably calling run_until_complete() without an explicit > timeout. In that case, it should not call run() but run_forever(), and it > should never raise TimeoutError. > Right, this was actually the case. I'll submit a code review addressing this. > You could still get a TimeoutError if a timeout was given; in that case, I > think you can fix this case by passing call_later() a helper function that > sets a nonlocal variable which you then inspect. > Sure, but then what should we do: return like nothing has happened or raise an exception so that the caller knows that the future is not complete? > You could also use a different mechanism, e.g. call cancel() on a Future > when the timeout occurs. (But I think that might be less reliable, since I > think Tasks can catch cancellations.) > Yeah, that would not work, in my example the task is already started, I don't think cancel() will work midway. -- /Sa?l http://saghul.net | http://sipdoc.net From guido at python.org Mon Feb 11 20:37:39 2013 From: guido at python.org (Guido van Rossum) Date: Mon, 11 Feb 2013 11:37:39 -0800 Subject: [Python-ideas] PEP 3156 / Tulip: stopping the loop while in run_until_complete In-Reply-To: References: <5118E258.90705@gmail.com> <511935EA.3040201@gmail.com> Message-ID: On Mon, Feb 11, 2013 at 11:29 AM, Sa?l Ibarra Corretg? wrote: > > But you were probably calling run_until_complete() without an explicit > > timeout. In that case, it should not call run() but run_forever(), and it > > should never raise TimeoutError. > > > > Right, this was actually the case. I'll submit a code review addressing > this. > Cool. > > You could still get a TimeoutError if a timeout was given; in that > case, I > > think you can fix this case by passing call_later() a helper function > that > > sets a nonlocal variable which you then inspect. > > Sure, but then what should we do: return like nothing has happened or > raise an exception so that the caller knows that the future is not > complete? > The same thing as when no timeout is specified. I presume this means returning None. The alternative would be to call future.result() in all cases -- it would raise InvalidStateError, but that doesn't feel very useful. Perhaps this is actually the correct response? What is the reason you have something else that calls stop()? > > You could also use a different mechanism, e.g. call cancel() on a Future > > when the timeout occurs. (But I think that might be less reliable, since > I > > think Tasks can catch cancellations.) > > Yeah, that would not work, in my example the task is already started, > I don't think cancel() will work midway. > Actually, it may. There is code to throw CancelledError into the generator, in _step(). But I don't think it will necessarily break out of all blocking I/O. (Though it really should -- perhaps this is something useful to add tests for.) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From timothy.c.delaney at gmail.com Mon Feb 11 21:09:41 2013 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Tue, 12 Feb 2013 07:09:41 +1100 Subject: [Python-ideas] constant/enum type in stdlib In-Reply-To: References: <51085AAB.6090303@canterbury.ac.nz> <5108A87D.9000207@canterbury.ac.nz> <20130130082639.0b28d7eb@pitrou.net> <20130201103656.0b24c40d@anarchist.wooz.org> Message-ID: On 12 February 2013 00:28, Eli Bendersky wrote: > On Sun, Feb 10, 2013 at 2:43 PM, Tim Delaney wrote: > >> On 4 February 2013 11:17, Tim Delaney wrote: >> >>> On 4 February 2013 10:53, Jo?o Bernardo wrote: >>> >>>> Particularly It would be nice to have custom attributes and methods >>>> besides the value and the name. >>>> >>> >> I've also made it so that you can override the metaclass to return a >> subclass of EnumValue. >> >> Can you elaborate on the utility of this feature? What realistic use > cases do you see for it? I think that at this point it's important to weigh > all benefits of features vs. implementation complexity, and there's > absolutely no need to support every feature every other enum implementation > has. I want to stress again that the most important characteristic of your > implementation is the clean syntax which means that enums are so easy to > define they don't really need special Python syntax and a library feature > can do. However, there's a big leap from this to defining custom > metaclasses for enums. > The custom metaclass is purely a mechanism to make it easy to use a custom class for the enum value. It wold be possible to manually assign a custom enum type to each enum, but then you wold lose the ability to just define the names. By using a custom metaclass, you can have it automatically assign the enum value type that you want. My next email specifies a simplified syntax for specifying the custom metaclass (you don't need to create one at all). Supporting this functionality was actually very simple. However, I am wondering though how useful this is without being able to specify additional parameters for the enums. I've often used enums with quite complex behaviour (e.g. in java) - the enum part is purely to have a unique value assigned to each instance of the class and not repeating myself for no good reason. I couldn't quite do the same with this as it currently is - whilst I can have whatever behaviour I want, there's nothing to key it off except the name and value which probably isn't enough. I've been trying to think of a syntax which would work to pass additional parameters and I can't think of anything cleaner than having a specialised class to pass the additional parameters - but I need to somehow be able to specify the optional integer value of the enum. Maybe have the first parameter be None? class EnumParams(): def __init__(self, value=None, *p, **kw): self.value = value .... def _resolve_proxies(self): # names would have been affected by EnumValues.__getattr__ - need to resolve them from _EnumProxy to the actual values ... class MyEnumValue(EnumValue): def __new__(cls, key, value, *p): e = super().__new__(cls, key, value) e.p = p def dump(self): print(self.p) class MyEnum(Enum, metaclass=Enum.subtype(MyEnumValue)): A = EnumParams(None, 'extra', 'params') B = EnumParams(3, 'more', 'params') Thoughts? Is the extra complexity worth it? The thing is, this doesn't take away from the ability to specify the very simple clean enums - but it would give the enums pretty much the full capabilities of java enums. I'd like to have all these features available so that any PEP could reference them and discuss the pros and cons (including how well they work in practice). Tim Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: From timothy.c.delaney at gmail.com Mon Feb 11 21:11:35 2013 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Tue, 12 Feb 2013 07:11:35 +1100 Subject: [Python-ideas] constant/enum type in stdlib In-Reply-To: References: <51085AAB.6090303@canterbury.ac.nz> <5108A87D.9000207@canterbury.ac.nz> <20130130082639.0b28d7eb@pitrou.net> <20130201103656.0b24c40d@anarchist.wooz.org> Message-ID: And I've just realised that my enums will be broken when assigning a literal containing a name lookup e.g. v = 1 class MyEnum(Enum): A, B other_attr = (v,) other_attr will have an _EnumProxy instance. But I think I can do away with the _EnumProxy entirely now that I'm using sys._getframe(). Let me try something. Tim Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: From timothy.c.delaney at gmail.com Mon Feb 11 21:22:03 2013 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Tue, 12 Feb 2013 07:22:03 +1100 Subject: [Python-ideas] constant/enum type in stdlib In-Reply-To: References: <51085AAB.6090303@canterbury.ac.nz> <5108A87D.9000207@canterbury.ac.nz> <20130130082639.0b28d7eb@pitrou.net> <20130201103656.0b24c40d@anarchist.wooz.org> Message-ID: On 12 February 2013 07:11, Tim Delaney wrote: > And I've just realised that my enums will be broken when assigning a > literal containing a name lookup e.g. > > v = 1 > > class MyEnum(Enum): > A, B > other_attr = (v,) > > other_attr will have an _EnumProxy instance. But I think I can do away > with the _EnumProxy entirely now that I'm using sys._getframe(). Let me try > something. > Too early in the morning. The above would avoid the whole _EnumProxy because 'v' would be successfully looked up in the global namespace. But I may still get rid of some instances of _EnumProxy. Tim Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: From saghul at gmail.com Mon Feb 11 21:53:45 2013 From: saghul at gmail.com (=?ISO-8859-1?Q?Sa=FAl_Ibarra_Corretg=E9?=) Date: Mon, 11 Feb 2013 21:53:45 +0100 Subject: [Python-ideas] PEP 3156 / Tulip: stopping the loop while in run_until_complete In-Reply-To: References: <5118E258.90705@gmail.com> <511935EA.3040201@gmail.com> Message-ID: <51195A59.1020500@gmail.com> > > The same thing as when no timeout is specified. I presume this means > returning None. The alternative would be to call future.result() in all > cases -- it would raise InvalidStateError, but that doesn't feel very > useful. Perhaps this is actually the correct response? What is the > reason you have something else that calls stop()? > I didn't run into this in practice, I just thought of it :-) I guess that some exception handler or some third party lib could do it without you knowing, but I don't have a good example on why someone would want to do it. > > Yeah, that would not work, in my example the task is already started, > I don't think cancel() will work midway. > > Actually, it may. There is code to throw CancelledError into the > generator, in _step(). But I don't think it will necessarily break out > of all blocking I/O. (Though it really should -- perhaps this is > something useful to add tests for.) > Well, it may work for tasks, but since run_until_complete applies to futures in general I'm not sure if that would still work. I added the following review which fixes the problem and adds a test case for it: https://codereview.appspot.com/7301076/ A few days ago I also submitted this one: https://codereview.appspot.com/7312046/ which adds a few more tests for run_until_complete. Regards, -- Sa?l Ibarra Corretg? http://saghul.net/blog | http://about.me/saghul From guido at python.org Mon Feb 11 22:42:13 2013 From: guido at python.org (Guido van Rossum) Date: Mon, 11 Feb 2013 13:42:13 -0800 Subject: [Python-ideas] PEP 3156 / Tulip: stopping the loop while in run_until_complete In-Reply-To: <51195A59.1020500@gmail.com> References: <5118E258.90705@gmail.com> <511935EA.3040201@gmail.com> <51195A59.1020500@gmail.com> Message-ID: On Mon, Feb 11, 2013 at 12:53 PM, Sa?l Ibarra Corretg? wrote: > > >> The same thing as when no timeout is specified. I presume this means >> returning None. The alternative would be to call future.result() in all >> cases -- it would raise InvalidStateError, but that doesn't feel very >> useful. Perhaps this is actually the correct response? What is the >> reason you have something else that calls stop()? >> >> > I didn't run into this in practice, I just thought of it :-) I guess that > some exception handler or some third party lib could do it without you > knowing, but I don't have a good example on why someone would want to do it. > I think there are actually two cases: where stop() is called from the task given to run_until_complete(), or from another task. In the former case, because the exception bubbles out of the task's coroutine, the task will be marked as done (and its exception set to _StopError) -- at least I think so. In the latter case, the task will still be runnable, and it will continue to run when the event loop is started again. (Hm... maybe run_forever() should be called start(), to match stop()?) I think your unittest only tests the former case. I'm not sure what run_until_complete() should return in the latter case. In fact it makes me doubt the behavior of run_until_complete() in general. Maybe it should return the Future (although that's kind of redundant)? Or an enum indicating what happened? (Cases would be result, exception, timeout, running -- and maybe cancelled?) > > >> Yeah, that would not work, in my example the task is already started, >> I don't think cancel() will work midway. >> >> Actually, it may. There is code to throw CancelledError into the >> generator, in _step(). But I don't think it will necessarily break out >> of all blocking I/O. (Though it really should -- perhaps this is >> something useful to add tests for.) >> >> > Well, it may work for tasks, but since run_until_complete applies to > futures in general I'm not sure if that would still work. > > I added the following review which fixes the problem and adds a test case > for it: https://codereview.appspot.com/7301076/ A few days ago I also > submitted this one: https://codereview.appspot.com/7312046/ which adds a > few more tests for run_until_complete. > Approved the latter. Gave some detailed feedback on the former; but see my higher-order mumblings above... -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From saghul at gmail.com Mon Feb 11 23:23:35 2013 From: saghul at gmail.com (=?ISO-8859-1?Q?Sa=FAl_Ibarra_Corretg=E9?=) Date: Mon, 11 Feb 2013 23:23:35 +0100 Subject: [Python-ideas] PEP 3156 / Tulip: stopping the loop while in run_until_complete In-Reply-To: References: <5118E258.90705@gmail.com> <511935EA.3040201@gmail.com> <51195A59.1020500@gmail.com> Message-ID: <51196F67.6080609@gmail.com> > > I think there are actually two cases: where stop() is called from the > task given to run_until_complete(), or from another task. In the former > case, because the exception bubbles out of the task's coroutine, the > task will be marked as done (and its exception set to _StopError) -- at > least I think so. In the latter case, the task will still be runnable, > and it will continue to run when the event loop is started again. (Hm... > maybe run_forever() should be called start(), to match stop()?) > > I think your unittest only tests the former case. > Well, I think they are actually the same. In the gist I posted earlier I called loop.stop(), I didn't raise the exception inside the task, and loop.stop() will call call_soon so the actual raising will happen in the context on _run_once unless I'm mistaken. Here is an updated simple test with it's output: https://gist.github.com/saghul/4758151 > I'm not sure what run_until_complete() should return in the latter case. > In fact it makes me doubt the behavior of run_until_complete() in > general. Maybe it should return the Future (although that's kind of > redundant)? Or an enum indicating what happened? (Cases would be result, > exception, timeout, running -- and maybe cancelled?) > I think the current behavior is ok, it the user really wants to check if the future was actually completed she can check with done(), if she saved the future in a variable, that is. My other preferred approach would be to raise an exception, since I consider this an exceptional case: I told the loop to run until this future is completed so I expect that to happen. Since it didn't, NotCompletedError (or something alike) sounds good to me. -- Sa?l Ibarra Corretg? http://saghul.net/blog | http://about.me/saghul From guido at python.org Mon Feb 11 23:31:34 2013 From: guido at python.org (Guido van Rossum) Date: Mon, 11 Feb 2013 14:31:34 -0800 Subject: [Python-ideas] PEP 3156 / Tulip: stopping the loop while in run_until_complete In-Reply-To: <51196F67.6080609@gmail.com> References: <5118E258.90705@gmail.com> <511935EA.3040201@gmail.com> <51195A59.1020500@gmail.com> <51196F67.6080609@gmail.com> Message-ID: On Mon, Feb 11, 2013 at 2:23 PM, Sa?l Ibarra Corretg? wrote: > > >> I think there are actually two cases: where stop() is called from the >> task given to run_until_complete(), or from another task. In the former >> case, because the exception bubbles out of the task's coroutine, the >> task will be marked as done (and its exception set to _StopError) -- at >> least I think so. In the latter case, the task will still be runnable, >> and it will continue to run when the event loop is started again. (Hm... >> maybe run_forever() should be called start(), to match stop()?) >> >> I think your unittest only tests the former case. >> >> > Well, I think they are actually the same. In the gist I posted earlier I > called loop.stop(), I didn't raise the exception inside the task, and > loop.stop() will call call_soon so the actual raising will happen in the > context on _run_once unless I'm mistaken. > Oh, you're right! Not my day for reading carefully. :-) > Here is an updated simple test with it's output: > https://gist.github.com/saghul/4758151 > > > I'm not sure what run_until_complete() should return in the latter case. >> In fact it makes me doubt the behavior of run_until_complete() in >> general. Maybe it should return the Future (although that's kind of >> redundant)? Or an enum indicating what happened? (Cases would be result, >> exception, timeout, running -- and maybe cancelled?) >> >> > I think the current behavior is ok, it the user really wants to check if > the future was actually completed she can check with done(), if she saved > the future in a variable, that is. > > My other preferred approach would be to raise an exception, since I > consider this an exceptional case: I told the loop to run until this future > is completed so I expect that to happen. Since it didn't, NotCompletedError > (or something alike) sounds good to me. > Good. How about calling future.result(), and letting it raise InvalidStateError ? (Except if the timeout handler was called.) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From saghul at gmail.com Mon Feb 11 23:35:39 2013 From: saghul at gmail.com (=?ISO-8859-1?Q?Sa=FAl_Ibarra_Corretg=E9?=) Date: Mon, 11 Feb 2013 23:35:39 +0100 Subject: [Python-ideas] PEP 3156 / Tulip: stopping the loop while in run_until_complete In-Reply-To: References: <5118E258.90705@gmail.com> <511935EA.3040201@gmail.com> <51195A59.1020500@gmail.com> <51196F67.6080609@gmail.com> Message-ID: <5119723B.8060703@gmail.com> > > Good. How about calling future.result(), and letting it raise > InvalidStateError ? (Except if the timeout handler was called.) > I did a quick test to see how it feels and I do like it. I'll update the review with this change plus the other suggested ones. -- Sa?l Ibarra Corretg? http://saghul.net/blog | http://about.me/saghul From guido at python.org Mon Feb 11 23:36:52 2013 From: guido at python.org (Guido van Rossum) Date: Mon, 11 Feb 2013 14:36:52 -0800 Subject: [Python-ideas] PEP 3156 / Tulip: stopping the loop while in run_until_complete In-Reply-To: <5119723B.8060703@gmail.com> References: <5118E258.90705@gmail.com> <511935EA.3040201@gmail.com> <51195A59.1020500@gmail.com> <51196F67.6080609@gmail.com> <5119723B.8060703@gmail.com> Message-ID: Great! On Mon, Feb 11, 2013 at 2:35 PM, Sa?l Ibarra Corretg? wrote: > > >> Good. How about calling future.result(), and letting it raise >> InvalidStateError ? (Except if the timeout handler was called.) >> >> > I did a quick test to see how it feels and I do like it. I'll update the > review with this change plus the other suggested ones. > > > -- > Sa?l Ibarra Corretg? > http://saghul.net/blog | http://about.me/saghul > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From timothy.c.delaney at gmail.com Tue Feb 12 07:49:23 2013 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Tue, 12 Feb 2013 17:49:23 +1100 Subject: [Python-ideas] constant/enum type in stdlib In-Reply-To: References: <51085AAB.6090303@canterbury.ac.nz> <5108A87D.9000207@canterbury.ac.nz> <20130130082639.0b28d7eb@pitrou.net> <20130201103656.0b24c40d@anarchist.wooz.org> Message-ID: On 12 February 2013 07:09, Tim Delaney wrote: > > Supporting this functionality was actually very simple. However, I am > wondering though how useful this is without being able to specify > additional parameters for the enums. I've often used enums with quite > complex behaviour (e.g. in java) - the enum part is purely to have a unique > value assigned to each instance of the class and not repeating myself for > no good reason. I couldn't quite do the same with this as it currently is - > whilst I can have whatever behaviour I want, there's nothing to key it off > except the name and value which probably isn't enough. I've been trying to > think of a syntax which would work to pass additional parameters and I > can't think of anything cleaner than having a specialised class to pass the > additional parameters - but I need to somehow be able to specify the > optional integer value of the enum. Maybe have the first parameter be None? > OK - I've implemented this. EnumValue has grown 'args' and 'kwargs' attributes. Enums can be constructed like: Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:57:17) [MSC v.1600 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> from enum import Enum, EnumParams, _ >>> >>> class MyEnum(Enum): ... A, B ... C = EnumParams(None, 'i', 'j', k='k', l='l') ... D = EnumParams(10, 'm', n='n') ... E ... F = EnumParams(None, 'o') ... G = 20 ... H, I ... J = EnumParams(None, p='p') ... >>> print(repr(MyEnum)) , , , , , , , , , }> _ is aliased to EnumParams so you can do: >>> class MyEnum(Enum):... A, B ... C = _(None, 'i', 'j', k='k', l='l') ... D = _(10, 'm', n='n') ... E ... F = _(None, 'o') ... G = 20 ... H, I ... J = _(None, p='p') ... >>> print(repr(MyEnum)) , , , , , , , , , }> which looks a bit cleaner to me, but might be a little confusing. Thoughts? Tim Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: From timothy.c.delaney at gmail.com Tue Feb 12 09:06:38 2013 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Tue, 12 Feb 2013 19:06:38 +1100 Subject: [Python-ideas] constant/enum type in stdlib In-Reply-To: References: <51085AAB.6090303@canterbury.ac.nz> <5108A87D.9000207@canterbury.ac.nz> <20130130082639.0b28d7eb@pitrou.net> <20130201103656.0b24c40d@anarchist.wooz.org> Message-ID: I'm evil. I just had a very bad thought, and I think I kinda like it. Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:57:17) [MSC v.1600 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> from enum import Enum, _ >>> >>> class Color(Enum): ... RED ... GREEN = _(None, 'green') ... BLUE = _(None, 'blue', hex='0000FF') ... def __init__(self): ... print('__init__', repr(self), self.args, self.kwargs) ... def dump(self): ... print(self, self.args, self.kwargs) ... __init__ () {} __init__ ('green',) {} __init__ ('blue',) {'hex': '0000FF'} >>> print(repr(Color)) , , }> >>> >>> for e in Color: ... e.dump() ... Color(e).dump() ... Color.RED () {} Color.RED () {} Color.GREEN ('green',) {} Color.GREEN ('green',) {} Color.BLUE ('blue',) {'hex': '0000FF'} Color.BLUE ('blue',) {'hex': '0000FF'} When you request an attribute on the EnumValue that doesn't exist, it gets it from the owning Enum class. If the attribute is an instance method (or static method - they have the same type) then it gets turned into an instance method of the EnumValue. Also, when the EnumValue becomes owned by Enum, Enum.__init__ is called as an EnumValue instance method (with no parameters). This actually makes a kind of perverse sense. Conceptually the actual EnumValues are the instances of the Enum (and in fact, I've made them pretend to actually be so). I think it also entirely does away with the need for subclasses of EnumValue and EnumMeta - you can do everything by specifying attributes and behaviour in the Enum instance methods. I've pushed this to https://bitbucket.org/magao/enum - have a play and see how evil you think this is. Tim Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: From eliben at gmail.com Tue Feb 12 16:21:04 2013 From: eliben at gmail.com (Eli Bendersky) Date: Tue, 12 Feb 2013 07:21:04 -0800 Subject: [Python-ideas] PEP for enum library type? Message-ID: Hi all, The ideas thrown around and Tim's prototype in the thread "constant/enum type in stdlib" show that it's possible to use some metaclass magic to implement very convenient syntax for enums, while only including them in the stdlib (no language changes): from enum import Enum class Color(Enum): RED, BLUE, GREEN I'm, for one, convinced that with this in hand we should *not* add special syntax for enums, since the gains would be minimal over the above proposal. While the parallel thread (and Tim's Bitbucket issue tracker) discusses his proposed implementation, I think it may be worthwhile to start from the other direction by writing a PEP that aims to include this in 3.4. The way I see it, the PEP should discuss and help us settle upon a minimal set of features deemed important in an enum. If that sounds OK to people, then someone should write that PEP :-) This could be Tim, or Barry who's been maintaining flufl.enum for a long time. If no one steps up, I will gladly do it. Thoughts? Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsbueno at python.org.br Tue Feb 12 16:30:20 2013 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Tue, 12 Feb 2013 13:30:20 -0200 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: References: Message-ID: On 12 February 2013 13:21, Eli Bendersky wrote: > Hi all, > > The ideas thrown around and Tim's prototype in the thread "constant/enum > type in stdlib" show that it's possible to use some metaclass magic to > implement very convenient syntax for enums, while only including them in the > stdlib (no language changes): > > from enum import Enum > > class Color(Enum): > RED, BLUE, GREEN > > I'm, for one, convinced that with this in hand we should *not* add special > syntax for enums, since the gains would be minimal over the above proposal. > > While the parallel thread (and Tim's Bitbucket issue tracker) discusses his > proposed implementation, I think it may be worthwhile to start from the > other direction by writing a PEP that aims to include this in 3.4. The way I > see it, the PEP should discuss and help us settle upon a minimal set of > features deemed important in an enum. > > If that sounds OK to people, then someone should write that PEP :-) This > could be Tim, or Barry who's been maintaining flufl.enum for a long time. If > no one steps up, I will gladly do it. I'd be fine with writing the PEP as well - maybe we could do it together? > > Thoughts? > > Eli > > From solipsis at pitrou.net Tue Feb 12 16:31:02 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 12 Feb 2013 16:31:02 +0100 Subject: [Python-ideas] PEP for enum library type? References: Message-ID: <20130212163102.50dbf570@pitrou.net> Le Tue, 12 Feb 2013 07:21:04 -0800, Eli Bendersky a ?crit : > > from enum import Enum > > class Color(Enum): > RED, BLUE, GREEN I think I'd favour a namedtuple-style approach, e.g.: class Color(Enum): fields = ('RED', 'BLUE', 'GREEN') (which allows you to build enums programmatically too) Also, I think you have to provide several classes, at least SequenceEnum and StringEnum, and perhaps also BitmaskEnum. It makes sense for them to be separate types, 1) because it makes it explicit how the given enum behaves, 2) because combining integer and str constants in an enum would be crazy. Regards Antoine. From barry at python.org Tue Feb 12 16:42:02 2013 From: barry at python.org (Barry Warsaw) Date: Tue, 12 Feb 2013 10:42:02 -0500 Subject: [Python-ideas] constant/enum type in stdlib References: <20130201103656.0b24c40d@anarchist.wooz.org> Message-ID: <20130212104202.550f5ae7@anarchist.wooz.org> On Feb 12, 2013, at 05:49 PM, Tim Delaney wrote: >_ is aliased to EnumParams so you can do: _() is very commonly used for gettext translations, by long established convention. This is convention is inherited by Python from GNU gettext. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From ethan at stoneleaf.us Tue Feb 12 16:31:42 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 12 Feb 2013 07:31:42 -0800 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: References: Message-ID: <511A605E.3090104@stoneleaf.us> On 02/12/2013 07:21 AM, Eli Bendersky wrote: > While the parallel thread (and Tim's Bitbucket issue tracker) discusses > his proposed implementation, I think it may be worthwhile to start from > the other direction by writing a PEP that aims to include this in 3.4. > The way I see it, the PEP should discuss and help us settle upon a > minimal set of features deemed important in an enum. > > If that sounds OK to people, then someone should write that PEP :-) This > could be Tim, or Barry who's been maintaining flufl.enum for a long > time. If no one steps up, I will gladly do it. A PEP would be excellent. I tried searching for one a couple days ago to help my understanding with Tim's methods, but sadly didn't find one. :( -- ~Ethan~ From ethan at stoneleaf.us Tue Feb 12 18:07:12 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 12 Feb 2013 09:07:12 -0800 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: <20130212163102.50dbf570@pitrou.net> References: <20130212163102.50dbf570@pitrou.net> Message-ID: <511A76C0.4070005@stoneleaf.us> On 02/12/2013 07:31 AM, Antoine Pitrou wrote: > Le Tue, 12 Feb 2013 07:21:04 -0800, > Eli Bendersky a ?crit : >> >> from enum import Enum >> >> class Color(Enum): >> RED, BLUE, GREEN > > I think I'd favour a namedtuple-style approach, e.g.: > > class Color(Enum): > fields = ('RED', 'BLUE', 'GREEN') > > (which allows you to build enums programmatically too) > > Also, I think you have to provide several classes, at least > SequenceEnum and StringEnum, and perhaps also BitmaskEnum. > It makes sense for them to be separate types, 1) because it makes it > explicit how the given enum behaves, 2) because combining integer and > str constants in an enum would be crazy. I partially agree with Antoine; there should be three types of Enum available: - sequence (most common: 0, 1, 2, 3, etc.) - flag (1, 2, 4, 8, etc.) - unique ('on', 'off'; 'north', 'south', 'east', 'west') and they shouldn't be mixed together higgledy-piggledy in the same class. However, I like the simplicity of Tim's approach. The result of the implementation I've been playing with looks something like: class Color(Enum): type = 'sequence' RED, GREEN, BLUE class Geometry(Enum): type = 'unique' LEFT, RIGHT, TOP, BOTTOM WEST, EAST, NORTH, SOUTH class FieldType(Enum): type = 'flag' BINARY, AUTOINC, NULLABLE The metaclass takes care of assigning the next value and selecting the proper subclass; 'unique' values are lower-cased (LEFT -> 'left'); subclassing is possible; and each Enum value is properly an instance of it's Enum class. One important reason for having the different types of Enum is for prototyping. -- ~Ethan~ From jsbueno at python.org.br Tue Feb 12 19:05:44 2013 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Tue, 12 Feb 2013 16:05:44 -0200 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: <20130212163102.50dbf570@pitrou.net> References: <20130212163102.50dbf570@pitrou.net> Message-ID: On 12 February 2013 13:31, Antoine Pitrou wrote: > Le Tue, 12 Feb 2013 07:21:04 -0800, > Eli Bendersky a ?crit : >> >> from enum import Enum >> >> class Color(Enum): >> RED, BLUE, GREEN > > I think I'd favour a namedtuple-style approach, e.g.: > > class Color(Enum): > fields = ('RED', 'BLUE', 'GREEN') > > (which allows you to build enums programmatically too) That is too much ' ' typing - I think it would be ok, to have it like that, but a helper function that would work just like the namedtuple call: Color = Enum("Color", "RED GREEN BLUE", int) The magic of class Color(Enum): RED, GREEN, BLUE is tempting, but basically, what is required so that it does not break class Color(Enum): RED = other_module.value (without making "other_module" a new enum value :-) ) if not worth it, IMHO. such magic is implemented in Tim's proof of concept. A midle term could be to allow declarations like class Color(IntEnum): RED, BLUE, GREEN but with no other logic or expressions in the class body -- all names showing up are strictly part of the sequence, and them have a constructor call, with keyword arguments for all the other cases. > > Also, I think you have to provide several classes, at least > SequenceEnum and StringEnum, and perhaps also BitmaskEnum. > It makes sense for them to be separate types, 1) because it makes it > explicit how the given enum behaves, 2) because combining integer and > str constants in an enum would be crazy. +1 One other important point, that is missed somewhere along Tim's implementation: the repr and str of the enum values should be the value's name - no need to the fancy representation. Actually, the fancy representation defeats most of the purpose of having the constants/enums They could have a property like Enum._qualified_name if one needs it. > > Regards > > Antoine. > > From guido at python.org Tue Feb 12 19:08:44 2013 From: guido at python.org (Guido van Rossum) Date: Tue, 12 Feb 2013 10:08:44 -0800 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: <511A76C0.4070005@stoneleaf.us> References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> Message-ID: I'm torn. I like the clean look of Tim's: class Flag(Enum): RED, WHITE, BLUE However, I don't like the magic needed for its implementation -- and anybody who thinks they know Python pretty well and thinks about this syntax for more than two seconds has to wonder how on earth it's done (and the answer is pretty horrible). It is also pretty brittle to depend on the lookup order -- and I suspect it will be prone to masking other bugs (any typo in a name used in class-level code will essentially generate a bogus new enum value). OTOH I don't like having the string quotes of Antoine's counter-proposal: class Flag(Enum): fields = ('RED', 'WHITE', 'BLUE') Whenever I see quoted names that are also used unquoted, I cringe a little (and this includes calls to hasattr() or getattr() with a string literal for the name). The compromise that I would personally lean towards is more like this: class Flag(Enum): RED, WHITE, BLUE = Even if the requires you to know how many values you are defining, like range(3). If we have to have a somewhat more verbose syntax that doesn't bother me too much. FWIW I do like being able to define enums that are ints and strings (but not mixed); masks/flags I see more as a special case of ints (if you use this you are likely trying to match some API defined in C or C++). I also agree that it must be possible to customize the enum class and the behavior of the values by defining methods on the class. -- --Guido van Rossum (python.org/~guido) From ethan at stoneleaf.us Tue Feb 12 19:31:30 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 12 Feb 2013 10:31:30 -0800 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> Message-ID: <511A8A82.4010602@stoneleaf.us> On 02/12/2013 10:08 AM, Guido van Rossum wrote: > I'm torn. I like the clean look of Tim's: > > class Flag(Enum): > RED, WHITE, BLUE > > However, I don't like the magic needed for its implementation -- and > anybody who thinks they know Python pretty well and thinks about this > syntax for more than two seconds has to wonder how on earth it's done > (and the answer is pretty horrible). It is also pretty brittle to > depend on the lookup order -- and I suspect it will be prone to > masking other bugs (any typo in a name used in class-level code will > essentially generate a bogus new enum value). It seems to me the point of an enum is to give names to an order, so why would the lookup order be a problem? > OTOH I don't like having the string quotes of Antoine's counter-proposal: > > class Flag(Enum): > fields = ('RED', 'WHITE', 'BLUE') > > The compromise that I would personally lean towards is more like this: > > class Flag(Enum): > RED, WHITE, BLUE = > > Even if the requires you to know how many values you are > defining, like range(3). If we have to have a somewhat more verbose > syntax that doesn't bother me too much. Having just dealt with using a poor-person's Enum (THIS, THAT, THE_OTHER = range(3)) and then having to update it a few times (including all the places in the code that depended on the length), I can safely say that the range(n) construct is a PITA. If you don't mind verbose, we could always go for something like: class Color(Enum): start_defs() BLACK RED, GREEN, BLUE CYAN, YELLOW, MAGENTA end_defs() and if we mandate that the enum names must come first we can even ditch the 'start_defs' call. As far as typos go, I don't think that's a new problem (it certainly isn't for me, anyway ;) and my best defense is plenty of tests. -- ~Ethan~ From guido at python.org Tue Feb 12 20:13:37 2013 From: guido at python.org (Guido van Rossum) Date: Tue, 12 Feb 2013 11:13:37 -0800 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: <511A8A82.4010602@stoneleaf.us> References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <511A8A82.4010602@stoneleaf.us> Message-ID: On Tue, Feb 12, 2013 at 10:31 AM, Ethan Furman wrote: > On 02/12/2013 10:08 AM, Guido van Rossum wrote: >> >> I'm torn. I like the clean look of Tim's: >> >> class Flag(Enum): >> RED, WHITE, BLUE >> >> However, I don't like the magic needed for its implementation -- and >> anybody who thinks they know Python pretty well and thinks about this >> syntax for more than two seconds has to wonder how on earth it's done >> (and the answer is pretty horrible). It is also pretty brittle to >> depend on the lookup order -- and I suspect it will be prone to >> masking other bugs (any typo in a name used in class-level code will >> essentially generate a bogus new enum value). > > > It seems to me the point of an enum is to give names to an order, so why > would the lookup order be a problem? Because the language requirement that the expressions in a tuple are evaluated left-to-right is fairly weak. (There are other, similar contexts where the order is not l2r.) >> OTOH I don't like having the string quotes of Antoine's counter-proposal: >> >> class Flag(Enum): >> fields = ('RED', 'WHITE', 'BLUE') >> >> The compromise that I would personally lean towards is more like this: >> >> class Flag(Enum): >> RED, WHITE, BLUE = >> >> Even if the requires you to know how many values you are >> defining, like range(3). If we have to have a somewhat more verbose >> syntax that doesn't bother me too much. > > > > Having just dealt with using a poor-person's Enum (THIS, THAT, THE_OTHER = > range(3)) and then having to update it a few times (including all the places > in the code that depended on the length), I can safely say that the range(n) > construct is a PITA. > > If you don't mind verbose, we could always go for something like: > > class Color(Enum): > start_defs() > BLACK > RED, GREEN, BLUE > CYAN, YELLOW, MAGENTA > end_defs() > > and if we mandate that the enum names must come first we can even ditch the > 'start_defs' call. If that's where you're leaning I would much rather do this: class Color(Enum): BLACK = val() RED = val() # etc. EIBTI. > As far as typos go, I don't think that's a new problem (it certainly isn't > for me, anyway ;) and my best defense is plenty of tests. So with Tim's implementation, what happens here: class Color(Enum): RED, GREEN, BLUE if sys.platform == 'win32': MAGENTA and you happen to have no "import sys" in your module? The sys lookup will succeed, create a new enum, and then you will get an error something like this: AttributeError: 'Enum' object has no attribute 'platform' -- --Guido van Rossum (python.org/~guido) From barry at python.org Tue Feb 12 20:37:48 2013 From: barry at python.org (Barry Warsaw) Date: Tue, 12 Feb 2013 14:37:48 -0500 Subject: [Python-ideas] PEP for enum library type? References: <20130212163102.50dbf570@pitrou.net> Message-ID: <20130212143748.23112336@anarchist.wooz.org> On Feb 12, 2013, at 04:31 PM, Antoine Pitrou wrote: >(which allows you to build enums programmatically too) I prefer an alternative API for creating enums programmatically, e.g.: http://pythonhosted.org/flufl.enum/docs/using.html#alternative-api -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at python.org Tue Feb 12 20:45:18 2013 From: barry at python.org (Barry Warsaw) Date: Tue, 12 Feb 2013 14:45:18 -0500 Subject: [Python-ideas] PEP for enum library type? References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> Message-ID: <20130212144518.3be491c5@anarchist.wooz.org> On Feb 12, 2013, at 09:07 AM, Ethan Furman wrote: >The result of the implementation I've been playing with looks something like: > >class Color(Enum): > type = 'sequence' > RED, GREEN, BLUE > >class Geometry(Enum): > type = 'unique' > LEFT, RIGHT, TOP, BOTTOM > WEST, EAST, NORTH, SOUTH > >class FieldType(Enum): > type = 'flag' > BINARY, AUTOINC, NULLABLE IMHO, none of these approach the simplicity and explicitness of this API: class Color(Enum): RED = 1 GREEN = 2 BLUE = 3 class FieldType(Enum): BINARY = 1 AUTOINC = 2 NULLABLE = 4 The savings in a few characters, does not (again IMHO) make up for all the magic and implicitness you have to guess at with the top versions. I also don't find much value in the 'unique' style, which seem like they're just another way to name string constants. Here though, you could argue that the DRY principle at least makes them interesting, since IIUC, the explicit alternative would be class Geometry: LEFT = 'left' RIGHT = 'right' TOP = 'top' # ... and so on ... I just wouldn't call these "enums" though. ;) Again, being a little more explicit seems like a win, or at least not so bad. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at python.org Tue Feb 12 20:50:36 2013 From: barry at python.org (Barry Warsaw) Date: Tue, 12 Feb 2013 14:50:36 -0500 Subject: [Python-ideas] PEP for enum library type? References: <20130212163102.50dbf570@pitrou.net> Message-ID: <20130212145036.32a98928@anarchist.wooz.org> On Feb 12, 2013, at 04:05 PM, Joao S. O. Bueno wrote: >That is too much ' ' typing - I think it would be ok, to have it like that, >but a helper function that would work just like the namedtuple call: Color = >Enum("Color", "RED GREEN BLUE", int) How about: >>> from flufl.enum import make >>> make('Colors', 'RED GREEN BLUE'.split()) -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From guido at python.org Tue Feb 12 20:59:54 2013 From: guido at python.org (Guido van Rossum) Date: Tue, 12 Feb 2013 11:59:54 -0800 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <511A8A82.4010602@stoneleaf.us> Message-ID: After all the defenses I still don't like Tim's proposed syntax. Color me Barry. On Tue, Feb 12, 2013 at 11:54 AM, Tim Delaney wrote: > On 13 February 2013 06:13, Guido van Rossum wrote: >> >> So with Tim's implementation, what happens here: >> >> class Color(Enum): >> RED, GREEN, BLUE >> if sys.platform == 'win32': >> MAGENTA >> >> and you happen to have no "import sys" in your module? The sys lookup >> will succeed, create a new enum, and then you will get an error >> something like this: >> >> AttributeError: 'Enum' object has no attribute 'platform' > > > Well, that particular case would work (you'd get a NameError: sys) due to > having done an attribute lookup on sys, but the following: > > class Color(Enum): > RED, GREEN, BLUE > if platfor == 'win32': > MAGENTA > > would create an enum value 'platfor'. > > Personally, I don't think it's possible to have an enum with a simple > interface and powerful semantics with just python code without it being > fragile. I think I've got it fairly close, but there is horrible magic in > there (multiple kinds) and there are definitely still edge cases. Any > complete enum implementation is going to need some special handling by the > parser I think. > > I'm actually thinking that to simplify things, I need a sentinel object to > mark the end of the enum list (which allows other names after it). But that > still wouldn't handle the case above (the if statement). > > BTW, for anyone who hasn't seen the magic code (largely uncommented, no > explanations yet of how it's doing it - I probably won't get to that until > the weekend) it's here: https://bitbucket.org/magao/enum > > Tim Delaney -- --Guido van Rossum (python.org/~guido) From ethan at stoneleaf.us Tue Feb 12 20:26:28 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 12 Feb 2013 11:26:28 -0800 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <511A8A82.4010602@stoneleaf.us> Message-ID: <511A9764.2060902@stoneleaf.us> On 02/12/2013 11:13 AM, Guido van Rossum wrote: > On Tue, Feb 12, 2013 at 10:31 AM, Ethan Furman wrote: >> On 02/12/2013 10:08 AM, Guido van Rossum wrote: >>> >>> I'm torn. I like the clean look of Tim's: >>> >>> class Flag(Enum): >>> RED, WHITE, BLUE >>> >>> However, I don't like the magic needed for its implementation -- and >>> anybody who thinks they know Python pretty well and thinks about this >>> syntax for more than two seconds has to wonder how on earth it's done >>> (and the answer is pretty horrible). It is also pretty brittle to >>> depend on the lookup order -- and I suspect it will be prone to >>> masking other bugs (any typo in a name used in class-level code will >>> essentially generate a bogus new enum value). >> >> >> It seems to me the point of an enum is to give names to an order, so why >> would the lookup order be a problem? > > Because the language requirement that the expressions in a tuple are > evaluated left-to-right is fairly weak. (There are other, similar > contexts where the order is not l2r.) Ah. >> If you don't mind verbose, we could always go for something like: >> >> class Color(Enum): >> start_defs() >> BLACK >> RED, GREEN, BLUE >> CYAN, YELLOW, MAGENTA >> end_defs() >> >> and if we mandate that the enum names must come first we can even ditch the >> 'start_defs' call. > > If that's where you're leaning I would much rather do this: > > class Color(Enum): > BLACK = val() > RED = val() > # etc. > > EIBTI. Personally, I'd be fine with: class Color(Enum): BLACK RED BLUE PURPLE which avoids the l2r issue, and the repetitive use of val() (or whatever it's called). >> As far as typos go, I don't think that's a new problem (it certainly isn't >> for me, anyway ;) and my best defense is plenty of tests. > > So with Tim's implementation, what happens here: > > class Color(Enum): > RED, GREEN, BLUE > if sys.platform == 'win32': > MAGENTA > > and you happen to have no "import sys" in your module? The sys lookup > will succeed, create a new enum, and then you will get an error > something like this: > > AttributeError: 'Enum' object has no attribute 'platform' We could minimize that issue by requiring that enum names be ALLCAPS -- if something comes through that isn't, don't supply a value and let the AttributeError perculate up. -- ~Ethan~ From timothy.c.delaney at gmail.com Tue Feb 12 21:13:21 2013 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Wed, 13 Feb 2013 07:13:21 +1100 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <511A8A82.4010602@stoneleaf.us> Message-ID: Sorry to anyone who gets this twice - sent it from my work address which is not subscribed to Python-Ideas. On 13 February 2013 06:13, Guido van Rossum wrote: > > As far as typos go, I don't think that's a new problem (it certainly > isn't > > for me, anyway ;) and my best defense is plenty of tests. > > So with Tim's implementation, what happens here: > > class Color(Enum): > RED, GREEN, BLUE > if sys.platform == 'win32': > MAGENTA > > and you happen to have no "import sys" in your module? The sys lookup > will succeed, create a new enum, and then you will get an error > something like this: > > AttributeError: 'Enum' object has no attribute 'platform' Well, that particular case would work (you'd get a NameError: sys) due to having done an attribute lookup on sys, but the following: class Color(Enum): RED, GREEN, BLUE if platfor == 'win32': MAGENTA would create an enum value 'platfor'. Personally, I don't think it's possible to have an enum with a simple interface and powerful semantics with just python code without it being fragile. I think I've got it fairly close, but there is horrible magic in there (multiple kinds) and there are definitely still edge cases. Any complete enum implementation is going to need some special handling by the parser I think. I'm actually thinking that to simplify things, I need a sentinel object to mark the end of the enum list (which allows other names after it). But that still wouldn't handle the case above (the if statement). BTW, for anyone who hasn't seen the magic code (largely uncommented, no explanations yet of how it's doing it - I probably won't get to that until the weekend) it's here: https://bitbucket.org/magao/enum Tim Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: From timothy.c.delaney at gmail.com Tue Feb 12 21:14:59 2013 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Wed, 13 Feb 2013 07:14:59 +1100 Subject: [Python-ideas] constant/enum type in stdlib In-Reply-To: <20130212104202.550f5ae7@anarchist.wooz.org> References: <20130201103656.0b24c40d@anarchist.wooz.org> <20130212104202.550f5ae7@anarchist.wooz.org> Message-ID: On 13 February 2013 02:42, Barry Warsaw wrote: > On Feb 12, 2013, at 05:49 PM, Tim Delaney wrote: > > >_ is aliased to EnumParams so you can do: > > _() is very commonly used for gettext translations, by long established > convention. This is convention is inherited by Python from GNU gettext. > Good point - it's actually what inspired me to use it, but a bad idea for me to do so. I've taken that out. Tim Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Tue Feb 12 20:58:30 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 12 Feb 2013 11:58:30 -0800 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: <20130212144518.3be491c5@anarchist.wooz.org> References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <20130212144518.3be491c5@anarchist.wooz.org> Message-ID: <511A9EE6.5070803@stoneleaf.us> On 02/12/2013 11:45 AM, Barry Warsaw wrote: > On Feb 12, 2013, at 09:07 AM, Ethan Furman wrote: > >> The result of the implementation I've been playing with looks something like: >> >> class Color(Enum): >> type = 'sequence' >> RED, GREEN, BLUE >> >> class Geometry(Enum): >> type = 'unique' >> LEFT, RIGHT, TOP, BOTTOM >> WEST, EAST, NORTH, SOUTH >> >> class FieldType(Enum): >> type = 'flag' >> BINARY, AUTOINC, NULLABLE > > IMHO, none of these approach the simplicity and explicitness of this API: > > class Color(Enum): > RED = 1 > GREEN = 2 > BLUE = 3 > > class FieldType(Enum): > BINARY = 1 > AUTOINC = 2 > NULLABLE = 4 > > The savings in a few characters, does not (again IMHO) make up for all the > magic and implicitness you have to guess at with the top versions. 0) Once you read the docs you don't have to guess ;) 1) Magic is fun. :) 2) I hate repetition (= 1 = 2 = 3 is repetitive) 3) I make mistakes when repeating stuff (... = 11 = 11 = 13) > I also don't find much value in the 'unique' style, which seem like they're > just another way to name string constants. Here though, you could argue that > the DRY principle at least makes them interesting, since IIUC, the explicit > alternative would be > > class Geometry: > LEFT = 'left' > RIGHT = 'right' > TOP = 'top' > # ... and so on ... Class membership can also be an advantage. -- ~Ethan~ From jsbueno at python.org.br Tue Feb 12 21:33:13 2013 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Tue, 12 Feb 2013 18:33:13 -0200 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: <20130212145036.32a98928@anarchist.wooz.org> References: <20130212163102.50dbf570@pitrou.net> <20130212145036.32a98928@anarchist.wooz.org> Message-ID: On 12 February 2013 17:50, Barry Warsaw wrote: >>That is too much ' ' typing - I think it would be ok, to have it like that, >>but a helper function that would work just like the namedtuple call: Color = >>Enum("Color", "RED GREEN BLUE", int) > > How about: > > >>> from flufl.enum import make > >>> make('Colors', 'RED GREEN BLUE'.split()) > Not bad - I think that wahtever is agreeded in a nice api that takes advantage of the "class" statement we should have a callabale constructor that takes names as strings. And moreover, it is nice to have a way to load the created enums/constants to the current namespace - and it can be done, without "magic" to the global namespace having a call that takes a "namespace" argument to which one would normally pass in blobals() So: from enum import load, make would allow for: >>> Colors = make("Colors", "RED GREEN BLUE".split()) >>> load(Colors, globals()) >>> RED RED (I think that allowing a call without the "split()" would be better - ) That would make some *linters complain - but those should update themselves to the fact in time. I consider a way to load the created constants into the module namespace very important to the concept of enums/constant itself. Agreeing on a call that can get "globals()" and update it would preclude the idea of being able to do "from MyEnumClass import *" that showed up on the previous thread. js -><- From ubershmekel at gmail.com Tue Feb 12 21:38:21 2013 From: ubershmekel at gmail.com (Yuval Greenfield) Date: Tue, 12 Feb 2013 22:38:21 +0200 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: <511A9EE6.5070803@stoneleaf.us> References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <20130212144518.3be491c5@anarchist.wooz.org> <511A9EE6.5070803@stoneleaf.us> Message-ID: On Tue, Feb 12, 2013 at 9:58 PM, Ethan Furman wrote: > 1) Magic is fun. :) > I heartily agree. Check this out: >>> RED, GREEN, BLUE = enum() >>> print(RED, GREEN, BLUE) 0 1 2 >>> LEFT, TOP, RIGHT, DOWN = enum() >>> print(LEFT, TOP, RIGHT, DOWN) 0 1 2 3 And here's the dirty trick.... import inspect def enum(): """If you use this your computer will most likely burst ablaze and your teeth will fall off""" frame = inspect.stack()[1][0] # 4 = 3 for the function call "enum()", 1 for the opcode "UNPACK_SEQUENCE", # the next two bytes represent how many things we need to unpack. a, b = frame.f_code.co_code[frame.f_lasti + 4:frame.f_lasti + 6] enum_len = a + b * 256 return range(enum_len) Cheers, Yuval -------------- next part -------------- An HTML attachment was scrubbed... URL: From joshua.landau.ws at gmail.com Tue Feb 12 21:43:50 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Tue, 12 Feb 2013 20:43:50 +0000 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: <511A9764.2060902@stoneleaf.us> References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <511A8A82.4010602@stoneleaf.us> <511A9764.2060902@stoneleaf.us> Message-ID: On 12 February 2013 19:26, Ethan Furman wrote: > On 02/12/2013 11:13 AM, Guido van Rossum wrote: > >> On Tue, Feb 12, 2013 at 10:31 AM, Ethan Furman >> wrote: >> >>> On 02/12/2013 10:08 AM, Guido van Rossum wrote: >>> >>>> >>>> I'm torn. I like the clean look of Tim's: >>>> >>>> class Flag(Enum): >>>> RED, WHITE, BLUE >>>> >>>> However, I don't like the magic needed for its implementation -- and >>>> anybody who thinks they know Python pretty well and thinks about this >>>> syntax for more than two seconds has to wonder how on earth it's done >>>> (and the answer is pretty horrible). It is also pretty brittle to >>>> depend on the lookup order -- and I suspect it will be prone to >>>> masking other bugs (any typo in a name used in class-level code will >>>> essentially generate a bogus new enum value). >>>> >>> >>> >>> It seems to me the point of an enum is to give names to an order, so why >>> would the lookup order be a problem? >>> >> >> Because the language requirement that the expressions in a tuple are >> evaluated left-to-right is fairly weak. (There are other, similar >> contexts where the order is not l2r.) > > > Personally, I'd be fine with: > > class Color(Enum): > BLACK > RED > BLUE > PURPLE > > which avoids the l2r issue, and the repetitive use of val() (or whatever > it's called). Can you force people not to use: class Color(Enum): BLACK, RED, BLUE, PURPLE ? If I'm understanding the implementation, if you support one then you support the other. Hence the problem will still stand. That said, I'm not personally too worried about it. As far as typos go, I don't think that's a new problem (it certainly isn't >>> for me, anyway ;) and my best defense is plenty of tests. >>> >> >> So with Tim's implementation, what happens here: >> >> class Color(Enum): >> RED, GREEN, BLUE >> if sys.platform == 'win32': >> MAGENTA >> >> and you happen to have no "import sys" in your module? The sys lookup >> will succeed, create a new enum, and then you will get an error >> something like this: >> >> AttributeError: 'Enum' object has no attribute 'platform' >> > > We could minimize that issue by requiring that enum names be ALLCAPS -- if > something comes through that isn't, don't supply a value and let the > AttributeError perculate up. > I think it's worse than that. I'm using Tim Delaney's code and I'm not really sure why it's doing what it's doing. import sys class Color(Enum): RED, GREEN, BLUE if sys.platform: MAGENTA works and doesn't create Color.sys. import sys class Color(Enum): RED, GREEN, BLUE if sys: MAGENTA works and _does_ create Color.sys. So somehow there's enough magic to realise that if you are accessing an attribute, it's not going to be an enum. In other words, you don't need the ALLCAPS anyway. You might think "oh, good!". I think "oh noes!". It's way too magic for me. The other idea: class Color(Enum): values = "RED", "GREEN", "BLUE", "MAGENTA", "OTHER", "COLORS", "HERE", "TOO" could be written class Color(Enum): values = "RED GREEN BLUE MAGENTA OTHER COLORS HERE TOO".split() so I'm OK with that, but then you're in the territory when you might as well go with the method Barry suggested. -------------- next part -------------- An HTML attachment was scrubbed... URL: From eliben at gmail.com Tue Feb 12 21:58:44 2013 From: eliben at gmail.com (Eli Bendersky) Date: Tue, 12 Feb 2013 12:58:44 -0800 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <511A8A82.4010602@stoneleaf.us> Message-ID: On Tue, Feb 12, 2013 at 11:59 AM, Guido van Rossum wrote: > After all the defenses I still don't like Tim's proposed syntax. Color me > Barry. > In this case, I see no reason not to use Barry's flufl.enum - it's pretty good and has been around for a while. The main advantage of Tim's implementation syntax (wasn't it Michael Foord who originally proposed it?) is that it lets us do less typing which is great. IOW, sure with flufl.enum I can do this and be reasonably happy: class Color(Enum): RED = 1 BLUE = 2 GREEN = 3 But in 99% of the cases in Python code I absolutely don't care about the values of these enumerations (*), so I'd much, *much* rather write: class Color(Enum): RED, BLUE, GREEN Yes, the implementation is messy, but it can be documented and tested well, and it's fairly small and confined. But this is a matter of preference, and I can see your point (Guido) about preferring the more explicit syntax in order to avoid too much magic. Eli (*) And where I do care, I'd still enjoy them being generated automatically. -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Tue Feb 12 22:04:12 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 12 Feb 2013 22:04:12 +0100 Subject: [Python-ideas] PEP for enum library type? References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <511A8A82.4010602@stoneleaf.us> Message-ID: <20130212220412.4bd3e604@pitrou.net> On Tue, 12 Feb 2013 12:58:44 -0800 Eli Bendersky wrote: > On Tue, Feb 12, 2013 at 11:59 AM, Guido van Rossum wrote: > > > After all the defenses I still don't like Tim's proposed syntax. Color me > > Barry. > > > > In this case, I see no reason not to use Barry's flufl.enum - it's pretty > good and has been around for a while. The main advantage of Tim's > implementation syntax (wasn't it Michael Foord who originally proposed it?) > is that it lets us do less typing which is great. IOW, sure with flufl.enum > I can do this and be reasonably happy: > > class Color(Enum): > RED = 1 > BLUE = 2 > GREEN = 3 I still hope enum values are strings by default - or, if not, that they get nice reprs. Integer enums are only useful for interoperability with stubborn low-level libraries :-) Regards Antoine. From timothy.c.delaney at gmail.com Tue Feb 12 22:14:51 2013 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Wed, 13 Feb 2013 08:14:51 +1100 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <511A8A82.4010602@stoneleaf.us> Message-ID: On 13 February 2013 07:58, Eli Bendersky wrote: > On Tue, Feb 12, 2013 at 11:59 AM, Guido van Rossum wrote: > >> After all the defenses I still don't like Tim's proposed syntax. Color me >> Barry. >> > > The main advantage of Tim's implementation syntax (wasn't it Michael > Foord who originally proposed it?) > Indeed it was and I don't want to take any of the "credit" away from Michael. I simply took his original inspired idea and added several extra layers of magic on top ;) I've updated my README to make that clear. Tim Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: From drekin at gmail.com Tue Feb 12 22:19:13 2013 From: drekin at gmail.com (drekin at gmail.com) Date: Tue, 12 Feb 2013 13:19:13 -0800 (PST) Subject: [Python-ideas] PEP for enum library type? In-Reply-To: Message-ID: <511ab1d1.83b80e0a.01e6.057d@mx.google.com> Hello. > masks/flags I see more as a special case of ints (if you use this you are likely trying to match some API defined in C or C++). Comfortably encoding flags as int is just a half of FlagsEnum functionality. It has also abstract set semantics, see http://mail.python.org/pipermail/python-ideas/2013-January/019150.html . Regards, Drekin From timothy.c.delaney at gmail.com Tue Feb 12 22:19:41 2013 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Wed, 13 Feb 2013 08:19:41 +1100 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <511A8A82.4010602@stoneleaf.us> Message-ID: On 13 February 2013 07:58, Eli Bendersky wrote: > > But in 99% of the cases in Python code I absolutely don't care about the > values of these enumerations (*), so I'd much, *much* rather write: > > class Color(Enum): > RED, BLUE, GREEN > > Yes, the implementation is messy, but it can be documented and tested > well, and it's fairly small and confined. But this is a matter of > preference, and I can see your point (Guido) about preferring the more > explicit syntax in order to avoid too much magic. > > Eli > > (*) And where I do care, I'd still enjoy them being generated > automatically. > I'm of the opinion that the most fundamental property of an enum is that it can generate its values automatically. Everything else is a nice-to-have, but without that property it's not much use as an enum. Tim Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Feb 12 22:29:17 2013 From: guido at python.org (Guido van Rossum) Date: Tue, 12 Feb 2013 13:29:17 -0800 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <511A8A82.4010602@stoneleaf.us> Message-ID: Please. Anything *except* class Color(Enum): RED, BLUE, GREEN And not this eiter: class Color(Enum): RED BLUE GREEN On Tue, Feb 12, 2013 at 12:58 PM, Eli Bendersky wrote: > On Tue, Feb 12, 2013 at 11:59 AM, Guido van Rossum wrote: >> >> After all the defenses I still don't like Tim's proposed syntax. Color me >> Barry. > > > In this case, I see no reason not to use Barry's flufl.enum - it's pretty > good and has been around for a while. The main advantage of Tim's > implementation syntax (wasn't it Michael Foord who originally proposed it?) > is that it lets us do less typing which is great. IOW, sure with flufl.enum > I can do this and be reasonably happy: > > > class Color(Enum): > RED = 1 > BLUE = 2 > GREEN = 3 > > But in 99% of the cases in Python code I absolutely don't care about the > values of these enumerations (*), so I'd much, *much* rather write: > > class Color(Enum): > RED, BLUE, GREEN > > Yes, the implementation is messy, but it can be documented and tested well, > and it's fairly small and confined. But this is a matter of preference, and > I can see your point (Guido) about preferring the more explicit syntax in > order to avoid too much magic. > > Eli > > (*) And where I do care, I'd still enjoy them being generated automatically. > > -- --Guido van Rossum (python.org/~guido) From rosuav at gmail.com Tue Feb 12 22:36:11 2013 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 13 Feb 2013 08:36:11 +1100 Subject: [Python-ideas] constant/enum type in stdlib In-Reply-To: References: <51085AAB.6090303@canterbury.ac.nz> <5108A87D.9000207@canterbury.ac.nz> <20130130082639.0b28d7eb@pitrou.net> <20130201103656.0b24c40d@anarchist.wooz.org> Message-ID: On Tue, Feb 12, 2013 at 7:06 PM, Tim Delaney wrote: > This actually makes a kind of perverse sense. Conceptually the actual > EnumValues are the instances of the Enum (and in fact, I've made them > pretend to actually be so). Not so perverse. I think that makes very good sense (but then, I know C++ enums, so maybe I'm tainted). Looks good, though I've not actually played with the code. ChrisA From stefan at bytereef.org Tue Feb 12 22:46:55 2013 From: stefan at bytereef.org (Stefan Krah) Date: Tue, 12 Feb 2013 22:46:55 +0100 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <511A8A82.4010602@stoneleaf.us> Message-ID: <20130212214655.GA20475@sleipnir.bytereef.org> Guido van Rossum wrote: > Please. Anything *except* > > class Color(Enum): > RED, BLUE, GREEN > > And not this eiter: > > class Color(Enum): > RED > BLUE > GREEN Is a new enum keyword still a possibility? To me that seems to be the cleanest way. Stefan Krah From ryan at ryanhiebert.com Tue Feb 12 22:55:06 2013 From: ryan at ryanhiebert.com (ryan at ryanhiebert.com) Date: Tue, 12 Feb 2013 13:55:06 -0800 Subject: [Python-ideas] enums vs symbols In-Reply-To: <20130212214655.GA20475@sleipnir.bytereef.org> References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <511A8A82.4010602@stoneleaf.us> <20130212214655.GA20475@sleipnir.bytereef.org> Message-ID: <28F639F9-CF62-4AF7-BB80-9CFEEA2F385C@ryanhiebert.com> Forgive me, a lurker, but I wonder: how does an enum differ in usage from something like a Symbol from ruby, other than that an enum is required to be declared beforehand? I'm definitely a python guy, but the idea of identifiers that are their own value interests me, and seems like it could have some overlap with the concept and usage of enums. If there's not much difference in usage between enums and symbols apart from declaration, perhaps we could make the declaration optional and combine the concepts? This would certainly require parser changes. Ryan From guido at python.org Tue Feb 12 22:58:04 2013 From: guido at python.org (Guido van Rossum) Date: Tue, 12 Feb 2013 13:58:04 -0800 Subject: [Python-ideas] enums vs symbols In-Reply-To: <28F639F9-CF62-4AF7-BB80-9CFEEA2F385C@ryanhiebert.com> References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <511A8A82.4010602@stoneleaf.us> <20130212214655.GA20475@sleipnir.bytereef.org> <28F639F9-CF62-4AF7-BB80-9CFEEA2F385C@ryanhiebert.com> Message-ID: I don't know Ruby symbols, but if they are anything like Lisp symbols, they are completely different. Lisp symbols are interned strings -- Python just uses strings (and if they happen to be interned you get a modest speedup). This is for things like: if hasattr(x, 'foo'): x.foo() On Tue, Feb 12, 2013 at 1:55 PM, wrote: > Forgive me, a lurker, but I wonder: how does an enum differ in usage from something like a Symbol from ruby, other than that an enum is required to be declared beforehand? > > I'm definitely a python guy, but the idea of identifiers that are their own value interests me, and seems like it could have some overlap with the concept and usage of enums. > > If there's not much difference in usage between enums and symbols apart from declaration, perhaps we could make the declaration optional and combine the concepts? This would certainly require parser changes. > > Ryan > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas -- --Guido van Rossum (python.org/~guido) From guido at python.org Tue Feb 12 22:58:49 2013 From: guido at python.org (Guido van Rossum) Date: Tue, 12 Feb 2013 13:58:49 -0800 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: <20130212220412.4bd3e604@pitrou.net> References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <511A8A82.4010602@stoneleaf.us> <20130212220412.4bd3e604@pitrou.net> Message-ID: Definitely they need nice reprs. On Tue, Feb 12, 2013 at 1:04 PM, Antoine Pitrou wrote: > On Tue, 12 Feb 2013 12:58:44 -0800 > Eli Bendersky wrote: > >> On Tue, Feb 12, 2013 at 11:59 AM, Guido van Rossum wrote: >> >> > After all the defenses I still don't like Tim's proposed syntax. Color me >> > Barry. >> > >> >> In this case, I see no reason not to use Barry's flufl.enum - it's pretty >> good and has been around for a while. The main advantage of Tim's >> implementation syntax (wasn't it Michael Foord who originally proposed it?) >> is that it lets us do less typing which is great. IOW, sure with flufl.enum >> I can do this and be reasonably happy: >> >> class Color(Enum): >> RED = 1 >> BLUE = 2 >> GREEN = 3 > > I still hope enum values are strings by default - or, if not, that they > get nice reprs. Integer enums are only useful for interoperability with > stubborn low-level libraries :-) > > Regards > > Antoine. > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas -- --Guido van Rossum (python.org/~guido) From guido at python.org Tue Feb 12 22:59:44 2013 From: guido at python.org (Guido van Rossum) Date: Tue, 12 Feb 2013 13:59:44 -0800 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <511A8A82.4010602@stoneleaf.us> Message-ID: And I'm of the opinion that the most fundamental property of an enum is that it doesn't print as an int. On Tue, Feb 12, 2013 at 1:19 PM, Tim Delaney wrote: > On 13 February 2013 07:58, Eli Bendersky wrote: >> >> >> But in 99% of the cases in Python code I absolutely don't care about the >> values of these enumerations (*), so I'd much, *much* rather write: >> >> class Color(Enum): >> RED, BLUE, GREEN >> >> Yes, the implementation is messy, but it can be documented and tested >> well, and it's fairly small and confined. But this is a matter of >> preference, and I can see your point (Guido) about preferring the more >> explicit syntax in order to avoid too much magic. >> >> Eli >> >> (*) And where I do care, I'd still enjoy them being generated >> automatically. > > > I'm of the opinion that the most fundamental property of an enum is that it > can generate its values automatically. Everything else is a nice-to-have, > but without that property it's not much use as an enum. > > Tim Delaney -- --Guido van Rossum (python.org/~guido) From timothy.c.delaney at gmail.com Tue Feb 12 23:09:04 2013 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Wed, 13 Feb 2013 09:09:04 +1100 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <511A8A82.4010602@stoneleaf.us> Message-ID: On 13 February 2013 08:59, Guido van Rossum wrote: > And I'm of the opinion that the most fundamental property of an enum > is that it doesn't print as an int. That part is easy. Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:57:17) [MSC v.1600 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> from enum import Enum >>> class Color(Enum): ... RED ... >>> str(Color.RED) 'Color.RED' >>> repr(Color.RED) "" >>> str(Color) '{RED:0}' >>> repr(Color) "}>" Making an appropriate str/repr just requires deciding what it should be. I chose to make the reprs give more information, but there's no reason they couldn't be simplified. I'll see if I can come up with a syntax for declaration that we're both happy with, but I doubt I'll be as happy with it as what I've currently got (despite its edge cases). Tim Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: From ryan at ryanhiebert.com Tue Feb 12 23:13:53 2013 From: ryan at ryanhiebert.com (ryan at ryanhiebert.com) Date: Tue, 12 Feb 2013 14:13:53 -0800 Subject: [Python-ideas] enums vs symbols In-Reply-To: References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <511A8A82.4010602@stoneleaf.us> <20130212214655.GA20475@sleipnir.bytereef.org> <28F639F9-CF62-4AF7-BB80-9CFEEA2F385C@ryanhiebert.com> Message-ID: <0233EDDB-8034-42D6-9052-505E16D878E4@ryanhiebert.com> Thanks for that. I think that when I consider the cases that I don't like the idea of using strings, using a proper enum makes better sense anyway. On Feb 12, 2013, at 1:58 PM, Guido van Rossum wrote: > I don't know Ruby symbols, but if they are anything like Lisp symbols, > they are completely different. Lisp symbols are interned strings -- > Python just uses strings (and if they happen to be interned you get a > modest speedup). This is for things like: > > if hasattr(x, 'foo'): > x.foo() > > On Tue, Feb 12, 2013 at 1:55 PM, wrote: >> Forgive me, a lurker, but I wonder: how does an enum differ in usage from something like a Symbol from ruby, other than that an enum is required to be declared beforehand? >> >> I'm definitely a python guy, but the idea of identifiers that are their own value interests me, and seems like it could have some overlap with the concept and usage of enums. >> >> If there's not much difference in usage between enums and symbols apart from declaration, perhaps we could make the declaration optional and combine the concepts? This would certainly require parser changes. >> >> Ryan >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> http://mail.python.org/mailman/listinfo/python-ideas > > > > -- > --Guido van Rossum (python.org/~guido) From eliben at gmail.com Tue Feb 12 23:13:57 2013 From: eliben at gmail.com (Eli Bendersky) Date: Tue, 12 Feb 2013 14:13:57 -0800 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: <20130212220412.4bd3e604@pitrou.net> References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <511A8A82.4010602@stoneleaf.us> <20130212220412.4bd3e604@pitrou.net> Message-ID: On Tue, Feb 12, 2013 at 1:04 PM, Antoine Pitrou wrote: > On Tue, 12 Feb 2013 12:58:44 -0800 > Eli Bendersky wrote: > > > On Tue, Feb 12, 2013 at 11:59 AM, Guido van Rossum > wrote: > > > > > After all the defenses I still don't like Tim's proposed syntax. Color > me > > > Barry. > > > > > > > In this case, I see no reason not to use Barry's flufl.enum - it's pretty > > good and has been around for a while. The main advantage of Tim's > > implementation syntax (wasn't it Michael Foord who originally proposed > it?) > > is that it lets us do less typing which is great. IOW, sure with > flufl.enum > > I can do this and be reasonably happy: > > > > class Color(Enum): > > RED = 1 > > BLUE = 2 > > GREEN = 3 > > I still hope enum values are strings by default - or, if not, that they > get nice reprs. Integer enums are only useful for interoperability with > stubborn low-level libraries :-) > I agree, and Tim's implementation provides a very nice string representation, no matter what the value underneath is. I don't really care what the value underneath is, actually. Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Tue Feb 12 23:14:22 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 12 Feb 2013 14:14:22 -0800 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <20130212144518.3be491c5@anarchist.wooz.org> <511A9EE6.5070803@stoneleaf.us> Message-ID: <511ABEBE.90201@stoneleaf.us> On 02/12/2013 12:38 PM, Yuval Greenfield wrote: > On Tue, Feb 12, 2013 at 9:58 PM, Ethan Furman > wrote: > > 1) Magic is fun. :) > > > I heartily agree. Check this out: > > >>> RED, GREEN, BLUE = enum() > >>> print(RED, GREEN, BLUE) > 0 1 2 > >>> LEFT, TOP, RIGHT, DOWN = enum() > >>> print(LEFT, TOP, RIGHT, DOWN) > 0 1 2 3 > > > And here's the dirty trick.... > > import inspect > def enum(): > """If you use this your computer will most likely burst ablaze and > your teeth will fall off""" > frame = inspect.stack()[1][0] > # 4 = 3 for the function call "enum()", 1 for the opcode > "UNPACK_SEQUENCE", > # the next two bytes represent how many things we need to unpack. > a, b = frame.f_code.co_code[frame.f_lasti + 4:frame.f_lasti + 6] > enum_len = a + b * 256 > return range(enum_len) That is pretty cool. Now all it needs is a nice repr(), and appropriate comparisons with only int and same enum group. Is it cross-Python? -- ~Ethan~ From jonathan at slenders.be Tue Feb 12 23:29:38 2013 From: jonathan at slenders.be (Jonathan Slenders) Date: Tue, 12 Feb 2013 23:29:38 +0100 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: <20130212214655.GA20475@sleipnir.bytereef.org> References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <511A8A82.4010602@stoneleaf.us> <20130212214655.GA20475@sleipnir.bytereef.org> Message-ID: 2013/2/12 Stefan Krah > Guido van Rossum wrote: > > Please. Anything *except* > > > > class Color(Enum): > > RED, BLUE, GREEN > > > > And not this eiter: > > > > class Color(Enum): > > RED > > BLUE > > GREEN > > Is a new enum keyword still a possibility? To me that seems to be the > cleanest way. I was just thinking the same. We don't like to have more syntax than necessary, but we should also not introduce too much magic... -------------- next part -------------- An HTML attachment was scrubbed... URL: From timothy.c.delaney at gmail.com Tue Feb 12 23:32:25 2013 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Wed, 13 Feb 2013 09:32:25 +1100 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <511A8A82.4010602@stoneleaf.us> <20130212214655.GA20475@sleipnir.bytereef.org> Message-ID: On 13 February 2013 09:29, Jonathan Slenders wrote: > Is a new enum keyword still a possibility? To me that seems to be the > >> cleanest way. > > > I was just thinking the same. We don't like to have more syntax than > necessary, but we should also not introduce too much magic... > That's my conclusion as well. To get a nice* enum with guaranteed semantics and no edge cases, I think it needs parser support. * for my definition of "nice". Tim Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Tue Feb 12 23:51:50 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 13 Feb 2013 09:51:50 +1100 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: <20130212144518.3be491c5@anarchist.wooz.org> References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <20130212144518.3be491c5@anarchist.wooz.org> Message-ID: <511AC786.80807@pearwood.info> On 13/02/13 06:45, Barry Warsaw wrote: [...] > IMHO, none of these approach the simplicity and explicitness of this API: > > class Color(Enum): > RED = 1 > GREEN = 2 > BLUE = 3 > > class FieldType(Enum): > BINARY = 1 > AUTOINC = 2 > NULLABLE = 4 > > The savings in a few characters, does not (again IMHO) make up for all the > magic and implicitness you have to guess at with the top versions. > > I also don't find much value in the 'unique' style, which seem like they're > just another way to name string constants. Here though, you could argue that > the DRY principle at least makes them interesting, since IIUC, the explicit > alternative would be > > class Geometry: > LEFT = 'left' > RIGHT = 'right' > TOP = 'top' > # ... and so on ... > > I just wouldn't call these "enums" though. ;) Again, being a little more > explicit seems like a win, or at least not so bad. My perspective from the peanut gallery is, not so. That "little more" explicitness actually raises the barrier to using enums enough that they are underused. In my experience, good practice or not, too people avoid defining enum-like constants: LEFT = 'left' RIGHT = 'right' ... and just embed the string literals in their code instead. def move(distance, direction): if direction == 'left': ... It's hard to avoid the "magic constant" anti-pattern, and too easy to fall into the bad habit. Speaking as an ex-Pascal programmer, I really miss the simplicity of creating enumerated types. type units = (inches, feet, furlongs, miles); relationship = (parent, sibling, child); which would correspond to Tim's suggestion: class units(Enum): INCHES, FEET, FURLONGS, MILES class relationship(Enum): PARENT, SIBLING, CHILD I would use that, since I don't really care what the values of the enums are. All I care is: - they're unique within a type/class; - they print like their name (e.g. FEET not 1); - (optional) they can combine as bitmasks. Not having to set their value explicitly is a *good thing*. If I don't know the underlying value, I'm not tempted to do this: some_func(42, unit=1) # should be unit=FEET But if I had to use something like this: class units(Enum): INCHES = val() FEET = val() FURLONGS = val() MILES = val() or this: class relationship(Enum): PARENT, SIBLING, CHILD = range(3) I'd probably wouldn't bother. I'd stick to "poor man's enum", like this: PARENT, SIBLING, CHILD = range(3) or (to my shame, but I have to be realistic) magic constants. -- Steven From markus at unterwaditzer.net Tue Feb 12 23:50:01 2013 From: markus at unterwaditzer.net (Markus Unterwaditzer) Date: Tue, 12 Feb 2013 23:50:01 +0100 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> Message-ID: Guido van Rossum wrote: >I'm torn. I like the clean look of Tim's: > >class Flag(Enum): > RED, WHITE, BLUE > >However, I don't like the magic needed for its implementation -- and >anybody who thinks they know Python pretty well and thinks about this >syntax for more than two seconds has to wonder how on earth it's done >(and the answer is pretty horrible). It is also pretty brittle to >depend on the lookup order -- and I suspect it will be prone to >masking other bugs (any typo in a name used in class-level code will >essentially generate a bogus new enum value). > >OTOH I don't like having the string quotes of Antoine's >counter-proposal: > >class Flag(Enum): > fields = ('RED', 'WHITE', 'BLUE') > >Whenever I see quoted names that are also used unquoted, I cringe a >little (and this includes calls to hasattr() or getattr() with a >string literal for the name). > >The compromise that I would personally lean towards is more like this: > >class Flag(Enum): > RED, WHITE, BLUE = > >Even if the requires you to know how many values you are >defining, like range(3). If we have to have a somewhat more verbose >syntax that doesn't bother me too much. > >FWIW I do like being able to define enums that are ints and strings >(but not mixed); masks/flags I see more as a special case of ints (if >you use this you are likely trying to match some API defined in C or >C++). I also agree that it must be possible to customize the enum >class and the behavior of the values by defining methods on the class. > >-- >--Guido van Rossum (python.org/~guido) >_______________________________________________ >Python-ideas mailing list >Python-ideas at python.org >http://mail.python.org/mailman/listinfo/python-ideas Forgive my naiveness, but why do we need a new type for enums? Wouldn't a new builtin function that returns a dict suffice? Something like: def enum(*keys): return dict((k, i) for i, k in enumerate(keys)) loglevels = enum('debug', 'warning', 'error') -- Markus From guido at python.org Tue Feb 12 23:56:45 2013 From: guido at python.org (Guido van Rossum) Date: Tue, 12 Feb 2013 14:56:45 -0800 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: <511AC786.80807@pearwood.info> References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <20130212144518.3be491c5@anarchist.wooz.org> <511AC786.80807@pearwood.info> Message-ID: Frankly, enums are not that useful in small programs. For large programs or libraries, and especially for public APIs, the extra cost of defining the enum shouldn't count against them. Let's just import Barry's enums into the stdlib. -- --Guido van Rossum (python.org/~guido) From guido at python.org Tue Feb 12 23:58:15 2013 From: guido at python.org (Guido van Rossum) Date: Tue, 12 Feb 2013 14:58:15 -0800 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> Message-ID: On Tue, Feb 12, 2013 at 2:50 PM, Markus Unterwaditzer wrote: > Forgive my naiveness, but why do we need a new type for enums? Wouldn't a new builtin function that returns a dict suffice? Something like: > > def enum(*keys): > return dict((k, i) for i, k in enumerate(keys)) > > loglevels = enum('debug', 'warning', 'error') Because you shouldn't have to put the enum names in quotes. -- --Guido van Rossum (python.org/~guido) From steve at pearwood.info Wed Feb 13 00:00:38 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 13 Feb 2013 10:00:38 +1100 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: <20130212220412.4bd3e604@pitrou.net> References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <511A8A82.4010602@stoneleaf.us> <20130212220412.4bd3e604@pitrou.net> Message-ID: <511AC996.3080008@pearwood.info> On 13/02/13 08:04, Antoine Pitrou wrote: > I still hope enum values are strings by default - or, if not, that they > get nice reprs. 99.9% of the time I don't care what value enums have, in fact it is an advantage if I don't know. Pascal, as I remember it, doesn't give you any way to find out. But I consider enums displaying as their name to be essential to an enum. -- Steven From jonathan at slenders.be Wed Feb 13 00:01:41 2013 From: jonathan at slenders.be (Jonathan Slenders) Date: Wed, 13 Feb 2013 00:01:41 +0100 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <511A8A82.4010602@stoneleaf.us> <20130212214655.GA20475@sleipnir.bytereef.org> Message-ID: Just quickly thinking a little out of the box. Suppose, that we added grammar for scheme-like symbols. (You know, these atomic identifiers, prefixed with a quote. I don't know much about scheme, but as I understand, they are just constants.) In Python, the syntax could be similar, we could simply do in a global module scope: sym symbol which would translate to: locals()['symbol'] = Symbol('symbol') Where Symbol is a built-in that cannot change, which doesn't have a content, except for its name, and never equals anything but itself. From then on, we can access it through the name-binding 'symbol' Then we could probably create enums, using the classes as we have: class Colors: sym red sym green sym blue There are probably some gaps here in my reasoning, but this could also work. Cheers, Jonathan 2013/2/12 Tim Delaney > On 13 February 2013 09:29, Jonathan Slenders wrote: > >> Is a new enum keyword still a possibility? To me that seems to be the >> >>> cleanest way. >> >> >> I was just thinking the same. We don't like to have more syntax than >> necessary, but we should also not introduce too much magic... >> > > That's my conclusion as well. To get a nice* enum with guaranteed > semantics and no edge cases, I think it needs parser support. > > * for my definition of "nice". > > Tim Delaney > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eliben at gmail.com Wed Feb 13 00:06:36 2013 From: eliben at gmail.com (Eli Bendersky) Date: Tue, 12 Feb 2013 15:06:36 -0800 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <20130212144518.3be491c5@anarchist.wooz.org> <511AC786.80807@pearwood.info> Message-ID: On Tue, Feb 12, 2013 at 2:56 PM, Guido van Rossum wrote: > Frankly, enums are not that useful in small programs. For large > programs or libraries, and especially for public APIs, the extra cost > of defining the enum shouldn't count against them. > > Let's just import Barry's enums into the stdlib. > SGTM. So no need to write a PEP? Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Wed Feb 13 00:08:19 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 13 Feb 2013 10:08:19 +1100 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: <20130212145036.32a98928@anarchist.wooz.org> References: <20130212163102.50dbf570@pitrou.net> <20130212145036.32a98928@anarchist.wooz.org> Message-ID: <511ACB63.7030901@pearwood.info> On 13/02/13 06:50, Barry Warsaw wrote: > On Feb 12, 2013, at 04:05 PM, Joao S. O. Bueno wrote: > >> That is too much ' ' typing - I think it would be ok, to have it like that, >> but a helper function that would work just like the namedtuple call: Color = >> Enum("Color", "RED GREEN BLUE", int) > > How about: > > >>> from flufl.enum import make > >>> make('Colors', 'RED GREEN BLUE'.split()) > That reminds me of namedtuple. I could live with something close to that. Like namedtuple, it should do the split for you. -- Steven From timothy.c.delaney at gmail.com Wed Feb 13 00:09:05 2013 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Wed, 13 Feb 2013 10:09:05 +1100 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <20130212144518.3be491c5@anarchist.wooz.org> <511AC786.80807@pearwood.info> Message-ID: On 13 February 2013 09:56, Guido van Rossum wrote: > Frankly, enums are not that useful in small programs. For large > programs or libraries, and especially for public APIs, the extra cost > of defining the enum shouldn't count against them. > > Let's just import Barry's enums into the stdlib. That's entirely your call. FWIW I probably won't use them, as they fail to meet my needs for an enum, #1 being not having to specify their values in any way if I don't want to. Once you get past 3 or 4 values, it's too easy to miss or reuse a value. Tim Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Wed Feb 13 00:09:39 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 13 Feb 2013 10:09:39 +1100 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <20130212144518.3be491c5@anarchist.wooz.org> <511A9EE6.5070803@stoneleaf.us> Message-ID: <511ACBB3.2050509@pearwood.info> On 13/02/13 07:38, Yuval Greenfield wrote: > On Tue, Feb 12, 2013 at 9:58 PM, Ethan Furman wrote: > >> 1) Magic is fun. :) >> > > I heartily agree. Check this out: > >>>> RED, GREEN, BLUE = enum() >>>> print(RED, GREEN, BLUE) > 0 1 2 >>>> LEFT, TOP, RIGHT, DOWN = enum() >>>> print(LEFT, TOP, RIGHT, DOWN) > 0 1 2 3 > > > And here's the dirty trick.... [snip code] Awesome! :-) -- Steven From markus at unterwaditzer.net Wed Feb 13 00:09:49 2013 From: markus at unterwaditzer.net (Markus Unterwaditzer) Date: Wed, 13 Feb 2013 00:09:49 +0100 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> Message-ID: Guido van Rossum wrote: >On Tue, Feb 12, 2013 at 2:50 PM, Markus Unterwaditzer > wrote: >> Forgive my naiveness, but why do we need a new type for enums? >Wouldn't a new builtin function that returns a dict suffice? Something >like: >> >> def enum(*keys): >> return dict((k, i) for i, k in enumerate(keys)) >> >> loglevels = enum('debug', 'warning', 'error') > >Because you shouldn't have to put the enum names in quotes. Oh, I thought you were just against quoted names if they were also accessible without those: >Whenever I see quoted names that are also used unquoted, I cringe a little (and this includes calls to hasattr() or getattr() with a string literal for the name). -- Markus From guido at python.org Wed Feb 13 00:12:02 2013 From: guido at python.org (Guido van Rossum) Date: Tue, 12 Feb 2013 15:12:02 -0800 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> Message-ID: Well but you will use them as loglevels.debug, right? On Tue, Feb 12, 2013 at 3:09 PM, Markus Unterwaditzer wrote: > Guido van Rossum wrote: > >>On Tue, Feb 12, 2013 at 2:50 PM, Markus Unterwaditzer >> wrote: >>> Forgive my naiveness, but why do we need a new type for enums? >>Wouldn't a new builtin function that returns a dict suffice? Something >>like: >>> >>> def enum(*keys): >>> return dict((k, i) for i, k in enumerate(keys)) >>> >>> loglevels = enum('debug', 'warning', 'error') >> >>Because you shouldn't have to put the enum names in quotes. > > Oh, I thought you were just against quoted names if they were also accessible without those: > >>Whenever I see quoted names that are also used unquoted, I cringe a little (and this includes calls to hasattr() or getattr() with a string literal for the name). > -- Markus -- --Guido van Rossum (python.org/~guido) From ethan at stoneleaf.us Wed Feb 13 00:14:29 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 12 Feb 2013 15:14:29 -0800 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <20130212144518.3be491c5@anarchist.wooz.org> <511AC786.80807@pearwood.info> Message-ID: <511ACCD5.90000@stoneleaf.us> On 02/12/2013 03:09 PM, Tim Delaney wrote: > On 13 February 2013 09:56, Guido van Rossum wrote: > > Frankly, enums are not that useful in small programs. For large > programs or libraries, and especially for public APIs, the extra cost > of defining the enum shouldn't count against them. > > Let's just import Barry's enums into the stdlib. > > > That's entirely your call. FWIW I probably won't use them, as they fail > to meet my needs for an enum, #1 being not having to specify their > values in any way if I don't want to. Once you get past 3 or 4 values, > it's too easy to miss or reuse a value. +1 From greg.ewing at canterbury.ac.nz Wed Feb 13 00:26:21 2013 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 13 Feb 2013 12:26:21 +1300 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: <511AC996.3080008@pearwood.info> References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <511A8A82.4010602@stoneleaf.us> <20130212220412.4bd3e604@pitrou.net> <511AC996.3080008@pearwood.info> Message-ID: <511ACF9D.4090807@canterbury.ac.nz> Steven D'Aprano wrote: > 99.9% of the time I don't care what value enums have, in fact it is an > advantage if I don't know. Pascal, as I remember it, doesn't give you any > way to find out. Yes, it does: the ord() function. > But I consider enums displaying as their name to be > essential to an enum. That's one thing that, frustratingly, standard Pascal *doesn't* give you (although many dialects have extensions that do). -- Greg From g.brandl at gmx.net Wed Feb 13 00:57:38 2013 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 13 Feb 2013 00:57:38 +0100 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: <20130212144518.3be491c5@anarchist.wooz.org> References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <20130212144518.3be491c5@anarchist.wooz.org> Message-ID: Am 12.02.2013 20:45, schrieb Barry Warsaw: > On Feb 12, 2013, at 09:07 AM, Ethan Furman wrote: > >>The result of the implementation I've been playing with looks something like: >> >>class Color(Enum): >> type = 'sequence' >> RED, GREEN, BLUE >> >>class Geometry(Enum): >> type = 'unique' >> LEFT, RIGHT, TOP, BOTTOM >> WEST, EAST, NORTH, SOUTH >> >>class FieldType(Enum): >> type = 'flag' >> BINARY, AUTOINC, NULLABLE I am also against this. It requires special support and heuristics in static analysis tools in order not to flag these as errors. It will be annoying. > IMHO, none of these approach the simplicity and explicitness of this API: > > class Color(Enum): > RED = 1 > GREEN = 2 > BLUE = 3 We could even allow class Color(Enum): RED = 1 GREEN = ... # becomes 2 BLUE = ... # 3 MAGENTA = 5 FLUFL = ... # 6 class Color(FlagEnum): RED = 1 GREEN = ... # becomes 2 BLUE = ... # 4 MAGENTA = 16 FLUFL = ... class Color(StringEnum): RED = ... # becomes 'red' GREEN = ... # etc. BLUE = ... Georg From g.brandl at gmx.net Wed Feb 13 00:59:08 2013 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 13 Feb 2013 00:59:08 +0100 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: <511ACB63.7030901@pearwood.info> References: <20130212163102.50dbf570@pitrou.net> <20130212145036.32a98928@anarchist.wooz.org> <511ACB63.7030901@pearwood.info> Message-ID: Am 13.02.2013 00:08, schrieb Steven D'Aprano: > On 13/02/13 06:50, Barry Warsaw wrote: >> On Feb 12, 2013, at 04:05 PM, Joao S. O. Bueno wrote: >> >>> That is too much ' ' typing - I think it would be ok, to have it like that, >>> but a helper function that would work just like the namedtuple call: Color = >>> Enum("Color", "RED GREEN BLUE", int) >> >> How about: >> >> >>> from flufl.enum import make >> >>> make('Colors', 'RED GREEN BLUE'.split()) >> > > > That reminds me of namedtuple. I could live with something close to that. > > Like namedtuple, it should do the split for you. And like namedtuple, it's ugly as hell :) Georg From breamoreboy at yahoo.co.uk Wed Feb 13 01:11:33 2013 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Wed, 13 Feb 2013 00:11:33 +0000 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <511A8A82.4010602@stoneleaf.us> <20130212214655.GA20475@sleipnir.bytereef.org> Message-ID: On 12/02/2013 22:32, Tim Delaney wrote: > > * for my definition of "nice". Neurotic, Insecure, Crazy, Emotional? :) > > Tim Delaney > -- Cheers. Mark Lawrence From ericsnowcurrently at gmail.com Wed Feb 13 02:07:31 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Tue, 12 Feb 2013 18:07:31 -0700 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <20130212144518.3be491c5@anarchist.wooz.org> Message-ID: On Tue, Feb 12, 2013 at 4:57 PM, Georg Brandl wrote: > We could even allow > > class Color(Enum): > RED = 1 > GREEN = ... # becomes 2 > BLUE = ... # 3 > MAGENTA = 5 > FLUFL = ... # 6 > > class Color(FlagEnum): > RED = 1 > GREEN = ... # becomes 2 > BLUE = ... # 4 > MAGENTA = 16 > FLUFL = ... > > class Color(StringEnum): > RED = ... # becomes 'red' > GREEN = ... # etc. > BLUE = ... Nice. Both the use of ellipsis and having the semantics depend directly on which specific class you inherit from. At the same time, Steven D'Aprano's proposition of "the underlying value is irrelevant" and "must be bitwise comparable" appeals to me. If the underlying values matter then I'd call it a "constant" rather than an "enum". A proper enum type would remove the temptation to use the underlying value. I would say that there is room for both an enum type and one or more constants type (e.g. what Georg demonstrated above). Regardless, this discussion would be well-served by recognizing the distinction. -eric From timothy.c.delaney at gmail.com Wed Feb 13 02:45:24 2013 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Wed, 13 Feb 2013 12:45:24 +1100 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <511A8A82.4010602@stoneleaf.us> Message-ID: On 13 February 2013 07:13, Tim Delaney wrote: > Well, that particular case would work (you'd get a NameError: sys) due to > having done an attribute lookup on sys, but the following: > > class Color(Enum): > RED, GREEN, BLUE > if platfor == 'win32': > MAGENTA > > would create an enum value 'platfor'. > BTW, I've just addressed this in my latest version up on BitBucket. Any usage of an object that is more than just assignment (basically, anything that results in a call of a __dunder__ method on the object) will now mark the name as not an enum value and result in a NameError for an unknown name. Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:57:17) [MSC v.1600 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> from enum import Enum >>> class Color(Enum): ... RED ... if unknown_name == 'value': ... PINK ... Traceback (most recent call last): File "", line 1, in File "", line 3, in Color File ".\enum.py", line 123, in __eq__ def __eq__(self, other): return self._get(True) == other File ".\enum.py", line 98, in _get raise NameError(self.key) NameError: unknown_name Tim Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: From timothy.c.delaney at gmail.com Wed Feb 13 02:51:14 2013 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Wed, 13 Feb 2013 12:51:14 +1100 Subject: [Python-ideas] constant/enum type in stdlib In-Reply-To: References: <51085AAB.6090303@canterbury.ac.nz> <5108A87D.9000207@canterbury.ac.nz> <20130130082639.0b28d7eb@pitrou.net> <20130201103656.0b24c40d@anarchist.wooz.org> Message-ID: On 13 February 2013 08:36, Chris Angelico wrote: > On Tue, Feb 12, 2013 at 7:06 PM, Tim Delaney > wrote: > > This actually makes a kind of perverse sense. Conceptually the actual > > EnumValues are the instances of the Enum (and in fact, I've made them > > pretend to actually be so). > > Not so perverse. I think that makes very good sense (but then, I know > C++ enums, so maybe I'm tainted). Looks good, though I've not actually > played with the code. > And more ... Firstly, handles more edge cases that might have been mis-identified as enum values. EnumParams is now redundant (and will probably be removed), replaced by calling the (undefined) enum value name. Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:57:17) [MSC v.1600 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> from enum import Enum >>> class MyEnum(Enum): ... A, B ... C(None, 'c1', c2='2') ... D(10, 'd') ... E(None, e='e') ... >>> repr(MyEnum) ", , , , }>" >>> str(MyEnum) '{A:0, B:1, C:2, D:10, E:11}' >>> MyEnum.A.args, MyEnum.A.kwargs ((), {}) >>> MyEnum.C.args, MyEnum.C.kwargs (('c1',), {'c2': '2'}) Tim Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: From timothy.c.delaney at gmail.com Wed Feb 13 03:20:31 2013 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Wed, 13 Feb 2013 13:20:31 +1100 Subject: [Python-ideas] constant/enum type in stdlib In-Reply-To: References: <51085AAB.6090303@canterbury.ac.nz> <5108A87D.9000207@canterbury.ac.nz> <20130130082639.0b28d7eb@pitrou.net> <20130201103656.0b24c40d@anarchist.wooz.org> Message-ID: On 13 February 2013 12:51, Tim Delaney wrote: > And more ... > And inspired by the other thread, Ellipsis (...) is now used to indicate "use the next value". Ellipsis can either be assigned to an enum value, or passed as the first parameter to EnumParams or when calling the enum value during construction. None is no longer valid. Python 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:57:17) [MSC v.1600 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> from enum import Enum >>> class MyEnum(Enum): ... A ... B, C, D = ..., 5, ... ... E = ... ... F(..., 'f') ... >>> MyEnum , , , , , }> I'm stopping now. Tim Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Wed Feb 13 00:19:16 2013 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 13 Feb 2013 12:19:16 +1300 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: <511A8A82.4010602@stoneleaf.us> References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <511A8A82.4010602@stoneleaf.us> Message-ID: <511ACDF4.4040107@canterbury.ac.nz> Ethan Furman wrote: > If you don't mind verbose, we could always go for something like: > > class Color(Enum): > start_defs() > BLACK > RED, GREEN, BLUE > CYAN, YELLOW, MAGENTA > end_defs() Or class Color(Enum): values(BLACK, RED, GREEN, BLUE, CYAN, YELLOW, MAGENTA) where values() ignores its arguments and ends the definition phase. This would also provide a way to avoid relying on language guarantees of evaluation order. The initial lookup of the names could assign arbitrary values to them, and then values() could remap them to sequential values based on the order it receives them in. -- Greg From raymond.hettinger at gmail.com Wed Feb 13 04:25:53 2013 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Tue, 12 Feb 2013 19:25:53 -0800 Subject: [Python-ideas] constant/enum type in stdlib In-Reply-To: References: <51085AAB.6090303@canterbury.ac.nz> <5108A87D.9000207@canterbury.ac.nz> <20130130082639.0b28d7eb@pitrou.net> <20130201103656.0b24c40d@anarchist.wooz.org> Message-ID: <71DF1DA7-06D9-42F2-8E7A-B92732C4C424@gmail.com> On Feb 11, 2013, at 5:28 AM, Eli Bendersky wrote: > Can you elaborate on the utility of this feature? What realistic use cases do you see for it? I think that at this point it's important to weigh all benefits of features vs. implementation complexity, and there's absolutely no need to support every feature every other enum implementation has. I want to stress again that the most important characteristic of your implementation is the clean syntax which means that enums are so easy to define they don't really need special Python syntax and a library feature can do. However, there's a big leap from this to defining custom metaclasses for enums. Well said. I agree with you critique. In the absence of compelling use cases, the language is better-off without a complicated new feature. Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From timothy.c.delaney at gmail.com Wed Feb 13 04:38:27 2013 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Wed, 13 Feb 2013 14:38:27 +1100 Subject: [Python-ideas] constant/enum type in stdlib In-Reply-To: <71DF1DA7-06D9-42F2-8E7A-B92732C4C424@gmail.com> References: <51085AAB.6090303@canterbury.ac.nz> <5108A87D.9000207@canterbury.ac.nz> <20130130082639.0b28d7eb@pitrou.net> <20130201103656.0b24c40d@anarchist.wooz.org> <71DF1DA7-06D9-42F2-8E7A-B92732C4C424@gmail.com> Message-ID: On 13 February 2013 14:25, Raymond Hettinger wrote: > > On Feb 11, 2013, at 5:28 AM, Eli Bendersky wrote: > > Can you elaborate on the utility of this feature? What realistic use cases > do you see for it? I think that at this point it's important to weigh all > benefits of features vs. implementation complexity, and there's absolutely > no need to support every feature every other enum implementation has. I > want to stress again that the most important characteristic of your > implementation is the clean syntax which means that enums are so easy to > define they don't really need special Python syntax and a library feature > can do. However, there's a big leap from this to defining custom > metaclasses for enums. > > > Well said. I agree with you critique. > In the absence of compelling use cases, > the language is better-off without a complicated > new feature. > Absolutely. At the moment my implementation has everything in there as I've been incrementally finding syntactically cleaner ways of doing things. I intend to remove a lot of the extra functionality eventually - for example, supporting EnumMeta/EnumValue subclassing. In any case, it looks like Guido is strongly heading towards flufl.enum - doesn't mean I won't keep working on my implementation, but it doesn't look like it will be the basis for a stdlib enum. Tim Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsbueno at python.org.br Wed Feb 13 04:41:12 2013 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Wed, 13 Feb 2013 01:41:12 -0200 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <511A8A82.4010602@stoneleaf.us> Message-ID: On 12 February 2013 19:29, Guido van Rossum wrote: > Please. Anything *except* > > class Color(Enum): > RED, BLUE, GREEN > > And not this eiter: > > class Color(Enum): > RED > BLUE > GREEN Just throwing in another syntax that is valid, requires no quotes and no magic - in case someone is interested in developing on it: >>> Colors = make(RED=..., GREEN=..., BLUE=...) From stephen at xemacs.org Wed Feb 13 04:41:20 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 13 Feb 2013 12:41:20 +0900 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: <20130212214655.GA20475@sleipnir.bytereef.org> References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <511A8A82.4010602@stoneleaf.us> <20130212214655.GA20475@sleipnir.bytereef.org> Message-ID: <87a9r8yj6n.fsf@uwakimon.sk.tsukuba.ac.jp> Stefan Krah writes: > Guido van Rossum wrote: > > Please. Anything *except* > > > > class Color(Enum): > > RED, BLUE, GREEN I'll take you at your word. But maybe you should stop reading now. > Is a new enum keyword still a possibility? To me that seems to be the > cleanest way. If parser support were on the table, couldn't we avoid a new keyword by aping the Pascal syntax with class Color(Enum) = [RED, GREEN, BLUE] or something like that? I realize this is horrible because of the missing colon, and because availability of the syntax depends on the base class. Maybe it will suggest something to somebody, though. The base class seems to be needed to support variants like OrderedEnum and FlagEnum, as the implicit Enum class Color = [RED, GREEN, BLUE] can't disambiguate common use cases (ordered enums, or-able flags). Another possibility would be to allow the list of identifiers as the base class: class Color([RED, GREEN, BLUE]): pass I suppose this is better than the version using "=", which doesn't seem to allow explicit initialization of the enumerated ids, or operations on them, in the class body. Perhaps the Enum base class should be explicit: class Color(Enum, [RED, GREEN, BLUE]): pass In all cases the point of using lists instead of (implicit) tuples is to clearly guarantee order of evaluation to support OrderedEnum. Sets are out because FlagEnum doesn't need such support. From stephen at xemacs.org Wed Feb 13 04:14:13 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 13 Feb 2013 12:14:13 +0900 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <20130212144518.3be491c5@anarchist.wooz.org> <511AC786.80807@pearwood.info> Message-ID: <87boboykfu.fsf@uwakimon.sk.tsukuba.ac.jp> Guido van Rossum writes: > Frankly, enums are not that useful in small programs. This seems like a very personal thing to me. In some sense that I suppose can be made as objective as counting keystrokes[1], I have to agree with you. But I don't feel that Steven's cringing at "magic literals" is a less valid argument for supporting a particular syntax than your cringing at "quoted identifier names". Of course, technically speaking, Steven could perfectly well use "explicit-valued" enums[2] to eliminate magic literals. But from a beautiful language perspective, ISTM that he is just one among many to advocate that an enum *definition* should look like an *unordered list of identifiers*.[3] It's not about keystroke count, it's about correspondence of syntax to semantics. He's not doing himself any favors by replacing a few magic literals with a hard to read, overdefined enum declaration. I strongly suspect that if you put explicit-valued enums into the stdlib now, you will find that there remains a strong demand for "nice" enums as Tim calls them (although most RFEs will probably use the word "real"!) Footnotes: [1] But far more important, I grant you that. ;-) [2] Note that EIBTI is *not* an argument for explicit-valued enums, because an enum elements's name and class should tell you everything you need to know about its semantics. If it doesn't, you aren't using it as an enum! And "YELLOW = val()" is hardly explicit. :-( [3] If that "list" of names actually has more structure (a set that allows subsetting and union as commonly done with flags, or an order such as the spectrum of colors), it would be nice if that structure is imposed automatically as well -- but this should be done by a subclass such as OrderedEnum or FlagEnum which makes the structure explicit. From ncoghlan at gmail.com Wed Feb 13 05:07:48 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 13 Feb 2013 14:07:48 +1000 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <20130212144518.3be491c5@anarchist.wooz.org> <511AC786.80807@pearwood.info> Message-ID: On 13 Feb 2013 09:11, "Tim Delaney" wrote: > > On 13 February 2013 09:56, Guido van Rossum wrote: >> >> Frankly, enums are not that useful in small programs. For large >> programs or libraries, and especially for public APIs, the extra cost >> of defining the enum shouldn't count against them. >> >> Let's just import Barry's enums into the stdlib. > > > That's entirely your call. FWIW I probably won't use them, as they fail to meet my needs for an enum, #1 being not having to specify their values in any way if I don't want to. Once you get past 3 or 4 values, it's too easy to miss or reuse a value. What's wrong with enum.make? That just accepts the sequence of names namedtuple style, no values specified anywhere. Cheers, Nick. > > Tim Delaney > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Wed Feb 13 06:04:39 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 12 Feb 2013 21:04:39 -0800 Subject: [Python-ideas] another enum implementation Message-ID: <511B1EE7.4060408@stoneleaf.us> I realize this isn't going very far, but I would still appreciate feedback. :) The code is here: https://bitbucket.org/stoneleaf/enum It is based on ideas from Michael Foord, Antoine Pitrou, and (eventually) Ted Delaney. Enum only accepts upper-case names as enum candidates; all enum values must be the same: either 'sequence' (0, 1, 2, etc) or 'flag' (1, 2, 4, etc.) or 'unique' (north, south, east, west, etc.); and extending via subclassing is possible. Sample code: class Color(Enum): type = 'sequence' BLACK RED GREEN = 4 BLUE print Color # Color(BLACK:0, RED:1, GREEN:4, BLUE:5) class MoreColor(Color): MAGENTA YELLOW CYAN print MoreColor # MoreColor(BLACK:0, RED:1, GREEN:4, BLUE:5, MAGENTA:6, YELLOW:7, # CYAN:8) class DbfType(Enum): type = 'unique' CLP DB3 VFP --> print(enum.DbfType) # DbfType(DB3:'db3', CLP:'clp', VFP:'vfp') class SomeFlags(Enum): type = 'flag' ON_DISK HAS_MEMO LARGE_CHAR UNICODE --> print(enum.SomeFlags) # SomeFlags(ON_DISK:1, HAS_MEMO:2, LARGE_CHAR:4, UNICODE:8) class Error(Enum): type = 'sequence' THIS THAT The_Other # raises NameError THOSE = 1 # raises InvalidEnum -->enum.Color.RED Color('RED') -->enum.Color.RED == 1 True -->int(enum.Color.RED) 1 -->enum.SomeFlags.ON_DISK SomeFlags('ON_DISK') -->enum.SomeFlags.ON_DISK == 1 True -->int(enum.SomeFlags.ON_DISK) 1 -->enum.SomeFlags.ON_DISK == enum.Color.RED False -->enum.MoreColor.RED == 1 True -->enum.MoreColor.RED == enum.Color.RED True -->enum.Color(1) Color('RED') -->enum.Color('RED') Color('RED') --> for color in enum.Color: ... print(color) ... BLACK RED GREEN BLUE --> for color in enum.Color: ... print(repr(color)) ... Color('BLACK') Color('RED') Color('GREEN') Color('BLUE') I would appreciate any comments on both the API, and the code behind it. Thanks. -- ~Ethan~ From yselivanov.ml at gmail.com Wed Feb 13 06:24:15 2013 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Wed, 13 Feb 2013 00:24:15 -0500 Subject: [Python-ideas] another enum implementation In-Reply-To: <511B1EE7.4060408@stoneleaf.us> References: <511B1EE7.4060408@stoneleaf.us> Message-ID: <89705A92-071D-42D2-A9DF-ABBEDF9A3E7C@gmail.com> I'm not sure why, but I didn't see anyone making a point about documentation. To me, it's important that stdlib's and other libraries' enums are well documented, and I think that the best way of providing and enforcing this is to mimic "property" builtin. class Color(enum.Sequence): "Some general docs about Color enum" BLACK = enum.value(doc='Black Color') or class Color(enum.Flag): "..." BLACK = enum(doc='Black Color') WHITE = enum() Maybe this is uglier than magic of just writing flags' names, but this implementation is much simpler, backwards compatible with python 2, and verbose. The latter point is both good and bad, but we don't write enums that often to make a bit of verbosity unacceptable. Just my two cents. - Yury On 2013-02-13, at 12:04 AM, Ethan Furman wrote: > I realize this isn't going very far, but I would still appreciate feedback. :) > > The code is here: > https://bitbucket.org/stoneleaf/enum > > It is based on ideas from Michael Foord, Antoine Pitrou, and (eventually) Ted Delaney. > > > Enum only accepts upper-case names as enum candidates; all enum values must be the same: either 'sequence' (0, 1, 2, etc) or 'flag' (1, 2, 4, etc.) or 'unique' (north, south, east, west, etc.); and extending via subclassing is possible. > > Sample code: > > class Color(Enum): > type = 'sequence' > BLACK > RED > GREEN = 4 > BLUE > print Color > # Color(BLACK:0, RED:1, GREEN:4, BLUE:5) > > class MoreColor(Color): > MAGENTA > YELLOW > CYAN > print MoreColor > # MoreColor(BLACK:0, RED:1, GREEN:4, BLUE:5, MAGENTA:6, YELLOW:7, > # CYAN:8) > > > class DbfType(Enum): > type = 'unique' > CLP > DB3 > VFP > --> print(enum.DbfType) > # DbfType(DB3:'db3', CLP:'clp', VFP:'vfp') > > class SomeFlags(Enum): > type = 'flag' > ON_DISK > HAS_MEMO > LARGE_CHAR > UNICODE > --> print(enum.SomeFlags) > # SomeFlags(ON_DISK:1, HAS_MEMO:2, LARGE_CHAR:4, UNICODE:8) > > class Error(Enum): > type = 'sequence' > THIS > THAT > The_Other # raises NameError > THOSE = 1 # raises InvalidEnum > > -->enum.Color.RED > Color('RED') > > -->enum.Color.RED == 1 > True > > -->int(enum.Color.RED) > 1 > > -->enum.SomeFlags.ON_DISK > SomeFlags('ON_DISK') > > -->enum.SomeFlags.ON_DISK == 1 > True > > -->int(enum.SomeFlags.ON_DISK) > 1 > > -->enum.SomeFlags.ON_DISK == enum.Color.RED > False > > -->enum.MoreColor.RED == 1 > True > > -->enum.MoreColor.RED == enum.Color.RED > True > > -->enum.Color(1) > Color('RED') > > -->enum.Color('RED') > Color('RED') > > --> for color in enum.Color: > ... print(color) > ... > BLACK > RED > GREEN > BLUE > > --> for color in enum.Color: > ... print(repr(color)) > ... > Color('BLACK') > Color('RED') > Color('GREEN') > Color('BLUE') > > > I would appreciate any comments on both the API, and the code behind it. > > Thanks. > > -- > ~Ethan~ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas From senthil at uthcode.com Wed Feb 13 07:18:03 2013 From: senthil at uthcode.com (Senthil Kumaran) Date: Tue, 12 Feb 2013 22:18:03 -0800 Subject: [Python-ideas] another enum implementation In-Reply-To: <511B1EE7.4060408@stoneleaf.us> References: <511B1EE7.4060408@stoneleaf.us> Message-ID: Two comments. 1. When it is fully generalized, is type is the way to to specify the type of enums? Or there will be helper functions or classes that can specify the type of enum one desires. 2. Personally, I have resorted to using namedtuples for enum like behavior. How would this class benefit me? Any edification would be helpful. On Tue, Feb 12, 2013 at 9:04 PM, Ethan Furman wrote: > I realize this isn't going very far, but I would still appreciate feedback. > :) > > The code is here: > https://bitbucket.org/stoneleaf/enum > > It is based on ideas from Michael Foord, Antoine Pitrou, and (eventually) > Ted Delaney. > > > Enum only accepts upper-case names as enum candidates; all enum values must > be the same: either 'sequence' (0, 1, 2, etc) or 'flag' (1, 2, 4, etc.) or > 'unique' (north, south, east, west, etc.); and extending via subclassing is > possible. > > Sample code: > > class Color(Enum): > type = 'sequence' > BLACK > RED > GREEN = 4 > BLUE > print Color > # Color(BLACK:0, RED:1, GREEN:4, BLUE:5) > > class MoreColor(Color): > MAGENTA > YELLOW > CYAN > print MoreColor > # MoreColor(BLACK:0, RED:1, GREEN:4, BLUE:5, MAGENTA:6, YELLOW:7, > # CYAN:8) > > > class DbfType(Enum): > type = 'unique' > CLP > DB3 > VFP > --> print(enum.DbfType) > # DbfType(DB3:'db3', CLP:'clp', VFP:'vfp') > > class SomeFlags(Enum): > type = 'flag' > ON_DISK > HAS_MEMO > LARGE_CHAR > UNICODE > --> print(enum.SomeFlags) > # SomeFlags(ON_DISK:1, HAS_MEMO:2, LARGE_CHAR:4, UNICODE:8) > > class Error(Enum): > type = 'sequence' > THIS > THAT > The_Other # raises NameError > THOSE = 1 # raises InvalidEnum > > -->enum.Color.RED > Color('RED') > > -->enum.Color.RED == 1 > True > > -->int(enum.Color.RED) > 1 > > -->enum.SomeFlags.ON_DISK > SomeFlags('ON_DISK') > > -->enum.SomeFlags.ON_DISK == 1 > True > > -->int(enum.SomeFlags.ON_DISK) > 1 > > -->enum.SomeFlags.ON_DISK == enum.Color.RED > False > > -->enum.MoreColor.RED == 1 > True > > -->enum.MoreColor.RED == enum.Color.RED > True > > -->enum.Color(1) > Color('RED') > > -->enum.Color('RED') > Color('RED') > > --> for color in enum.Color: > ... print(color) > ... > BLACK > RED > GREEN > BLUE > > --> for color in enum.Color: > ... print(repr(color)) > ... > Color('BLACK') > Color('RED') > Color('GREEN') > Color('BLUE') > > > I would appreciate any comments on both the API, and the code behind it. > > Thanks. > > -- > ~Ethan~ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas From senthil at uthcode.com Wed Feb 13 07:35:33 2013 From: senthil at uthcode.com (Senthil Kumaran) Date: Tue, 12 Feb 2013 22:35:33 -0800 Subject: [Python-ideas] another enum implementation In-Reply-To: References: <511B1EE7.4060408@stoneleaf.us> Message-ID: On Tue, Feb 12, 2013 at 10:18 PM, Senthil Kumaran wrote: > Two comments. I should say, I responded without looking at the other thread. If the questions are relevant, please proceed, if not I shall find the answer the other thread which seem to have discussed this topic in greater detail. From tjreedy at udel.edu Wed Feb 13 07:45:54 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 13 Feb 2013 01:45:54 -0500 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <511A8A82.4010602@stoneleaf.us> <511A9764.2060902@stoneleaf.us> Message-ID: On 2/12/2013 3:43 PM, Joshua Landau wrote: > class Color(Enum): > values = "RED GREEN BLUE MAGENTA OTHER COLORS HERE TOO".split() > > so I'm OK with that, but then you're in the territory when you might as > well go with the method Barry suggested. A custom mapping can automatically do the split. This is an interesting compromise between no quotes and multiple quotes. -- Terry Jan Reedy From ericsnowcurrently at gmail.com Wed Feb 13 07:49:01 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Tue, 12 Feb 2013 23:49:01 -0700 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <20130212144518.3be491c5@anarchist.wooz.org> Message-ID: On Tue, Feb 12, 2013 at 6:07 PM, Eric Snow wrote: > At the same time, Steven D'Aprano's proposition of "the underlying > value is irrelevant" and "must be bitwise comparable" appeals to me. > If the underlying values matter then I'd call it a "constant" rather > than an "enum". A proper enum type would remove the temptation to use > the underlying value. Here's an implementation of what I'm talking about: http://code.activestate.com/recipes/578455/ Each enum value is simply an object and not backed by any underlying value. The bitwise operators are implemented. The implementation is a little rough, but hopefully gets the idea across. -eric From ethan at stoneleaf.us Wed Feb 13 08:01:30 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 12 Feb 2013 23:01:30 -0800 Subject: [Python-ideas] another enum implementation In-Reply-To: <89705A92-071D-42D2-A9DF-ABBEDF9A3E7C@gmail.com> References: <511B1EE7.4060408@stoneleaf.us> <89705A92-071D-42D2-A9DF-ABBEDF9A3E7C@gmail.com> Message-ID: <511B3A4A.2000608@stoneleaf.us> On 02/12/2013 09:24 PM, Yury Selivanov wrote: > I'm not sure why, but I didn't see anyone making a point about > documentation. To me, it's important that stdlib's and other > libraries' enums are well documented, and I think that the best > way of providing and enforcing this is to mimic "property" builtin. > > class Color(enum.Sequence): > "Some general docs about Color enum" > BLACK = enum.value(doc='Black Color') > > or > > class Color(enum.Flag): > "..." > BLACK = enum(doc='Black Color') > WHITE = enum() > > Maybe this is uglier than magic of just writing flags' names, > but this implementation is much simpler, backwards compatible with > python 2, and verbose. The latter point is both good and bad, but > we don't write enums that often to make a bit of verbosity unacceptable. > > Just my two cents. You definitely have a good point about documentation, and I'm going to add the functionality that Tim has in his which will allow other attributes on enum values (the difference with his is that mine are actual instances of their class so all values of one class will have the same attributes). However, for small enums, well chosen names should be documentation enough, so doc strings will not be required. Even more importantly, if I'm just trying something out I do not want to be forced to add stuff I don't need. -- ~Ethan~ From solipsis at pitrou.net Wed Feb 13 08:21:29 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 13 Feb 2013 08:21:29 +0100 Subject: [Python-ideas] PEP for enum library type? References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <20130212144518.3be491c5@anarchist.wooz.org> Message-ID: <20130213082129.5d917b00@pitrou.net> On Wed, 13 Feb 2013 00:57:38 +0100 Georg Brandl wrote: > > IMHO, none of these approach the simplicity and explicitness of this API: > > > > class Color(Enum): > > RED = 1 > > GREEN = 2 > > BLUE = 3 > > We could even allow > > class Color(Enum): > RED = 1 > GREEN = ... # becomes 2 > BLUE = ... # 3 > MAGENTA = 5 > FLUFL = ... # 6 The fact that the ellipsis variant requires *more* typing sounds a bit wrong to me. The common case should be the easiest to type. Regards Antoine. From g.brandl at gmx.net Wed Feb 13 08:34:21 2013 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 13 Feb 2013 08:34:21 +0100 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: <20130213082129.5d917b00@pitrou.net> References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <20130212144518.3be491c5@anarchist.wooz.org> <20130213082129.5d917b00@pitrou.net> Message-ID: Am 13.02.2013 08:21, schrieb Antoine Pitrou: > On Wed, 13 Feb 2013 00:57:38 +0100 > Georg Brandl wrote: >> > IMHO, none of these approach the simplicity and explicitness of this API: >> > >> > class Color(Enum): >> > RED = 1 >> > GREEN = 2 >> > BLUE = 3 >> >> We could even allow >> >> class Color(Enum): >> RED = 1 >> GREEN = ... # becomes 2 >> BLUE = ... # 3 >> MAGENTA = 5 >> FLUFL = ... # 6 > > The fact that the ellipsis variant requires *more* typing sounds a bit > wrong to me. The common case should be the easiest to type. Well, nobody forbids you from writing the integers out. But one of the complaints was that you shouldn't need to know or write the constant values (and "..." is shorter than "val()" which was proposed for that purpose). And for stringy enums that argument is void :) Georg From ethan at stoneleaf.us Wed Feb 13 07:38:11 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 12 Feb 2013 22:38:11 -0800 Subject: [Python-ideas] another enum implementation In-Reply-To: References: <511B1EE7.4060408@stoneleaf.us> Message-ID: <511B34D3.40009@stoneleaf.us> On 02/12/2013 10:18 PM, Senthil Kumaran wrote: > Two comments. > > 1. When it is fully generalized, is type is the way to to specify the > type of enums? Or there will be helper functions or classes that can > specify the type of enum one desires. Creating enums with the class method is so easy I don't see myself spending time on helper functions. I'll have to think some more about using type = 'unique' vs. class Geometry(UniqueEnum) although I feel myself leaning towards the UniqueEnum style. > 2. Personally, I have resorted to using namedtuples for enum like > behavior. How would this class benefit me? Any edification would be > helpful. namedtuples are designed to have many instances with different values in the same slots; the Enum class is a singleton, and its instances are the values. The Enum class' instances are derived from either int or str, so they can be an easy drop-in replacement to API's that currently expect ints or strs. Better repr()'s. :) -- ~Ethan~ P.S. Definitely read the other thread, and no worries. From solipsis at pitrou.net Wed Feb 13 08:38:43 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 13 Feb 2013 08:38:43 +0100 Subject: [Python-ideas] PEP for enum library type? References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <20130212144518.3be491c5@anarchist.wooz.org> <20130213082129.5d917b00@pitrou.net> Message-ID: <20130213083843.7dbc7598@pitrou.net> On Wed, 13 Feb 2013 08:34:21 +0100 Georg Brandl wrote: > Am 13.02.2013 08:21, schrieb Antoine Pitrou: > > On Wed, 13 Feb 2013 00:57:38 +0100 > > Georg Brandl wrote: > >> > IMHO, none of these approach the simplicity and explicitness of this API: > >> > > >> > class Color(Enum): > >> > RED = 1 > >> > GREEN = 2 > >> > BLUE = 3 > >> > >> We could even allow > >> > >> class Color(Enum): > >> RED = 1 > >> GREEN = ... # becomes 2 > >> BLUE = ... # 3 > >> MAGENTA = 5 > >> FLUFL = ... # 6 > > > > The fact that the ellipsis variant requires *more* typing sounds a bit > > wrong to me. The common case should be the easiest to type. > > Well, nobody forbids you from writing the integers out. But one of the > complaints was that you shouldn't need to know or write the constant > values (and "..." is shorter than "val()" which was proposed for that > purpose). It would still be shorter to type something like: class Color(Enum): values = ('RED', 'GREEN', 'BLUE', 'MAGENTA', 'FLUFL') > And for stringy enums that argument is void :) I'm not sure how it's void. What is the proposed syntax for string enums? Regards Antoine. From brett at python.org Wed Feb 13 17:26:26 2013 From: brett at python.org (Brett Cannon) Date: Wed, 13 Feb 2013 11:26:26 -0500 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <20130212144518.3be491c5@anarchist.wooz.org> <511AC786.80807@pearwood.info> Message-ID: On Tue, Feb 12, 2013 at 11:07 PM, Nick Coghlan wrote: > > On 13 Feb 2013 09:11, "Tim Delaney" wrote: > > > > On 13 February 2013 09:56, Guido van Rossum wrote: > >> > >> Frankly, enums are not that useful in small programs. For large > >> programs or libraries, and especially for public APIs, the extra cost > >> of defining the enum shouldn't count against them. > >> > >> Let's just import Barry's enums into the stdlib. > > > > > > That's entirely your call. FWIW I probably won't use them, as they fail > to meet my needs for an enum, #1 being not having to specify their values > in any way if I don't want to. Once you get past 3 or 4 values, it's too > easy to miss or reuse a value. > > What's wrong with enum.make? That just accepts the sequence of names > namedtuple style, no values specified anywhere. > > > Georg called it ugly, but that's the only complaint I remember reading. I too find them ugly but not enough to not say this isn't the best solution considering everyone's complaints of verbosity (e.g. Guido not liking excessive quoting), magic (e.g. all non-explicit proposals that don't involve calling a classmethod like val()), and conciseness (e.g. not having to specify every value). Heck, if it was me I would just do:: import types def make_enum(keys, *, value=None): """For incremental numbers: ``lambda key, index: index``. For bitwise flags: ``lambda key, index: 2**index``. Could change to take an iterator instead of a lambda and simply special-case the `None` argument for strings, but that seems needlessly limiting when the integer case is probably for interfacing with some C code. """ if value is None: value = lambda key, index: key items = {} for key, index in enumerate(keys.split()): items[key] = value(key, index) return types.SimpleNamespace(**items) # Creating a read-only subclass is trivial But I know people will want fancier reprs, globally unique objects, etc. instead of just some simple way to create an object with attributes referencing automatically-generated, unique, predictable values (which is all I would want from an enum). -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Wed Feb 13 17:38:07 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 13 Feb 2013 17:38:07 +0100 Subject: [Python-ideas] PEP for enum library type? References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <20130212144518.3be491c5@anarchist.wooz.org> <511AC786.80807@pearwood.info> Message-ID: <20130213173807.1465d42b@pitrou.net> Le Wed, 13 Feb 2013 14:07:48 +1000, Nick Coghlan a ?crit : > On 13 Feb 2013 09:11, "Tim Delaney" > wrote: > > > > On 13 February 2013 09:56, Guido van Rossum > > wrote: > >> > >> Frankly, enums are not that useful in small programs. For large > >> programs or libraries, and especially for public APIs, the extra > >> cost of defining the enum shouldn't count against them. > >> > >> Let's just import Barry's enums into the stdlib. > > > > > > That's entirely your call. FWIW I probably won't use them, as they > > fail > to meet my needs for an enum, #1 being not having to specify their > values in any way if I don't want to. Once you get past 3 or 4 > values, it's too easy to miss or reuse a value. > > What's wrong with enum.make? That just accepts the sequence of names > namedtuple style, no values specified anywhere. What's wrong is that TSBOOWTDI. With enum and enum.make, you have two different idioms for declaring enums, while a single class-based definition style should suffice. Regards Antoine. From brett at python.org Wed Feb 13 18:09:19 2013 From: brett at python.org (Brett Cannon) Date: Wed, 13 Feb 2013 12:09:19 -0500 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: <20130213173807.1465d42b@pitrou.net> References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <20130212144518.3be491c5@anarchist.wooz.org> <511AC786.80807@pearwood.info> <20130213173807.1465d42b@pitrou.net> Message-ID: On Wed, Feb 13, 2013 at 11:38 AM, Antoine Pitrou wrote: > Le Wed, 13 Feb 2013 14:07:48 +1000, > Nick Coghlan a > ?crit : > > On 13 Feb 2013 09:11, "Tim Delaney" > > wrote: > > > > > > On 13 February 2013 09:56, Guido van Rossum > > > wrote: > > >> > > >> Frankly, enums are not that useful in small programs. For large > > >> programs or libraries, and especially for public APIs, the extra > > >> cost of defining the enum shouldn't count against them. > > >> > > >> Let's just import Barry's enums into the stdlib. > > > > > > > > > That's entirely your call. FWIW I probably won't use them, as they > > > fail > > to meet my needs for an enum, #1 being not having to specify their > > values in any way if I don't want to. Once you get past 3 or 4 > > values, it's too easy to miss or reuse a value. > > > > What's wrong with enum.make? That just accepts the sequence of names > > namedtuple style, no values specified anywhere. > > What's wrong is that TSBOOWTDI. With enum and enum.make, you have two > different idioms for declaring enums, while a single class-based > definition style should suffice. > Or just a function idiom. We don't have to support the class-based definition style either. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ubershmekel at gmail.com Wed Feb 13 18:33:00 2013 From: ubershmekel at gmail.com (Yuval Greenfield) Date: Wed, 13 Feb 2013 19:33:00 +0200 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <20130212144518.3be491c5@anarchist.wooz.org> <511AC786.80807@pearwood.info> <20130213173807.1465d42b@pitrou.net> Message-ID: On Wed, Feb 13, 2013 at 7:09 PM, Brett Cannon wrote: > Or just a function idiom. We don't have to support the class-based > definition style either. > Well, namedtuple has an enum for tuple indexes. I think we should be consistent which entails: Colors = collections.enum('RED GREEN BLUE') Cheers, Yuval -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Wed Feb 13 19:00:33 2013 From: barry at python.org (Barry Warsaw) Date: Wed, 13 Feb 2013 13:00:33 -0500 Subject: [Python-ideas] PEP for enum library type? References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <20130212144518.3be491c5@anarchist.wooz.org> Message-ID: <20130213130033.61cb8aa6@anarchist.wooz.org> On Feb 13, 2013, at 12:57 AM, Georg Brandl wrote: >class Color(Enum): > RED = 1 > GREEN = ... # becomes 2 > BLUE = ... # 3 > MAGENTA = 5 > FLUFL = ... # 6 That's pretty cute. >class Color(FlagEnum): > RED = 1 > GREEN = ... # becomes 2 > BLUE = ... # 4 > MAGENTA = 16 > FLUFL = ... > >class Color(StringEnum): > RED = ... # becomes 'red' > GREEN = ... # etc. > BLUE = ... I want to think about the latter two (especially the string one), but I've filed this bug to track the feature. https://bugs.launchpad.net/flufl.enum/+bug/1124357 Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at python.org Wed Feb 13 19:04:29 2013 From: barry at python.org (Barry Warsaw) Date: Wed, 13 Feb 2013 13:04:29 -0500 Subject: [Python-ideas] PEP for enum library type? References: <20130212163102.50dbf570@pitrou.net> <20130212145036.32a98928@anarchist.wooz.org> <511ACB63.7030901@pearwood.info> Message-ID: <20130213130429.6e99dc94@anarchist.wooz.org> On Feb 13, 2013, at 10:08 AM, Steven D'Aprano wrote: >> >>> from flufl.enum import make >> >>> make('Colors', 'RED GREEN BLUE'.split()) >> > >That reminds me of namedtuple. I could live with something close to that. > >Like namedtuple, it should do the split for you. Part of the reason for defining make() the way it is, is so that you can use any iterator in the second argument. I suppose you could special case splitting if the second argument is a single string, but I'm not sure this special case is special enough. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at python.org Wed Feb 13 19:05:17 2013 From: barry at python.org (Barry Warsaw) Date: Wed, 13 Feb 2013 13:05:17 -0500 Subject: [Python-ideas] PEP for enum library type? References: <20130212163102.50dbf570@pitrou.net> <20130212145036.32a98928@anarchist.wooz.org> <511ACB63.7030901@pearwood.info> Message-ID: <20130213130517.2ac6afc1@anarchist.wooz.org> On Feb 13, 2013, at 12:59 AM, Georg Brandl wrote: >>> >>> from flufl.enum import make >>> >>> make('Colors', 'RED GREEN BLUE'.split()) >>> >And like namedtuple, it's ugly as hell :) Suggestions for a better repr are welcome. ;) -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at python.org Wed Feb 13 19:14:03 2013 From: barry at python.org (Barry Warsaw) Date: Wed, 13 Feb 2013 13:14:03 -0500 Subject: [Python-ideas] PEP for enum library type? References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <511A8A82.4010602@stoneleaf.us> <20130212220412.4bd3e604@pitrou.net> Message-ID: <20130213131403.3edece68@anarchist.wooz.org> On Feb 12, 2013, at 02:13 PM, Eli Bendersky wrote: >I agree, and Tim's implementation provides a very nice string >representation, no matter what the value underneath is. I don't really care >what the value underneath is, actually. Except that sometimes you do, which is why flufl.enum has a nice concise str and a more detailed repr. The former doesn't include the underlying value while the latter does. Edible cake. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at python.org Wed Feb 13 19:19:14 2013 From: barry at python.org (Barry Warsaw) Date: Wed, 13 Feb 2013 13:19:14 -0500 Subject: [Python-ideas] PEP for enum library type? References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <511A8A82.4010602@stoneleaf.us> Message-ID: <20130213131914.5a303f8b@anarchist.wooz.org> On Feb 13, 2013, at 08:19 AM, Tim Delaney wrote: >I'm of the opinion that the most fundamental property of an enum is that it >can generate its values automatically. Everything else is a nice-to-have, >but without that property it's not much use as an enum. See, I don't care about that much because I define an enum once, but I *use* it hundreds of times. Thus a little more verbosity/explicitness at definition time is a one-time cost, and a very low one at that. It might not seems that way when you're writing them in so many mailing list replies though ;). In real code, having a really good str and repr, along with good semantics makes them much more useful for debugging, and very pleasant to use. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at python.org Wed Feb 13 19:12:39 2013 From: barry at python.org (Barry Warsaw) Date: Wed, 13 Feb 2013 13:12:39 -0500 Subject: [Python-ideas] PEP for enum library type? References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <511A8A82.4010602@stoneleaf.us> <20130212220412.4bd3e604@pitrou.net> Message-ID: <20130213131239.421aaebd@anarchist.wooz.org> On Feb 12, 2013, at 10:04 PM, Antoine Pitrou wrote: >I still hope enum values are strings by default - or, if not, that they >get nice reprs. Integer enums are only useful for interoperability with >stubborn low-level libraries :-) >>> from flufl.enum import make >>> Colors = make('Colors', 'RED GREEN BLUE'.split()) >>> print(Colors.RED) Colors.RED >>> Colors.RED Having the enum name in the str and (admittedly uglier but more informational) repr is very important to me, because I typically use enums like so: from mailman.interfaces.members import DeliveryMode ... def send_plaintext(member): ... if member.delivery_mode is DeliveryMode.regular: ... elif member.delivery_mode is DeliveryMode.mime_digests: ... Now, if something goes wrong, I can set a breakpoint or debug-print right before the conditional and get the value of `member.delivery_mode`. The fact that it will print `DeliveryMode.regular` makes debugging the problem a *much* better experience. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From timothy.c.delaney at gmail.com Wed Feb 13 20:22:11 2013 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Thu, 14 Feb 2013 06:22:11 +1100 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: <20130213131403.3edece68@anarchist.wooz.org> References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <511A8A82.4010602@stoneleaf.us> <20130212220412.4bd3e604@pitrou.net> <20130213131403.3edece68@anarchist.wooz.org> Message-ID: On 14 February 2013 05:14, Barry Warsaw wrote: > On Feb 12, 2013, at 02:13 PM, Eli Bendersky wrote: > > >I agree, and Tim's implementation provides a very nice string > >representation, no matter what the value underneath is. I don't really > care > >what the value underneath is, actually. > > Except that sometimes you do, which is why flufl.enum has a nice concise > str > and a more detailed repr. The former doesn't include the underlying value > while the latter does. Edible cake. > Yes - my enum's exactly the same - simple, concise str (for both the enum and values) and detailed repr. The actual repr is a little different to flufl.enum repr (I think the str is the same) but conveys the same information. Color.RED {RED:0, GREEN:1, BLUE:2, CYAN:10, MAGENTA:11, YELLOW:12, BLACK:127} , , , , , , }> Tim Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: From timothy.c.delaney at gmail.com Wed Feb 13 20:28:37 2013 From: timothy.c.delaney at gmail.com (Tim Delaney) Date: Thu, 14 Feb 2013 06:28:37 +1100 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: <20130213131914.5a303f8b@anarchist.wooz.org> References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <511A8A82.4010602@stoneleaf.us> <20130213131914.5a303f8b@anarchist.wooz.org> Message-ID: On 14 February 2013 05:19, Barry Warsaw wrote: > On Feb 13, 2013, at 08:19 AM, Tim Delaney wrote: > > >I'm of the opinion that the most fundamental property of an enum is that > it > >can generate its values automatically. Everything else is a nice-to-have, > >but without that property it's not much use as an enum. > > See, I don't care about that much because I define an enum once, but I > *use* > it hundreds of times. Thus a little more verbosity/explicitness at > definition > time is a one-time cost, and a very low one at that. It might not seems > that > way when you're writing them in so many mailing list replies though ;). > My main concern regarding this is ensuring that: 1. Values don't get repeated. 2. Values don't get skipped (unless explicitly skipped); #1 obviously is the most important property, but in practice I find #2 to be very important as well. Whilst it's more verbose, the ... syntax isn't bad, and I've added support to it to my enum to play with. Currently my enum implementation includes the kitchen sink, but I'm looking to do some renovations soon. > In real code, having a really good str and repr, along with good semantics > makes them much more useful for debugging, and very pleasant to use. > Absolutely. But it's not just that either. If you can convert from the string name to the enum, it makes the enum suitable as a transport mechanism. Each end can have different values for the enum in practice so long as the names don't change - it can make things much more robust across versions. Tim Delaney -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Wed Feb 13 20:30:05 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 13 Feb 2013 20:30:05 +0100 Subject: [Python-ideas] PEP for enum library type? References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <511A8A82.4010602@stoneleaf.us> <20130213131914.5a303f8b@anarchist.wooz.org> Message-ID: <20130213203005.7562643e@pitrou.net> On Wed, 13 Feb 2013 13:19:14 -0500 Barry Warsaw wrote: > On Feb 13, 2013, at 08:19 AM, Tim Delaney wrote: > > >I'm of the opinion that the most fundamental property of an enum is that it > >can generate its values automatically. Everything else is a nice-to-have, > >but without that property it's not much use as an enum. > > See, I don't care about that much because I define an enum once, but I *use* > it hundreds of times. Thus a little more verbosity/explicitness at definition > time is a one-time cost, and a very low one at that. The cost is not low when you have many values. Also, with many values and one value per line, it can make your declaration very long vertically. And it's not always true that you use an enum much more often than you define it. For example, you may define many error codes (e.g. errnos) for compatibility with another system, but only check a few of them explicitly in your application code. Regards Antoine. From solipsis at pitrou.net Wed Feb 13 20:27:31 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 13 Feb 2013 20:27:31 +0100 Subject: [Python-ideas] PEP for enum library type? References: <20130212163102.50dbf570@pitrou.net> <20130212145036.32a98928@anarchist.wooz.org> <511ACB63.7030901@pearwood.info> <20130213130517.2ac6afc1@anarchist.wooz.org> Message-ID: <20130213202731.6e52c09f@pitrou.net> On Wed, 13 Feb 2013 13:05:17 -0500 Barry Warsaw wrote: > On Feb 13, 2013, at 12:59 AM, Georg Brandl wrote: > > >>> >>> from flufl.enum import make > >>> >>> make('Colors', 'RED GREEN BLUE'.split()) > >>> > > >And like namedtuple, it's ugly as hell :) > > Suggestions for a better repr are welcome. ;) It's not about the repr, it's about the make(...) call. Regards Antoine. From jonathan at slenders.be Wed Feb 13 21:48:13 2013 From: jonathan at slenders.be (Jonathan Slenders) Date: Wed, 13 Feb 2013 21:48:13 +0100 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: <20130213130033.61cb8aa6@anarchist.wooz.org> References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <20130212144518.3be491c5@anarchist.wooz.org> <20130213130033.61cb8aa6@anarchist.wooz.org> Message-ID: How is this a bug? It can be implemented in Python 3, I think. Only Enum.__metaclass__ would not be able to know the order of it's elements. (But unless you want to do bitwise operations, it's irrelevant. Is alphabetical order okay?) 2013/2/13 Barry Warsaw > > On Feb 13, 2013, at 12:57 AM, Georg Brandl wrote: > > >class Color(Enum): > > RED = 1 > > GREEN = ... # becomes 2 > > BLUE = ... # 3 > > MAGENTA = 5 > > FLUFL = ... # 6 > > That's pretty cute. > > >class Color(FlagEnum): > > RED = 1 > > GREEN = ... # becomes 2 > > BLUE = ... # 4 > > MAGENTA = 16 > > FLUFL = ... > > > >class Color(StringEnum): > > RED = ... # becomes 'red' > > GREEN = ... # etc. > > BLUE = ... > > I want to think about the latter two (especially the string one), but I've > filed this bug to track the feature. > > https://bugs.launchpad.net/flufl.enum/+bug/1124357 > > Cheers, > -Barry From random832 at fastmail.us Wed Feb 13 22:02:50 2013 From: random832 at fastmail.us (random832 at fastmail.us) Date: Wed, 13 Feb 2013 16:02:50 -0500 Subject: [Python-ideas] frozenset literals In-Reply-To: <510D20B6.5040705@pearwood.info> References: <510D20B6.5040705@pearwood.info> Message-ID: <1360789370.24461.140661191087829.377CE510@webmail.messagingengine.com> On Sat, Feb 2, 2013, at 9:20, Steven D'Aprano wrote: > Unfortunately the proposal to use f{ ... } for frozen sets cannot work > within the constraints of Python's lexer: > > http://mail.python.org/pipermail/python-3000/2008-January/011838.html > > Unfortunately we're running out of useful, easy to enter symbols for > literals. Until such time (Python4000 perhaps, or more likely Python5000) > as we can use a rich set of Unicode literals, I don't think there is any > clear way to have a frozenset literal. I was going to post about not being sure what the objection is (if it's multiple tokens, let it be multiple tokens - the contents are multiple tokens anyway - and saying it would block a future syntax extension doesn't seem like a reasonable objection to a proposed syntax extension), but I had a new idea so I'll post that instead: { as frozenset { ... } } The sequence "{ as" can't occur (to my knowledge) anywhere now. So, the thing after it is a keyword in that context (and only that context, otherwise "frozenset" remains an identifier naming an ordinary builtin) and specifies what kind of literal the following sequence is. You could also extend it to alternate forms for some other builtin types - for example { as bytes [1, 2, 3, 4, 5] } instead of b"\x1\x2\x3\x4\x5". Or... { as set { } } From joshua.landau.ws at gmail.com Wed Feb 13 22:30:36 2013 From: joshua.landau.ws at gmail.com (Joshua Landau) Date: Wed, 13 Feb 2013 21:30:36 +0000 Subject: [Python-ideas] frozenset literals In-Reply-To: <1360789370.24461.140661191087829.377CE510@webmail.messagingengine.com> References: <510D20B6.5040705@pearwood.info> <1360789370.24461.140661191087829.377CE510@webmail.messagingengine.com> Message-ID: On 13 February 2013 21:02, wrote: > On Sat, Feb 2, 2013, at 9:20, Steven D'Aprano wrote: > > Unfortunately the proposal to use f{ ... } for frozen sets cannot work > > within the constraints of Python's lexer: > > > > http://mail.python.org/pipermail/python-3000/2008-January/011838.html > > > > Unfortunately we're running out of useful, easy to enter symbols for > > literals. Until such time (Python4000 perhaps, or more likely Python5000) > > as we can use a rich set of Unicode literals, I don't think there is any > > clear way to have a frozenset literal. > > I was going to post about not being sure what the objection is (if it's > multiple tokens, let it be multiple tokens - the contents are multiple > tokens anyway - and saying it would block a future syntax extension > doesn't seem like a reasonable objection to a proposed syntax > extension), but I had a new idea so I'll post that instead: > > { as frozenset { ... } } > > The sequence "{ as" can't occur (to my knowledge) anywhere now. So, the > thing after it is a keyword in that context (and only that context, > otherwise "frozenset" remains an identifier naming an ordinary builtin) > and specifies what kind of literal the following sequence is. You could > also extend it to alternate forms for some other builtin types - for > example { as bytes [1, 2, 3, 4, 5] } instead of b"\x1\x2\x3\x4\x5". > Or... { as set { } } I'm really not sure I like this idea, but surely: LITERAL as KEYWORD { ... } as frozenset [1, 2, 3, 4, 5] as bytes {} as set would work better. However, I'm not happy on the idea that an identifier can be a keyword in another context. -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Wed Feb 13 22:57:53 2013 From: barry at python.org (Barry Warsaw) Date: Wed, 13 Feb 2013 16:57:53 -0500 Subject: [Python-ideas] PEP for enum library type? References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <20130212144518.3be491c5@anarchist.wooz.org> <20130213130033.61cb8aa6@anarchist.wooz.org> Message-ID: <20130213165753.3c5b1a6c@anarchist.wooz.org> On Feb 13, 2013, at 09:48 PM, Jonathan Slenders wrote: >How is this a bug? It's just an item in flufl.enum's bug tracker. It's marked "wish list" so it's more correct to call it a feature request. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at python.org Wed Feb 13 23:00:39 2013 From: barry at python.org (Barry Warsaw) Date: Wed, 13 Feb 2013 17:00:39 -0500 Subject: [Python-ideas] PEP for enum library type? References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <511A8A82.4010602@stoneleaf.us> <20130212220412.4bd3e604@pitrou.net> <20130213131403.3edece68@anarchist.wooz.org> Message-ID: <20130213170039.1507a75b@anarchist.wooz.org> On Feb 14, 2013, at 06:22 AM, Tim Delaney wrote: >Yes - my enum's exactly the same - simple, concise str (for both the enum >and values) and detailed repr. The actual repr is a little different to >flufl.enum repr (I think the str is the same) but conveys the same >information. I like that you came to the same conclusion, namely that the str should be simple and the repr more detailed. The exactly color of that detail can be bikeshedded to death now. :) -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at python.org Wed Feb 13 23:25:49 2013 From: barry at python.org (Barry Warsaw) Date: Wed, 13 Feb 2013 17:25:49 -0500 Subject: [Python-ideas] PEP for enum library type? References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <511A8A82.4010602@stoneleaf.us> <20130213131914.5a303f8b@anarchist.wooz.org> Message-ID: <20130213172549.328dfc00@anarchist.wooz.org> On Feb 14, 2013, at 06:28 AM, Tim Delaney wrote: >1. Values don't get repeated. This is easy, and flufl.enum guarantees this, even for subclasses: >>> class Colors(Enum): ... red = 1 ... blue = 1 ... Traceback (most recent call last): ... TypeError: Multiple enum values: 1 >>> class Colors(Enum): ... red = 1 ... blue = 2 ... >>> class MoreColors(Colors): ... green = 2 ... Traceback (most recent call last): ... TypeError: Multiple enum values: 2 >2. Values don't get skipped (unless explicitly skipped); Is #2 is the reason why you have different subclasses for flag enums, since the definition of "skipped" is different? Many folks have said they don't care about the actual enum values, so for them, skips don't matter. Setting aside explicit skips, you can add such verification with a class decorator, e.g.: -----snip snip----- from flufl.enum import Enum def noskips(cls): for i, intval in enumerate(cls, start=1): if i != int(intval): raise TypeError('Skipped value: {}'.format(i)) return cls @noskips class GoodColors(Enum): green = 3 blue = 1 red = 2 @noskips class BadColors(Enum): green = 4 blue = 1 red = 2 $ python3 noskips.py Traceback (most recent call last): ... TypeError: Skipped value: 3 -----snip snip----- >If you can convert from the string name to the enum, it makes the enum >suitable as a transport mechanism. >>> from flufl.enum import make >>> Colors = make('Colors', 'red green blue'.split()) >>> Colors['red'] >>> Colors[2] >>> Colors[Colors.green.name] Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From barry at python.org Wed Feb 13 23:28:09 2013 From: barry at python.org (Barry Warsaw) Date: Wed, 13 Feb 2013 17:28:09 -0500 Subject: [Python-ideas] PEP for enum library type? References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <511A8A82.4010602@stoneleaf.us> <20130213131914.5a303f8b@anarchist.wooz.org> <20130213203005.7562643e@pitrou.net> Message-ID: <20130213172809.69932bcb@anarchist.wooz.org> On Feb 13, 2013, at 08:30 PM, Antoine Pitrou wrote: >The cost is not low when you have many values. Also, with many values >and one value per line, it can make your declaration very long >vertically. > >And it's not always true that you use an enum much more often than you >define it. For example, you may define many error codes (e.g. errnos) >for compatibility with another system, but only check a few of them >explicitly in your application code. Huge enums haven't been common in my experience, but in that case I'd probably just use the make() helper. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From ncoghlan at gmail.com Thu Feb 14 00:11:05 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 14 Feb 2013 09:11:05 +1000 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: <20130213173807.1465d42b@pitrou.net> References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <20130212144518.3be491c5@anarchist.wooz.org> <511AC786.80807@pearwood.info> <20130213173807.1465d42b@pitrou.net> Message-ID: On 14 Feb 2013 02:39, "Antoine Pitrou" wrote: > > Le Wed, 13 Feb 2013 14:07:48 +1000, > Nick Coghlan a > ?crit : > > On 13 Feb 2013 09:11, "Tim Delaney" > > wrote: > > > > > > On 13 February 2013 09:56, Guido van Rossum > > > wrote: > > >> > > >> Frankly, enums are not that useful in small programs. For large > > >> programs or libraries, and especially for public APIs, the extra > > >> cost of defining the enum shouldn't count against them. > > >> > > >> Let's just import Barry's enums into the stdlib. > > > > > > > > > That's entirely your call. FWIW I probably won't use them, as they > > > fail > > to meet my needs for an enum, #1 being not having to specify their > > values in any way if I don't want to. Once you get past 3 or 4 > > values, it's too easy to miss or reuse a value. > > > > What's wrong with enum.make? That just accepts the sequence of names > > namedtuple style, no values specified anywhere. > > What's wrong is that TSBOOWTDI. With enum and enum.make, you have two > different idioms for declaring enums, while a single class-based > definition style should suffice. Really? What happened to the principle of layered API complexity, where a simple *convenience* function is the obvious way to do it for the cases it covers, while the full, more powerful, but also more verbose, explicit subclassing API covers the more complex cases? Magic namespaces and metaclasses sure don't meet *my* definition of obvious. A normal subclassing API with a convenience function for simple cases? That's just good layered API design. Cheers, Nick. > > Regards > > Antoine. > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Thu Feb 14 00:48:23 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 14 Feb 2013 10:48:23 +1100 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: <20130213173807.1465d42b@pitrou.net> References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <20130212144518.3be491c5@anarchist.wooz.org> <511AC786.80807@pearwood.info> <20130213173807.1465d42b@pitrou.net> Message-ID: <511C2647.3030602@pearwood.info> On 14/02/13 03:38, Antoine Pitrou wrote: > Le Wed, 13 Feb 2013 14:07:48 +1000, > Nick Coghlan a ?crit : [...] >> What's wrong with enum.make? That just accepts the sequence of names >> namedtuple style, no values specified anywhere. > > What's wrong is that TSBOOWTDI. With enum and enum.make, you have two > different idioms for declaring enums, while a single class-based > definition style should suffice. I disagree. enum.make is the more obvious solution, less verbose and IMO nicer looking too. It's explicit that it makes enums, the API is familiar to anyone who has used namedtuple, and it's an expression rather than a statement so it's more flexible. You can't do this with the class-based syntax: [enum.make(name % i, factory()) for (i, factory) in enumerate(factories)] Besides, the presence of a second, non-obvious solution is not a violation of One Obvious Way. I find it amusing that we as a community put so much emphasis on the Zen which ironically includes one of Tim Peter's subtle jokes. http://bugs.python.org/issue3364 Python is not Perl, but really, there's hardly anything in Python that can't be done two ways if you really try. -- Steven From p.f.moore at gmail.com Thu Feb 14 00:54:51 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 13 Feb 2013 23:54:51 +0000 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: <511C2647.3030602@pearwood.info> References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <20130212144518.3be491c5@anarchist.wooz.org> <511AC786.80807@pearwood.info> <20130213173807.1465d42b@pitrou.net> <511C2647.3030602@pearwood.info> Message-ID: On 13 February 2013 23:48, Steven D'Aprano wrote: > I find it amusing that we as a community put so much emphasis on the Zen > which ironically includes one of Tim Peter's subtle jokes. > > http://bugs.python.org/issue3364 Thanks for that link - I too had missed that joke. Missing-Tim-even-more-now-ly y'rs Paul. From zuo at chopin.edu.pl Thu Feb 14 01:46:05 2013 From: zuo at chopin.edu.pl (Jan Kaliszewski) Date: Thu, 14 Feb 2013 01:46:05 +0100 Subject: [Python-ideas] =?utf-8?q?PEP_for_enum_library_type=3F?= In-Reply-To: <20130213083843.7dbc7598@pitrou.net> References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <20130212144518.3be491c5@anarchist.wooz.org> <20130213082129.5d917b00@pitrou.net> <20130213083843.7dbc7598@pitrou.net> Message-ID: <0579a31de42f8eeb4940d2d3ed2425a0@chopin.edu.pl> 13.02.2013 08:38, Antoine Pitrou wrote: > It would still be shorter to type something like: > > class Color(Enum): > values = ('RED', 'GREEN', 'BLUE', 'MAGENTA', 'FLUFL') Or even in a namedtuple-like manner: class Color(Enum): values = 'RED, GREEN, BLUE, MAGENTA, FLUFL' Cheers. *j From zuo at chopin.edu.pl Thu Feb 14 01:57:50 2013 From: zuo at chopin.edu.pl (Jan Kaliszewski) Date: Thu, 14 Feb 2013 01:57:50 +0100 Subject: [Python-ideas] =?utf-8?q?PEP_for_enum_library_type=3F?= In-Reply-To: References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <20130212144518.3be491c5@anarchist.wooz.org> Message-ID: <7958702a0d603cecaeccb8146df38d61@chopin.edu.pl> 13.02.2013 00:57, Georg Brandl wrote: > We could even allow > > class Color(Enum): > RED = 1 > GREEN = ... # becomes 2 > BLUE = ... # 3 > MAGENTA = 5 > FLUFL = ... # 6 > > class Color(FlagEnum): > RED = 1 > GREEN = ... # becomes 2 > BLUE = ... # 4 > MAGENTA = 16 > FLUFL = ... > > class Color(StringEnum): > RED = ... # becomes 'red' > GREEN = ... # etc. > BLUE = ... It's nice. But what about synonymous items? class Color(Enum): RED = R = 1 GREEN = ... BLUE = B = ... # we ment BLUE = B = 3, but effectively # we'll have BLUE = 3 and B = 4 :-| What about: (all comments are for explanation only) class Color(Enum): RED = 1 # 1 GREEN = +one # 2 BLUE = +one # 3 YELLOW = Y = +one # 4 BLACK = B = 10 # 10 ORANGE = -one # 9 VIOLET = -one # 8 class Flag(Enum): FOO = 1 # 1 BAR = +rot # 2 BAZ = +rot # 4 BOO = +rot # 8 SPAM = 256 # 256 HAM = -rot # 128 RAM = -rot # 64 class Color(Enum): RED = ... # 'RED' GREEN = ... # 'GREEN' BLUE = ... # 'BLUE' and maybe also: class Color(Enum): # 0 1 2 3 RED, GREEN, BLUE, YELLOW, *end = seq() class Color(Enum): # 3 4 5 6 RED, GREEN, BLUE, YELLOW, *end = seq(3) class Flag(Enum): # 1 2 4 8 16 FOO, BAR, BAZ, BOO, SPAM, *end = flags() (yes, it *is* possible to implement it without playing with stack frames...) Cheers. *j PS. And now for something completely different: :-) @enum def Color(v): v.RED # 0 v.GREEN # 1 v.BLUE # 2 v.YELLOW # 3 @enum def Color(v): v.RED = 7 v.GREEN # 8 v.BLUE # 9 v.YELLOW # 10 From ryan at ryanhiebert.com Thu Feb 14 02:58:57 2013 From: ryan at ryanhiebert.com (Ryan Hiebert) Date: Wed, 13 Feb 2013 17:58:57 -0800 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: <7958702a0d603cecaeccb8146df38d61@chopin.edu.pl> References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <20130212144518.3be491c5@anarchist.wooz.org> <7958702a0d603cecaeccb8146df38d61@chopin.edu.pl> Message-ID: <3014408E-0204-4CAE-AAB7-F3CD25A734DE@ryanhiebert.com> On Feb 13, 2013, at 4:57 PM, Jan Kaliszewski wrote: > and maybe also: > > class Color(Enum): > # 0 1 2 3 > RED, GREEN, BLUE, YELLOW, *end = seq() > > class Color(Enum): > # 3 4 5 6 > RED, GREEN, BLUE, YELLOW, *end = seq(3) > > class Flag(Enum): > # 1 2 4 8 16 > FOO, BAR, BAZ, BOO, SPAM, *end = flags() > > (yes, it *is* possible to implement it without playing with stack frames...) This suggestion interests me most. Would it require language changes for that *end stuff? From ethan at stoneleaf.us Thu Feb 14 05:50:36 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 13 Feb 2013 20:50:36 -0800 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: <3014408E-0204-4CAE-AAB7-F3CD25A734DE@ryanhiebert.com> References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <20130212144518.3be491c5@anarchist.wooz.org> <7958702a0d603cecaeccb8146df38d61@chopin.edu.pl> <3014408E-0204-4CAE-AAB7-F3CD25A734DE@ryanhiebert.com> Message-ID: <511C6D1C.2090206@stoneleaf.us> On 02/13/2013 05:58 PM, Ryan Hiebert wrote: > On Feb 13, 2013, at 4:57 PM, Jan Kaliszewski wrote: > >> and maybe also: >> >> class Color(Enum): >> # 0 1 2 3 >> RED, GREEN, BLUE, YELLOW, *end = seq() >> >> class Color(Enum): >> # 3 4 5 6 >> RED, GREEN, BLUE, YELLOW, *end = seq(3) >> >> class Flag(Enum): >> # 1 2 4 8 16 >> FOO, BAR, BAZ, BOO, SPAM, *end = flags() >> >> (yes, it *is* possible to implement it without playing with stack frames...) > > This suggestion interests me most. Would it require language changes for that *end stuff? Already is, since 3.0 I believe. http://www.python.org/dev/peps/pep-3132/ -- ~Ethan~ From ryan at ryanhiebert.com Thu Feb 14 06:34:53 2013 From: ryan at ryanhiebert.com (Ryan Hiebert) Date: Wed, 13 Feb 2013 21:34:53 -0800 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: <511C6D1C.2090206@stoneleaf.us> References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <20130212144518.3be491c5@anarchist.wooz.org> <7958702a0d603cecaeccb8146df38d61@chopin.edu.pl> <3014408E-0204-4CAE-AAB7-F3CD25A734DE@ryanhiebert.com> <511C6D1C.2090206@stoneleaf.us> Message-ID: <4943D1D0-932F-4C14-81BE-7824B178386F@ryanhiebert.com> On Feb 13, 2013, at 8:50 PM, Ethan Furman wrote: > On 02/13/2013 05:58 PM, Ryan Hiebert wrote: > >> On Feb 13, 2013, at 4:57 PM, Jan Kaliszewski wrote: >> >>> and maybe also: >>> >>> class Color(Enum): >>> # 0 1 2 3 >>> RED, GREEN, BLUE, YELLOW, *end = seq() >>> >>> class Color(Enum): >>> # 3 4 5 6 >>> RED, GREEN, BLUE, YELLOW, *end = seq(3) >>> >>> class Flag(Enum): >>> # 1 2 4 8 16 >>> FOO, BAR, BAZ, BOO, SPAM, *end = flags() >>> >>> (yes, it *is* possible to implement it without playing with stack frames...) >> >> This suggestion interests me most. Would it require language changes for that *end stuff? > > Already is, since 3.0 I believe. > > http://www.python.org/dev/peps/pep-3132/ Wow, I've been missing out on a great feature. How is it able to avoid the stack frames? If I knew that, I'd love to write up an implementation. From ncoghlan at gmail.com Thu Feb 14 07:18:12 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 14 Feb 2013 16:18:12 +1000 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: <511C2647.3030602@pearwood.info> References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <20130212144518.3be491c5@anarchist.wooz.org> <511AC786.80807@pearwood.info> <20130213173807.1465d42b@pitrou.net> <511C2647.3030602@pearwood.info> Message-ID: On Thu, Feb 14, 2013 at 9:48 AM, Steven D'Aprano wrote: > Besides, the presence of a second, non-obvious solution is not a violation > of One Obvious Way. Something that is often forgotten is that having two ways to do something is often *good* for your API design, because it lets you design one simple API that covers a large fraction of use cases, and then a more complex underlying API that covers all (or almost all) of the rest. A procedural wrapper around an object-oriented core is the classic means of achieving this, and the standard library does it all over the place (sometimes we don't publish the OO core, but the option is there if our users demand the extra flexibility). The trick to doing it well is to make sure that users are aware that when the simple API is sufficient, that's the one they should use. Only when that API is inadequate should they reach for the more complex one. It's very, very easy to fall into the trap of documenting the comprehensive core first, and then saying "oh, by the way, here's this convenient helper function that means you will probably never need to worry about all that underlying complexity". The subprocess docs used to fall into that trap: call, check_call and check_output cover many use cases, with Popen as the complex fallback, but the old docs hit readers in the face with Popen, and only mentioned the helper functions as an afterthought. The new docs have the helper functions front and centre, with Popen relegated to "if you really need it" status. The flufl.enum docs *do* currently fall into the trap of treating enum.make as an afterthought rather than as the preferred creation API for the cases that it can handle, though. Armin Ronacher has a lovely elaboration of this principle here: http://lucumr.pocoo.org/2013/2/13/moar-classes/ (although he's mostly complaining about the other direction, simple procedural APIs which *don't* expose a flexible OO core, rather than the one here where an OO core is exposed ). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Thu Feb 14 08:35:58 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 14 Feb 2013 08:35:58 +0100 Subject: [Python-ideas] PEP for enum library type? References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <20130212144518.3be491c5@anarchist.wooz.org> <511AC786.80807@pearwood.info> <20130213173807.1465d42b@pitrou.net> Message-ID: <20130214083558.77dc3424@pitrou.net> On Thu, 14 Feb 2013 09:11:05 +1000 Nick Coghlan wrote: > On 14 Feb 2013 02:39, "Antoine Pitrou" wrote: > > > > > > What's wrong with enum.make? That just accepts the sequence of names > > > namedtuple style, no values specified anywhere. > > > > What's wrong is that TSBOOWTDI. With enum and enum.make, you have two > > different idioms for declaring enums, while a single class-based > > definition style should suffice. > > Really? What happened to the principle of layered API complexity, where a > simple *convenience* function is the obvious way to do it for the cases it > covers, while the full, more powerful, but also more verbose, explicit > subclassing API covers the more complex cases? Yes, what happened to it? Do you think enums are an important enough feature to warrant a "layered API"? Why stop at 2 layers? And why do you think subclassing is actually a complex API? It's easily understood by any OO programmer out there, even non-Python experts. Why do you think Python ORMs choose a subclassing API instead of your so-called "obvious way to do it using a convenience function"? Hint: nobody would find the convenience function API as obvious as the subclassing API. > Magic namespaces and metaclasses sure don't meet *my* definition of > obvious. Perhaps, but Python is not a supermarket where you choose your own brand of obvious as you like. It's a programming language striving to provide a rather consistent experience. People with bizarre tastes can always publish their packages on PyPI. Regards Antoine. From solipsis at pitrou.net Thu Feb 14 10:19:52 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 14 Feb 2013 10:19:52 +0100 Subject: [Python-ideas] PEP for enum library type? References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <511A8A82.4010602@stoneleaf.us> <20130213131914.5a303f8b@anarchist.wooz.org> <20130213203005.7562643e@pitrou.net> <20130213172809.69932bcb@anarchist.wooz.org> Message-ID: <20130214101952.5534b471@pitrou.net> Le Wed, 13 Feb 2013 17:28:09 -0500, Barry Warsaw a ?crit : > On Feb 13, 2013, at 08:30 PM, Antoine Pitrou wrote: > > >The cost is not low when you have many values. Also, with many values > >and one value per line, it can make your declaration very long > >vertically. > > > >And it's not always true that you use an enum much more often than > >you define it. For example, you may define many error codes (e.g. > >errnos) for compatibility with another system, but only check a few > >of them explicitly in your application code. > > Huge enums haven't been common in my experience, but in that case I'd > probably just use the make() helper. One of my common use cases for wanting an enum has always been to map a third-party protocol or library's error codes. Really, the one screaming use case in the stdlib is in the errno module :-) For that you have to be able to define an enum that subclasses int, though. Regards Antoine. From ncoghlan at gmail.com Thu Feb 14 11:40:37 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 14 Feb 2013 20:40:37 +1000 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: <20130214083558.77dc3424@pitrou.net> References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <20130212144518.3be491c5@anarchist.wooz.org> <511AC786.80807@pearwood.info> <20130213173807.1465d42b@pitrou.net> <20130214083558.77dc3424@pitrou.net> Message-ID: On Thu, Feb 14, 2013 at 5:35 PM, Antoine Pitrou wrote: > Hint: nobody would find the convenience function API as obvious as the > subclassing API. # Status quo one, two, three, four = range(4) # flufl.enum.make -> types.new_enum # Factory function for classes # For when you don't care about the specific values # Close parallel to collections.namedtuple OneToFour = types.new_enum("OneToFour", "one two three four") # flufl.enum.Enum -> types.Enum # For when you do care about (some) specific values class OneToFourAndTen(OneToFour): ten = 10 collections.namedtuple is the normative example here, and while the repetition of the class name is still annoying, that's a minor irritation compared to completely violating everyone's expectations of normal class behaviour. >> Magic namespaces and metaclasses sure don't meet *my* definition of >> obvious. > > Perhaps, but Python is not a supermarket where you choose your own brand > of obvious as you like. It's a programming language striving to > provide a rather consistent experience. People with bizarre tastes can > always publish their packages on PyPI. In which case, they should stop discussing them on python-ideas. flufl.enum is a perfectly fine example of normal Pythonic class design, with similar examples already in the standard library. The approaches being thrown around on this list lately are very cool from a technical point of view, and impressive illustrations of what Python's metaclass machinery can do for you, but they're also completely unintuitive black magic (including "x = ..." having any result other than "x is Ellipsis"). Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Thu Feb 14 12:16:02 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 14 Feb 2013 12:16:02 +0100 Subject: [Python-ideas] PEP for enum library type? References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <20130212144518.3be491c5@anarchist.wooz.org> <511AC786.80807@pearwood.info> <20130213173807.1465d42b@pitrou.net> <20130214083558.77dc3424@pitrou.net> Message-ID: <20130214121602.3b89e23e@pitrou.net> Le Thu, 14 Feb 2013 20:40:37 +1000, Nick Coghlan a ?crit : > > collections.namedtuple is the normative example here, and while the > repetition of the class name is still annoying, that's a minor > irritation compared to completely violating everyone's expectations of > normal class behaviour. I find it slightly amusing that you're complaining about a violation one of your latest PEPs is trying to make easier to make. But, still, you are not convincing me that namedtuple is normative in any way. The fact that many people would prefer a class-based declarative syntax for namedtuple is proof. By the way, not only is the repetition of the class name annoying, but it also makes subclassing more annoying too. Most of my uses of namedtuple imply subclassing, because I add some behaviour (for example additional constructors or serializers). So, compare: _BaseProtocolItem = namedtuple('_BaseProtocolItem', ('serial_no', 'command')) class ProtocolItem(_BaseProtocolItem): # snip custom methods To the hypothetical: class ProtocolItem(NamedTuple): __fields__ = ('serial_no', 'command') # snip custom methods And you understand why the namedtuple() factory function is really an inferior solution with too much typing. > In which case, they should stop discussing them on python-ideas. > flufl.enum is a perfectly fine example of normal Pythonic class > design, with similar examples already in the standard library. The > approaches being thrown around on this list lately are very cool from > a technical point of view, and impressive illustrations of what > Python's metaclass machinery can do for you, but they're also > completely unintuitive black magic (including "x = ..." having any > result other than "x is Ellipsis"). You're still ignoring the less-magic solutions such as: class Errno(Enum): __enumvalues__ = ('EBADF', 'ENOENT',) Regards Antoine. From shibturn at gmail.com Thu Feb 14 12:34:23 2013 From: shibturn at gmail.com (Richard Oudkerk) Date: Thu, 14 Feb 2013 11:34:23 +0000 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: <7958702a0d603cecaeccb8146df38d61@chopin.edu.pl> References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <20130212144518.3be491c5@anarchist.wooz.org> <7958702a0d603cecaeccb8146df38d61@chopin.edu.pl> Message-ID: On 14/02/2013 12:57am, Jan Kaliszewski wrote: > and maybe also: > > class Color(Enum): > # 0 1 2 3 > RED, GREEN, BLUE, YELLOW, *end = seq() > > class Color(Enum): > # 3 4 5 6 > RED, GREEN, BLUE, YELLOW, *end = seq(3) > > class Flag(Enum): > # 1 2 4 8 16 > FOO, BAR, BAZ, BOO, SPAM, *end = flags() > > (yes, it *is* possible to implement it without playing with stack > frames...) Are seq and flags infinite generators/iterators? If so then this won't work because end will be a *list* containing the contents of the tail of the iterator. >>> def gen(): ... i = 0 ... while True: ... yield i ... i += 1 ... >>> a, b, *end = gen() -- Richard From ncoghlan at gmail.com Thu Feb 14 14:02:44 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 14 Feb 2013 23:02:44 +1000 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: <20130214121602.3b89e23e@pitrou.net> References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <20130212144518.3be491c5@anarchist.wooz.org> <511AC786.80807@pearwood.info> <20130213173807.1465d42b@pitrou.net> <20130214083558.77dc3424@pitrou.net> <20130214121602.3b89e23e@pitrou.net> Message-ID: On Thu, Feb 14, 2013 at 9:16 PM, Antoine Pitrou wrote: > You're still ignoring the less-magic solutions such as: > > class Errno(Enum): > __enumvalues__ = ('EBADF', 'ENOENT',) Yes, an approach like that, or anything similarly explicit would be fine with me (and has the virtue of potentially supporting pickle, unlike anonymous class generators like the current namedtuple implementation). It was the clever insanity of some of the other ideas being kicked around in these threads that I wanted to be clear I would fight tooth and nail if they were ever actually proposed for stdlib inclusion. My enthusiasm for flufl.enum in particular is mainly of the "I'm tired of seeing almost the exact same enum discussion get rehashed every couple of years, and that one looks good enough to me" variety, so it probably isn't wise to pay too much attention to me when it comes to the specific of the API :P Your comments did give me an idea about namedtuple though - I'll start another thread about that. > I find it slightly amusing that you're complaining about a violation > one of your latest PEPs is trying to make easier to make. [Replying to this second, since it is getting into philosophy of language design, and I actually agree with your key point above] I have a long history of trying to demystify various aspects of Python metaprogramming and make it easier for people to explore the language, and create suitable meta-frameworks for their own use, and for use within particular subcommunities. As I see it, violating normal expectations for class behaviour *within the context of a framework* is fine, *so long as you can't achieve reasonable syntax in a simpler way*. Environments like Django, SQL Alchemy, etc set up new norms about expected behaviour in different kinds of classes, and that's the entire reason I think these metaprograming tools are worth providing in the first place (although I acknowledge it *can* cause problems when people learn one of these frameworks *first*, and then later need to be taught the difference between the automatically provided framework level behaviour for Forms, Models, Views, etc and the underlying standard language behaviour). While the standard library already provides three distinct class level behaviours (normal classes, subclasses of type and ABCs), those are each intertwined deeply with the core type system (arguably, inheriting from builtins other than type or object counts as an additional case, since inheriting from a C defined class that isn't really designed for it has interesting consequences in CPython due to implementation limitations). Furthermore, while our variants change the way instantiation, descriptors, isinstance and issubclass behave after the class is created, they *don't* alter the runtime semantics of the class body itself. To my mind, defining enumerations doesn't even come close to reaching the bar that would justify highly customised class behaviour that a reader can't guess simply from knowing Python and reading the code (I get the impression you already agree with me on this point, which is why I moved this philosophical section after my agreement with you above). Cheers, Nick -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Thu Feb 14 14:19:51 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 14 Feb 2013 23:19:51 +1000 Subject: [Python-ideas] A subclassing API for named tuples? Message-ID: An exchange with Antoine in one of the enum threads sparked a thought. A recurring suggestion for collections.namedtuple is that it would be nice to be able to define them like this (as it not only avoids having to repeat the class name, but also allows them to play nicely with pickle and other name-based reference mechanisms): class MyTuple(collections.NamedTuple): __fields__ = "a b c d e".split() However, one of Raymond's long standing objections to such a design for namedtuple is the ugliness of people having to remember to include the right __slots__ definition to ensure it doesn't add any storage overhead above and beyond that for the underlying tuple. For the intended use case as a replacement for short tuples, an unused dict per instance is a *big* wasted overhead, so that concern can't be dismissed as premature optimisation: >>> import sys >>> class Slots(tuple): __slots__ = () ... >>> class InstanceDict(tuple): pass ... >>> sys.getsizeof(tuple([1, 2, 3])) 72 >>> x = Slots([1, 2, 3]) >>> sys.getsizeof(x) 72 >>> y = InstanceDict([1, 2, 3]) >>> sys.getsizeof(y) # All good, right? 72 >>> sys.getsizeof(y.__dict__) # Yeah, not so much... 96 However, the thought that occurred to me is that the right metaclass definition allows the default behaviour of __slots__ to be flipped, so that you get "__slots__ = ()" defined in your class namespace automatically, and you have to write "del __slots__" to get normal class behaviour back: >>> class SlotsMeta(type): ... def __prepare__(cls, *args, **kwds): ... return dict(__slots__=()) ... >>> class SlotsByDefault(metaclass = SlotsMeta): pass ... >>> class Slots(tuple, SlotsByDefault): pass ... >>> class InstanceDict(tuple, SlotsByDefault): del __slots__ ... >>> sys.getsizeof(Slots([1, 2, 3])) 72 >>> Slots().__dict__ Traceback (most recent call last): File "", line 1, in AttributeError: 'Slots' object has no attribute '__dict__' >>> sys.getsizeof(InstanceDict([1, 2, 3])) 72 >>> sys.getsizeof(InstanceDict([1, 2, 3]).__dict__) 96 So, what do people think? Too much magic? Or just the right amount to allow a cleaner syntax for named tuple definitions, without inadvertently encouraging people to do bad things to their memory usage? (Note: for backwards compatibility reasons, we couldn't use a custom metaclass for the classes returned by the existing collections namedtuple API. However, we could certainly provide a distinct collections.NamedTuple type which used a custom metaclass to behave this way). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Thu Feb 14 15:56:55 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 14 Feb 2013 15:56:55 +0100 Subject: [Python-ideas] A subclassing API for named tuples? References: Message-ID: <20130214155655.5cc3d582@pitrou.net> Le Thu, 14 Feb 2013 23:19:51 +1000, Nick Coghlan a ?crit : > An exchange with Antoine in one of the enum threads sparked a thought. > > A recurring suggestion for collections.namedtuple is that it would be > nice to be able to define them like this (as it not only avoids having > to repeat the class name, but also allows them to play nicely with > pickle and other name-based reference mechanisms): > > class MyTuple(collections.NamedTuple): > __fields__ = "a b c d e".split() > > However, one of Raymond's long standing objections to such a design > for namedtuple is the ugliness of people having to remember to include > the right __slots__ definition to ensure it doesn't add any storage > overhead above and beyond that for the underlying tuple. You don't *have* to remember to include it. You just have to include it if you really care about the memory footprint. Which is something that is easy to mention in the documentation. (also, an empty dict that never gets accessed is not much of a problem performance-wise) Speaking for myself, most uses of namedtuple are not performance-critical. They are for convenience: I want an immutable, hashable, comparable record-like class (with a useful definition of equality, which is also very convenient for unit tests :-)) and I don't want to write that behaviour by hand every time. Regards Antoine. From ericsnowcurrently at gmail.com Thu Feb 14 16:32:49 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Thu, 14 Feb 2013 08:32:49 -0700 Subject: [Python-ideas] A subclassing API for named tuples? In-Reply-To: References: Message-ID: On Feb 14, 2013 6:20 AM, "Nick Coghlan" wrote: > A recurring suggestion for collections.namedtuple is that it would be > nice to be able to define them like this (as it not only avoids having > to repeat the class name, but also allows them to play nicely with > pickle and other name-based reference mechanisms): > > class MyTuple(collections.NamedTuple): > __fields__ = "a b c d e".split() Something similar that I've been using: @as_namedtuple("a b c d e") class MyTuple: """My namedtuple with extra stuff.""" @property def something_special(self): """...""" return ... Support for default values is also something I've added, but that's relatively orthogonal to this discussion. -eric -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Thu Feb 14 16:42:53 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 14 Feb 2013 16:42:53 +0100 Subject: [Python-ideas] A subclassing API for named tuples? References: Message-ID: <20130214164253.0c4198b9@pitrou.net> Le Thu, 14 Feb 2013 08:32:49 -0700, Eric Snow a ?crit : > > Something similar that I've been using: > > @as_namedtuple("a b c d e") > class MyTuple: > """My namedtuple with extra stuff.""" > @property > def something_special(self): > """...""" > return ... > > Support for default values is also something I've added, but that's > relatively orthogonal to this discussion. Ah, right. That's also why I often need to subclass namedtuple. Regards Antoine. From steve at pearwood.info Thu Feb 14 23:09:57 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 15 Feb 2013 09:09:57 +1100 Subject: [Python-ideas] A subclassing API for named tuples? In-Reply-To: References: Message-ID: <511D60B5.7000903@pearwood.info> On 15/02/13 00:19, Nick Coghlan wrote: > An exchange with Antoine in one of the enum threads sparked a thought. > > A recurring suggestion for collections.namedtuple is that it would be > nice to be able to define them like this (as it not only avoids having > to repeat the class name, but also allows them to play nicely with > pickle and other name-based reference mechanisms): > > class MyTuple(collections.NamedTuple): > __fields__ = "a b c d e".split() How would that differ from this? class MyTuple(collections.namedtuple("MyTupleParent", "a b c d e")): pass Apart from the DRY violation in the class name, I find that perfectly acceptable, and it seems to work fine with pickling: py> t = MyTuple(2, 4, 8, 16, 32) py> t MyTuple(a=2, b=4, c=8, d=16, e=32) py> s = pickle.dumps(t) py> u = pickle.loads(s) py> u == t True -- Steven From greg.ewing at canterbury.ac.nz Thu Feb 14 23:31:47 2013 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 15 Feb 2013 11:31:47 +1300 Subject: [Python-ideas] A subclassing API for named tuples? In-Reply-To: References: Message-ID: <511D65D3.9000707@canterbury.ac.nz> Nick Coghlan wrote: > > class MyTuple(collections.NamedTuple): > __fields__ = "a b c d e".split() > > However, one of Raymond's long standing objections to such a design > for namedtuple is the ugliness of people having to remember to include > the right __slots__ definition to ensure it doesn't add any storage > overhead above and beyond that for the underlying tuple. So why not kill two definitions with one stone and spell it like this: class MyTuple(collections.NamedTuple): __slots__ = "a b c d e".split() -- Greg From solipsis at pitrou.net Thu Feb 14 23:40:08 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 14 Feb 2013 23:40:08 +0100 Subject: [Python-ideas] A subclassing API for named tuples? References: <511D60B5.7000903@pearwood.info> Message-ID: <20130214234008.17eb7ed0@pitrou.net> On Fri, 15 Feb 2013 09:09:57 +1100 Steven D'Aprano wrote: > On 15/02/13 00:19, Nick Coghlan wrote: > > An exchange with Antoine in one of the enum threads sparked a thought. > > > > A recurring suggestion for collections.namedtuple is that it would be > > nice to be able to define them like this (as it not only avoids having > > to repeat the class name, but also allows them to play nicely with > > pickle and other name-based reference mechanisms): > > > > class MyTuple(collections.NamedTuple): > > __fields__ = "a b c d e".split() > > > How would that differ from this? > > class MyTuple(collections.namedtuple("MyTupleParent", "a b c d e")): > pass > > > Apart from the DRY violation in the class name, I find that perfectly > acceptable, and it seems to work fine with pickling: Well, it's perfectly acceptable as long as you have one-letter field names. Try with real field names and it becomes rather unwieldy. Regards Antoine. From steve at pearwood.info Fri Feb 15 00:17:32 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 15 Feb 2013 10:17:32 +1100 Subject: [Python-ideas] A subclassing API for named tuples? In-Reply-To: <20130214234008.17eb7ed0@pitrou.net> References: <511D60B5.7000903@pearwood.info> <20130214234008.17eb7ed0@pitrou.net> Message-ID: <511D708C.9030208@pearwood.info> On 15/02/13 09:40, Antoine Pitrou wrote: > On Fri, 15 Feb 2013 09:09:57 +1100 > Steven D'Aprano wrote: > >> On 15/02/13 00:19, Nick Coghlan wrote: >>> An exchange with Antoine in one of the enum threads sparked a thought. >>> >>> A recurring suggestion for collections.namedtuple is that it would be >>> nice to be able to define them like this (as it not only avoids having >>> to repeat the class name, but also allows them to play nicely with >>> pickle and other name-based reference mechanisms): >>> >>> class MyTuple(collections.NamedTuple): >>> __fields__ = "a b c d e".split() >> >> >> How would that differ from this? >> >> class MyTuple(collections.namedtuple("MyTupleParent", "a b c d e")): >> pass >> >> >> Apart from the DRY violation in the class name, I find that perfectly >> acceptable, and it seems to work fine with pickling: > > Well, it's perfectly acceptable as long as you have one-letter field > names. Try with real field names and it becomes rather unwieldy. Exactly the same thing can be said about the __field__ line. The only difference is that in one case you reach "unwieldy" a little sooner than in the other. There are well-known ways to deal with excessively long lines of code which don't require a new collection type. class MyClassWithAnExtremelyLongName( collections.namedtuple("Cheese", ("cheddar swiss ricotta camembert gouda parmesan brie limburger havarti" " danish_blue roquefort greek_feta provolone mozzarella edam maasdam" " stilton wensleydale red_leicester american colby monterey_jack kapiti" " casu_marzu") ) ): pass Style arguments about the placement of closing brackets to /dev/null :-) I'm simply not seeing enough benefit to NamedTuple to make up for the invariable confusion between NamedTuple and namedtuple. -- Steven From zuo at chopin.edu.pl Fri Feb 15 00:33:57 2013 From: zuo at chopin.edu.pl (Jan Kaliszewski) Date: Fri, 15 Feb 2013 00:33:57 +0100 Subject: [Python-ideas] =?utf-8?q?A_subclassing_API_for_named_tuples=3F?= In-Reply-To: References: Message-ID: <05fea7e04f1550e3a91071e9f3c72281@chopin.edu.pl> Please remember that in the Py3k docs is a link to my namedtuple.abc recipe which covers most of the issues discussed in this thread (+ ABC registration/isinstance/issubclass checking). Cheers. *j PS. If we wanted to have the "automatic-always-__slots__" feature (discussed in this thread), we probably need to dedent line #57 of the recipe one indentation level left... PPS. As for Py3.3, the recipe is outdateed a bit as it uses deprecated @abstractproperty -- but it can be easily fixed. From jbvsmo at gmail.com Fri Feb 15 02:54:43 2013 From: jbvsmo at gmail.com (=?ISO-8859-1?Q?Jo=E3o_Bernardo?=) Date: Thu, 14 Feb 2013 23:54:43 -0200 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <20130212144518.3be491c5@anarchist.wooz.org> <511AC786.80807@pearwood.info> <20130213173807.1465d42b@pitrou.net> <20130214083558.77dc3424@pitrou.net> <20130214121602.3b89e23e@pitrou.net> Message-ID: And now for something completely different: If the Enums are used with attribute syntax, why not this? >>> Enum.Colors.red.green.blue >>> Enum.Colors.red >>> Enum.Colors.green >>> Enum.Colors.blue This is my 5-minute implementation (quite fun) Looks like everybody is fighting against the class syntax and it is very incompatible with the notion of enum we've got from other languages. BTW, I'm a big fan of flufl.enum Jo?o Bernardo 2013/2/14 Nick Coghlan > On Thu, Feb 14, 2013 at 9:16 PM, Antoine Pitrou > wrote: > > > You're still ignoring the less-magic solutions such as: > > > > class Errno(Enum): > > __enumvalues__ = ('EBADF', 'ENOENT',) > > Yes, an approach like that, or anything similarly explicit would be > fine with me (and has the virtue of potentially supporting pickle, > unlike anonymous class generators like the current namedtuple > implementation). It was the clever insanity of some of the other ideas > being kicked around in these threads that I wanted to be clear I would > fight tooth and nail if they were ever actually proposed for stdlib > inclusion. > > My enthusiasm for flufl.enum in particular is mainly of the "I'm tired > of seeing almost the exact same enum discussion get rehashed every > couple of years, and that one looks good enough to me" variety, so it > probably isn't wise to pay too much attention to me when it comes to > the specific of the API :P > > Your comments did give me an idea about namedtuple though - I'll start > another thread about that. > > > I find it slightly amusing that you're complaining about a violation > > one of your latest PEPs is trying to make easier to make. > > [Replying to this second, since it is getting into philosophy of > language design, and I actually agree with your key point above] > > I have a long history of trying to demystify various aspects of Python > metaprogramming and make it easier for people to explore the language, > and create suitable meta-frameworks for their own use, and for use > within particular subcommunities. > > As I see it, violating normal expectations for class behaviour *within > the context of a framework* is fine, *so long as you can't achieve > reasonable syntax in a simpler way*. Environments like Django, SQL > Alchemy, etc set up new norms about expected behaviour in different > kinds of classes, and that's the entire reason I think these > metaprograming tools are worth providing in the first place (although > I acknowledge it *can* cause problems when people learn one of these > frameworks *first*, and then later need to be taught the difference > between the automatically provided framework level behaviour for > Forms, Models, Views, etc and the underlying standard language > behaviour). > > While the standard library already provides three distinct class level > behaviours (normal classes, subclasses of type and ABCs), those are > each intertwined deeply with the core type system (arguably, > inheriting from builtins other than type or object counts as an > additional case, since inheriting from a C defined class that isn't > really designed for it has interesting consequences in CPython due to > implementation limitations). Furthermore, while our variants change > the way instantiation, descriptors, isinstance and issubclass behave > after the class is created, they *don't* alter the runtime semantics > of the class body itself. > > To my mind, defining enumerations doesn't even come close to reaching > the bar that would justify highly customised class behaviour that a > reader can't guess simply from knowing Python and reading the code (I > get the impression you already agree with me on this point, which is > why I moved this philosophical section after my agreement with you > above). > > Cheers, > Nick > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Fri Feb 15 03:11:38 2013 From: larry at hastings.org (Larry Hastings) Date: Thu, 14 Feb 2013 18:11:38 -0800 Subject: [Python-ideas] A subclassing API for named tuples? In-Reply-To: References: Message-ID: <511D995A.8000008@hastings.org> On 02/14/2013 05:19 AM, Nick Coghlan wrote: > A recurring suggestion for collections.namedtuple is that it would be > nice to be able to define them like this [...] FWIW, I think namedtuple is overused. Not that there's anything innately wrong with namedtuple; it's just that I think too many types are iterable which shouldn't be. (Nobody unpacks the stat_result anymore, that's like 1995 man.) I suggest people use types.SimpleNamespace unless iterability is specifically required. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Fri Feb 15 03:57:47 2013 From: barry at python.org (Barry Warsaw) Date: Thu, 14 Feb 2013 21:57:47 -0500 Subject: [Python-ideas] PEP for enum library type? References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <20130212144518.3be491c5@anarchist.wooz.org> <511AC786.80807@pearwood.info> <20130213173807.1465d42b@pitrou.net> <511C2647.3030602@pearwood.info> Message-ID: <20130214215747.7af1185d@anarchist.wooz.org> On Feb 14, 2013, at 04:18 PM, Nick Coghlan wrote: >The flufl.enum docs *do* currently fall into the trap of treating >enum.make as an afterthought rather than as the preferred creation API >for the cases that it can handle, though. I guess that's because .make() *was* an afterthought. ;) .make() was added long after the class syntax. IIRC, Michael Foord suggested that API to allow creation of enums programmatically instead of statically, which the class syntax requires, modulo exec(). I suppose I've never really thought of .make() as a convenience because of that, and because I personally prefer the class syntax, all things being equal. It's interesting that other folks view .make()'s primary benefit as being one of convenience. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From greg.ewing at canterbury.ac.nz Fri Feb 15 06:37:47 2013 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 15 Feb 2013 18:37:47 +1300 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <20130212144518.3be491c5@anarchist.wooz.org> <511AC786.80807@pearwood.info> <20130213173807.1465d42b@pitrou.net> <20130214083558.77dc3424@pitrou.net> <20130214121602.3b89e23e@pitrou.net> Message-ID: <511DC9AB.9060906@canterbury.ac.nz> Jo?o Bernardo wrote: > And now for something completely different: > > If the Enums are used with attribute syntax, why not this? > >> >> Enum.Colors.red.green.blue > > Interesting idea, but there seem to be a couple of problems. It looks like you intend Enum to keep a record of all the enum types you create with it. What happens if two unrelated modules both define an enum called Colors? To avoid this problem, you'd want to assign the newly created enum type to a local name, but then you need to write the name twice: Colors = Enum.Colors.red.green.blue or Colors = Enum("Colors").red.green.blue Then what happens if you mistakenly write: print(Colors.pineapple) Seems like this would inadvertently add a new value to the enum instead of producing an AttributeError. -- Greg From jbvsmo at gmail.com Fri Feb 15 07:38:22 2013 From: jbvsmo at gmail.com (=?ISO-8859-1?Q?Jo=E3o_Bernardo?=) Date: Fri, 15 Feb 2013 04:38:22 -0200 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: <511DC9AB.9060906@canterbury.ac.nz> References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <20130212144518.3be491c5@anarchist.wooz.org> <511AC786.80807@pearwood.info> <20130213173807.1465d42b@pitrou.net> <20130214083558.77dc3424@pitrou.net> <20130214121602.3b89e23e@pitrou.net> <511DC9AB.9060906@canterbury.ac.nz> Message-ID: This is just an idea to avoid class definition, but it can be improved. 2013/2/15 Greg Ewing > Jo?o Bernardo wrote: > >> And now for something completely different: >> >> If the Enums are used with attribute syntax, why not this? >> >> >> Enum.Colors.red.green.blue >>> >> >> >> > > Interesting idea, but there seem to be a couple of problems. > It looks like you intend Enum to keep a record of all the > enum types you create with it. What happens if two unrelated > modules both define an enum called Colors? > > Enum could write it in the current namespace, just like the `class` statement. >>> Enum.Colors.red.green.blue >>> Colors.red > To avoid this problem, you'd want to assign the newly > created enum type to a local name, but then you need to > write the name twice: > > Colors = Enum.Colors.red.green.blue > > or > > Colors = Enum("Colors").red.green.blue > > Then what happens if you mistakenly write: > > print(Colors.pineapple) > > This could be solved with a function call to tell the enum to stop accepting new values. >>> Enum.Colors.red.green.blue() >>> Colors.pineapple AttributeError: 'Color' object has no attribute 'pineaple' > Seems like this would inadvertently add a new value to the > enum instead of producing an AttributeError. > > -- > Greg > > ______________________________**_________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/**mailman/listinfo/python-ideas > Each attribute access on `Enum` would create a new class even if it's already defined in the same namespace. The use of the name `Enum` is just good for definition anyway. Jo?o Bernardo -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Fri Feb 15 08:03:45 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 15 Feb 2013 08:03:45 +0100 Subject: [Python-ideas] A subclassing API for named tuples? References: <511D60B5.7000903@pearwood.info> <20130214234008.17eb7ed0@pitrou.net> <511D708C.9030208@pearwood.info> Message-ID: <20130215080345.61d1ca8f@pitrou.net> On Fri, 15 Feb 2013 10:17:32 +1100 Steven D'Aprano wrote: > > class MyClassWithAnExtremelyLongName( > collections.namedtuple("Cheese", > ("cheddar swiss ricotta camembert gouda parmesan brie limburger havarti" > " danish_blue roquefort greek_feta provolone mozzarella edam maasdam" > " stilton wensleydale red_leicester american colby monterey_jack kapiti" > " casu_marzu") > ) > ): > pass I don't find that readable at all. Having many indentations in a single declaration line makes things quite messy in my opinion, and your important class declaration is now drowning in a sea of literals. The equivalent __fields__ *would* be readable, if properly formatted. Regards Antoine. From solipsis at pitrou.net Fri Feb 15 08:07:00 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 15 Feb 2013 08:07:00 +0100 Subject: [Python-ideas] A subclassing API for named tuples? References: <511D995A.8000008@hastings.org> Message-ID: <20130215080700.542770d0@pitrou.net> On Thu, 14 Feb 2013 18:11:38 -0800 Larry Hastings wrote: > On 02/14/2013 05:19 AM, Nick Coghlan wrote: > > A recurring suggestion for collections.namedtuple is that it would be > > nice to be able to define them like this [...] > > FWIW, I think namedtuple is overused. Not that there's anything > innately wrong with namedtuple; it's just that I think too many types > are iterable which shouldn't be. (Nobody unpacks the stat_result > anymore, that's like 1995 man.) I suggest people use > types.SimpleNamespace unless iterability is specifically required. There are tons of cases where tuple unpacking is a very convenient API. Regards Antoine. From ncoghlan at gmail.com Fri Feb 15 09:25:53 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 15 Feb 2013 18:25:53 +1000 Subject: [Python-ideas] A subclassing API for named tuples? In-Reply-To: <20130215080700.542770d0@pitrou.net> References: <511D995A.8000008@hastings.org> <20130215080700.542770d0@pitrou.net> Message-ID: On Fri, Feb 15, 2013 at 5:07 PM, Antoine Pitrou wrote: > On Thu, 14 Feb 2013 18:11:38 -0800 > Larry Hastings wrote: >> On 02/14/2013 05:19 AM, Nick Coghlan wrote: >> > A recurring suggestion for collections.namedtuple is that it would be >> > nice to be able to define them like this [...] >> >> FWIW, I think namedtuple is overused. Not that there's anything >> innately wrong with namedtuple; it's just that I think too many types >> are iterable which shouldn't be. (Nobody unpacks the stat_result >> anymore, that's like 1995 man.) I suggest people use >> types.SimpleNamespace unless iterability is specifically required. > > There are tons of cases where tuple unpacking is a very convenient API. Indeed. In particular, consuming namespaces from an iterator is substantially more annoying than consuming a tuple. I'd put the size cutoff for switching over to a namespace somewhere around the 5 item mark. At 2-3, the tuple is often clearly superior, at 4 it's arguable, at 5, unpacking starts to get a bit hard to read. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From jsbueno at python.org.br Fri Feb 15 12:11:35 2013 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Fri, 15 Feb 2013 09:11:35 -0200 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: <511DC9AB.9060906@canterbury.ac.nz> References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <20130212144518.3be491c5@anarchist.wooz.org> <511AC786.80807@pearwood.info> <20130213173807.1465d42b@pitrou.net> <20130214083558.77dc3424@pitrou.net> <20130214121602.3b89e23e@pitrou.net> <511DC9AB.9060906@canterbury.ac.nz> Message-ID: On 15 February 2013 03:37, Greg Ewing wrote: > Jo?o Bernardo wrote: >> >> And now for something completely different: >> >> If the Enums are used with attribute syntax, why not this? >> >>> >> Enum.Colors.red.green.blue >> >> >> > > > Interesting idea, but there seem to be a couple of problems. > It looks like you intend Enum to keep a record of all the > enum types you create with it. What happens if two unrelated > modules both define an enum called Colors? > Don't be picky - it is an interesting idea, and the obvious way to avoid such side effect would be to require: Colors = Enum("Colors").RED.GREEN.BLUE implementation might be tricky this way - you don't want new values genreated each time one fetches a new attribute from a what should be a constant - then maybe: Colors = Enum("Colors").RED.GREEN.BLUE._endEnum > To avoid this problem, you'd want to assign the newly > created enum type to a local name, but then you need to > write the name twice: > > Colors = Enum.Colors.red.green.blue Writing the name of the ENum set twice might be the ideal, but it is what one can do to get away today, in Python - and people are used to it (I'd favor a mechanism for an object to know its name, but that is another issue entirely.) > > -- > Greg > PS. all - please note Jo?o Bernardo and me (Jo?o Bueno) are different people ------------------------ From steve at pearwood.info Sat Feb 16 05:51:28 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 16 Feb 2013 15:51:28 +1100 Subject: [Python-ideas] A subclassing API for named tuples? In-Reply-To: <20130215080345.61d1ca8f@pitrou.net> References: <511D60B5.7000903@pearwood.info> <20130214234008.17eb7ed0@pitrou.net> <511D708C.9030208@pearwood.info> <20130215080345.61d1ca8f@pitrou.net> Message-ID: <511F1050.2000809@pearwood.info> On 15/02/13 18:03, Antoine Pitrou wrote: > On Fri, 15 Feb 2013 10:17:32 +1100 > Steven D'Aprano wrote: >> >> class MyClassWithAnExtremelyLongName( >> collections.namedtuple("Cheese", >> ("cheddar swiss ricotta camembert gouda parmesan brie limburger havarti" >> " danish_blue roquefort greek_feta provolone mozzarella edam maasdam" >> " stilton wensleydale red_leicester american colby monterey_jack kapiti" >> " casu_marzu") >> ) >> ): >> pass > > I don't find that readable at all. Having many indentations in a > single declaration line makes things quite messy in my opinion, and > your important class declaration is now drowning in a sea of literals. You don't have to stick the literals in the class definition. This is an easy problem to solve with existing techniques: from collections import namedtuple FIELDNAMES = """...""" # Format it however you like. class MyClassWithAnExtremelyLongName(namedtuple("Cheese", FIELDNAMES)): pass There is no need to introduce confusion and uncertainty, "Should I use namedtuple or NamedTuple?".To say nothing of the invariable cases where people use the wrong one and have to deal with the cryptic error: py> class C(collections.namedtuple): ... pass ... Traceback (most recent call last): File "", line 1, in TypeError: function() argument 1 must be code, not str Moving the field name definitions outside of the namedtuple into a magic dunder attribute adds complexity. To use namedtuple, I just need to remember the calling signature, which is trivial. To use the proposed NamedTuple, I have to remember a magic dunder attribute __fields__ that is not used anywhere else. I don't see this as a win for simplicity. -- Steven From solipsis at pitrou.net Sat Feb 16 11:18:48 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 16 Feb 2013 11:18:48 +0100 Subject: [Python-ideas] A subclassing API for named tuples? References: <511D60B5.7000903@pearwood.info> <20130214234008.17eb7ed0@pitrou.net> <511D708C.9030208@pearwood.info> <20130215080345.61d1ca8f@pitrou.net> <511F1050.2000809@pearwood.info> Message-ID: <20130216111848.745dd99c@pitrou.net> On Sat, 16 Feb 2013 15:51:28 +1100 Steven D'Aprano wrote: > > from collections import namedtuple > > FIELDNAMES = """...""" # Format it however you like. > > class MyClassWithAnExtremelyLongName(namedtuple("Cheese", FIELDNAMES)): > pass Still not very elegant IMO (and it introduces spurious globals in the module). Parameters for class construction shouldn't be outside of the class declaration. Really, the class declaration syntax has been *designed* to deal with all of this. It's counter-productive to try not to use it. (but, yes, now the official namedtuple API is the one we know, I agree that having two ways to do it may be one too many) Regards Antoine. From zuo at chopin.edu.pl Sat Feb 16 14:46:39 2013 From: zuo at chopin.edu.pl (Jan Kaliszewski) Date: Sat, 16 Feb 2013 14:46:39 +0100 Subject: [Python-ideas] =?utf-8?q?A_subclassing_API_for_named_tuples=3F?= In-Reply-To: <20130216111848.745dd99c@pitrou.net> References: <511D60B5.7000903@pearwood.info> <20130214234008.17eb7ed0@pitrou.net> <511D708C.9030208@pearwood.info> <20130215080345.61d1ca8f@pitrou.net> <511F1050.2000809@pearwood.info> <20130216111848.745dd99c@pitrou.net> Message-ID: 16.02.2013 11:18, Antoine Pitrou wrote: > On Sat, 16 Feb 2013 15:51:28 +1100 > Steven D'Aprano wrote: >> >> from collections import namedtuple >> >> FIELDNAMES = """...""" # Format it however you like. >> >> class MyClassWithAnExtremelyLongName(namedtuple("Cheese", >> FIELDNAMES)): >> pass > > Still not very elegant IMO I agree. I see some nicer (IMHO) alternatives... Apart from the recipe I mentioned in the recent post (although I am *not* convinced it should be added to the stdlib) some decorator-based way may be nice, e.g.: @namedtuple(fields='length weight is_poisonous') class Snake: def hiss(self): return 'hiss' + self.length * 's' Cheers. *j From greg at krypto.org Sun Feb 17 00:41:06 2013 From: greg at krypto.org (Gregory P. Smith) Date: Sat, 16 Feb 2013 15:41:06 -0800 Subject: [Python-ideas] A subclassing API for named tuples? In-Reply-To: <511D995A.8000008@hastings.org> References: <511D995A.8000008@hastings.org> Message-ID: On Thu, Feb 14, 2013 at 6:11 PM, Larry Hastings wrote: > On 02/14/2013 05:19 AM, Nick Coghlan wrote: > > A recurring suggestion for collections.namedtuple is that it would be > nice to be able to define them like this [...] > > > FWIW, I think namedtuple is overused. Not that there's anything innately > wrong with namedtuple; it's just that I think too many types are iterable > which shouldn't be. (Nobody unpacks the stat_result anymore, that's like > 1995 man.) I suggest people use types.SimpleNamespace unless iterability > is specifically required. > Ironically the types.SimpleNamespace documentation says: "However, for a structured record type use namedtuple() instead." -gps -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sun Feb 17 22:38:38 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 18 Feb 2013 08:38:38 +1100 Subject: [Python-ideas] A subclassing API for named tuples? In-Reply-To: References: <511D60B5.7000903@pearwood.info> <20130214234008.17eb7ed0@pitrou.net> <511D708C.9030208@pearwood.info> <20130215080345.61d1ca8f@pitrou.net> <511F1050.2000809@pearwood.info> <20130216111848.745dd99c@pitrou.net> Message-ID: <51214DDE.2020506@pearwood.info> On 17/02/13 00:46, Jan Kaliszewski wrote: > 16.02.2013 11:18, Antoine Pitrou wrote: >> On Sat, 16 Feb 2013 15:51:28 +1100 >> Steven D'Aprano wrote: >>> >>> from collections import namedtuple >>> >>> FIELDNAMES = """...""" # Format it however you like. >>> >>> class MyClassWithAnExtremelyLongName(namedtuple("Cheese", FIELDNAMES)): >>> pass >> >> Still not very elegant IMO > > I agree. I see some nicer (IMHO) alternatives... Apart from the recipe I mentioned > in the recent post (although I am *not* convinced it should be added to the stdlib) > some decorator-based way may be nice, e.g.: > > @namedtuple(fields='length weight is_poisonous') > class Snake: > def hiss(self): > return 'hiss' + self.length * 's' This implies that all namedtuples must be callable. Point3D = namedtuple('Point3D', 'x y z') pt = Point3D(2, 4, 8) I don't think it's appropriate for tuples, named or not, to be callable. -- Steven From zuo at chopin.edu.pl Mon Feb 18 00:45:15 2013 From: zuo at chopin.edu.pl (Jan Kaliszewski) Date: Mon, 18 Feb 2013 00:45:15 +0100 Subject: [Python-ideas] =?utf-8?q?A_subclassing_API_for_named_tuples=3F?= In-Reply-To: <51214DDE.2020506@pearwood.info> References: <511D60B5.7000903@pearwood.info> <20130214234008.17eb7ed0@pitrou.net> <511D708C.9030208@pearwood.info> <20130215080345.61d1ca8f@pitrou.net> <511F1050.2000809@pearwood.info> <20130216111848.745dd99c@pitrou.net> <51214DDE.2020506@pearwood.info> Message-ID: <2988b553eb09138a013b8a636cb9487b@chopin.edu.pl> 17.02.2013 22:38, Steven D'Aprano wrote: > On 17/02/13 00:46, Jan Kaliszewski wrote: >> 16.02.2013 11:18, Antoine Pitrou wrote: >>> On Sat, 16 Feb 2013 15:51:28 +1100 >>> Steven D'Aprano wrote: >>>> >>>> from collections import namedtuple >>>> >>>> FIELDNAMES = """...""" # Format it however you like. >>>> >>>> class MyClassWithAnExtremelyLongName(namedtuple("Cheese", >>>> FIELDNAMES)): >>>> pass >>> >>> Still not very elegant IMO >> >> I agree. I see some nicer (IMHO) alternatives... Apart from the >> recipe I mentioned >> in the recent post (although I am *not* convinced it should be added >> to the stdlib) >> some decorator-based way may be nice, e.g.: >> >> @namedtuple(fields='length weight is_poisonous') >> class Snake: >> def hiss(self): >> return 'hiss' + self.length * 's' > > > This implies that all namedtuples must be callable. No, not at all. It only means that signature specification of the namedtuple factory function would need to be extended a bit (or a separate decorator, such as namedtuple.subclass, would need to be added as an attribute) to support returning a decorator instead of a ready named tuple type. > Point3D = namedtuple('Point3D', 'x y z') > pt = Point3D(2, 4, 8) > > I don't think it's appropriate for tuples, named or not, to be > callable. I don't understand what do you mean. It's obvious that a named tuple *type* must be callable (as any other instantiable type) and that in 99% of cases named tuple *instances* should not be... Cheers. *j From techtonik at gmail.com Mon Feb 18 08:40:07 2013 From: techtonik at gmail.com (anatoly techtonik) Date: Mon, 18 Feb 2013 10:40:07 +0300 Subject: [Python-ideas] Poll about -h,--help options Message-ID: Hi, Is it interesting to know if people expect -h to work as a --help equivalent by default? As it directly affects best practices of using Python in command line, can we run such poll on python.org? I thought that this stuff is obvious, but it appears that it is not, so I'd like to see this usability study - it will be interesting for the Python community to know itself better. -- anatoly t. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ubershmekel at gmail.com Mon Feb 18 09:07:21 2013 From: ubershmekel at gmail.com (Yuval Greenfield) Date: Mon, 18 Feb 2013 10:07:21 +0200 Subject: [Python-ideas] Poll about -h,--help options In-Reply-To: References: Message-ID: On Mon, Feb 18, 2013 at 9:40 AM, anatoly techtonik wrote: > Hi, > > Is it interesting to know if people expect -h to work as a --help > equivalent by default? > > As it directly affects best practices of using Python in command line, can > we run such poll on python.org? I thought that this stuff is obvious, but > it appears that it is not, so I'd like to see this usability study - it > will be interesting for the Python community to know itself better. > -- > anatoly t. > I hate typing "-h" only to find that just "--help" works. I'm pretty sure it's obvious and standard. http://en.wikipedia.org/wiki/Command-line_interface#Built-in_usage_help Yuval Greenfield -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon Feb 18 10:39:28 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 18 Feb 2013 20:39:28 +1100 Subject: [Python-ideas] Poll about -h,--help options In-Reply-To: References: Message-ID: <5121F6D0.2010303@pearwood.info> On 18/02/13 18:40, anatoly techtonik wrote: > Hi, > > Is it interesting to know if people expect -h to work as a --help > equivalent by default? Yes, absolutely. -- Steven From raymond.hettinger at gmail.com Mon Feb 18 10:46:36 2013 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Mon, 18 Feb 2013 01:46:36 -0800 Subject: [Python-ideas] A subclassing API for named tuples? In-Reply-To: References: Message-ID: <567A53F7-E8A5-4080-A5D2-01A7C08CD434@gmail.com> On Feb 14, 2013, at 5:19 AM, Nick Coghlan wrote: > So, what do people think? Too much magic? Or just the right amount to > allow a cleaner syntax for named tuple definitions, without > inadvertently encouraging people to do bad things to their memory > usage? (Note: for backwards compatibility reasons, we couldn't use a > custom metaclass for the classes returned by the existing collections > namedtuple API. However, we could certainly provide a distinct > collections.NamedTuple type which used a custom metaclass to behave > this way). To me, this smells of over-engineering. For most uses, the current form of named tuple is simple and clean: CacheInfo = namedtuple("CacheInfo", ["hits", "misses", "maxsize", "currsize"]) The current form of namedtuple is also very flexible. The docs for it show how it can easily be subclassed to add computed fields, new reprs, etc. It also works great for implementing prototype instance, enums, etc. And it has a verbose option that makes it self-documenting. The namedtuple API was born out of mixing the best parts of many different implementations found in the field. It went through extensive evaluation, review and refinement as a recipe on ASPN. IMO, there is zero need to wreck its simplicity by throwing metaclass firepower into the mix. We really don't need a second way to do it. For people who care about the memory used by subclasses, it takes less effort to learn how to use slots that it does to learn and remember an second API for namedtuples. After all, the sole justification for __slots__ is to prevent the creation of an instance dictionary. That is what it's for. I think your real issue doesn't have anything to do with named tuples in particular. Instead, your issue is with subclassing *any* class that uses __slots__. Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.jerdonek at gmail.com Mon Feb 18 10:41:12 2013 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Mon, 18 Feb 2013 01:41:12 -0800 Subject: [Python-ideas] Poll about -h,--help options In-Reply-To: References: Message-ID: On Sun, Feb 17, 2013 at 11:40 PM, anatoly techtonik wrote: > Hi, > > Is it interesting to know if people expect -h to work as a --help equivalent > by default? > > As it directly affects best practices of using Python in command line, can > we run such poll on python.org? I thought that this stuff is obvious, but it > appears that it is not, so I'd like to see this usability study - it will be > interesting for the Python community to know itself better. The argparse and optparse modules in the standard library expose help via both -h and --help by default (and this is in the Python documentation and in the help strings generated by those modules), so I would expect Python users at least to be familiar with both invocations: http://docs.python.org/2/library/optparse.html#generating-help http://docs.python.org/2/library/argparse.html#add-help --Chris > -- > anatoly t. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > From ncoghlan at gmail.com Mon Feb 18 10:56:17 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 18 Feb 2013 19:56:17 +1000 Subject: [Python-ideas] A subclassing API for named tuples? In-Reply-To: <567A53F7-E8A5-4080-A5D2-01A7C08CD434@gmail.com> References: <567A53F7-E8A5-4080-A5D2-01A7C08CD434@gmail.com> Message-ID: On Mon, Feb 18, 2013 at 7:46 PM, Raymond Hettinger wrote: > After all, the sole justification for __slots__ is to prevent the creation > of > an instance dictionary. That is what it's for. I think your real issue > doesn't have anything to do with named tuples in particular. > Instead, your issue is with subclassing *any* class that uses __slots__. My real issue is with interminable enum discussions, this thread was merely a side effect of enum-induced frustration :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ubershmekel at gmail.com Mon Feb 18 11:30:54 2013 From: ubershmekel at gmail.com (Yuval Greenfield) Date: Mon, 18 Feb 2013 12:30:54 +0200 Subject: [Python-ideas] Poll about -h,--help options In-Reply-To: <5121F6D0.2010303@pearwood.info> References: <5121F6D0.2010303@pearwood.info> Message-ID: On Mon, Feb 18, 2013 at 11:39 AM, Steven D'Aprano wrote: > On 18/02/13 18:40, anatoly techtonik wrote: > >> Hi, >> >> Is it interesting to know if people expect -h to work as a --help >> equivalent by default? >> > > Yes, absolutely. > > > Which reminds me of the most annoying behavior in existence: user at pc:~$ ln -h ln: invalid option -- 'h' Try `ln --help' for more information. user at pc:~$ Yuval -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan at drees.name Mon Feb 18 11:59:13 2013 From: stefan at drees.name (Stefan Drees) Date: Mon, 18 Feb 2013 11:59:13 +0100 Subject: [Python-ideas] Poll about -h,--help options In-Reply-To: References: Message-ID: <51220981.2020709@drees.name> On 18.02.13 08:40, Anatoly Techtonik wrote: > ...Is it interesting to know if people expect -h to work as a --help > equivalent by default? > > As it directly affects best practices of using Python in command line, > can we run such poll on python.org ? I thought that > this stuff is obvious, but it appears that it is not, so I'd like to see > this usability study - it will be interesting for the Python community > to know itself better. > ... Polls are funny. Why not :-? In the gnu world (at least) appending " -h" MAY for some tools result in output transformed to "human-readable", ie. tools like ls, du or df all support the -h triggers meaning "human readable" output options. Asking for help or versions seems to be not a global priority ;-) All the best, Stefan. From tjreedy at udel.edu Tue Feb 19 01:43:07 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 18 Feb 2013 19:43:07 -0500 Subject: [Python-ideas] Poll about -h,--help options In-Reply-To: References: Message-ID: On 2/18/2013 2:40 AM, anatoly techtonik wrote: > Is it interesting to know if people expect -h to work as a --help > equivalent by default? Of course. Given that it already does for python itself " -? -h --help Print a short description of all command line options. " and at least some of its modules with a command line interface, I am not sure what you are getting at. If some python module is deficient, report it on the tracker. If you think the opt/argparse docs should say more than they do, make a suggestion here, but be prepared to accept a decision to not add anything, on the basis that it is mostly our policy to not tell people how to use Python. (PEP 8 is a style guide for the stdlib, and for other stuff only as people decide to use it.) > As it directly affects best practices of using Python in command line, > can we run such poll on python.org ? I thought that > this stuff is obvious, but it appears that it is not, I have no idea what you are referring to, but if some other software is deficient, take it up with the authors. -- Terry Jan Reedy From raymond.hettinger at gmail.com Tue Feb 19 06:48:48 2013 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Mon, 18 Feb 2013 21:48:48 -0800 Subject: [Python-ideas] A subclassing API for named tuples? In-Reply-To: References: <567A53F7-E8A5-4080-A5D2-01A7C08CD434@gmail.com> Message-ID: <05C3D9F7-111B-4AC3-AB76-D15AEBEB01CD@gmail.com> On Feb 18, 2013, at 1:56 AM, Nick Coghlan wrote: > My real issue is with interminable enum discussions, this thread was > merely a side effect of enum-induced frustration :) I understand. It never ends :-) Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Tue Feb 19 10:24:06 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 19 Feb 2013 10:24:06 +0100 Subject: [Python-ideas] A subclassing API for named tuples? References: <567A53F7-E8A5-4080-A5D2-01A7C08CD434@gmail.com> <05C3D9F7-111B-4AC3-AB76-D15AEBEB01CD@gmail.com> Message-ID: <20130219102406.735059be@pitrou.net> Le Mon, 18 Feb 2013 21:48:48 -0800, Raymond Hettinger a ?crit : > > On Feb 18, 2013, at 1:56 AM, Nick Coghlan > wrote: > > > My real issue is with interminable enum discussions, this thread was > > merely a side effect of enum-induced frustration :) > > I understand. It never ends :-) The problem with enum is that since it doesn't unlock any particularly interesting feature, it must at least cater to all existing use cases to be desirable for stdlib inclusion. Hence the long discussions. Regards Antoine. From miki.tebeka at gmail.com Tue Feb 19 17:03:16 2013 From: miki.tebeka at gmail.com (Miki Tebeka) Date: Tue, 19 Feb 2013 08:03:16 -0800 (PST) Subject: [Python-ideas] argparse - add support for environment variables Message-ID: <71869e7d-605c-4599-8010-0c195e86e982@googlegroups.com> Greetings, The usual way of resolving configuration is command line -> environment -> default. Currently argparse supports only command line -> default, I'd like to suggest an optional "env" keyword to add_argument that will also resolve from environment. (And also optional env dictionary to the ArgumentParser __init__ method [or to parse_args], which will default to os.environ). Example: [spam.py] parser = ArgumentParser() parser.add_argument('--spam', env='SPAM', default=7) args = parser.parse_args() print(args.spam) ./spam.py -> 7 ./spam.py --spam=12 -> 12 SPAM=9 ./spam.py -> 9 SPAM=9 ./spam.py --spam=12 -> 12 What do you think? -- Miki -------------- next part -------------- An HTML attachment was scrubbed... URL: From graffatcolmingov at gmail.com Tue Feb 19 17:39:24 2013 From: graffatcolmingov at gmail.com (Ian Cordasco) Date: Tue, 19 Feb 2013 11:39:24 -0500 Subject: [Python-ideas] Fwd: argparse - add support for environment variables In-Reply-To: References: <71869e7d-605c-4599-8010-0c195e86e982@googlegroups.com> Message-ID: ---------- Forwarded message ---------- From: Ian Cordasco Date: Tue, Feb 19, 2013 at 11:35 AM Subject: Re: [Python-ideas] argparse - add support for environment variables To: Miki Tebeka Cc: python-ideas at googlegroups.com Why not: parser.add_argument('--spam', default=os.environ.get('SPAM', 7)) This way if SPAM isn't set, your default is 7. If spam is set, your default becomes that. On Tue, Feb 19, 2013 at 11:03 AM, Miki Tebeka wrote: > Greetings, > > The usual way of resolving configuration is command line -> environment -> > default. > Currently argparse supports only command line -> default, I'd like to > suggest an optional "env" keyword to add_argument that will also resolve > from environment. (And also optional env dictionary to the ArgumentParser > __init__ method [or to parse_args], which will default to os.environ). > > Example: > [spam.py] > > parser = ArgumentParser() > > parser.add_argument('--spam', env='SPAM', default=7) > args = parser.parse_args() > print(args.spam) > > ./spam.py -> 7 > ./spam.py --spam=12 -> 12 > SPAM=9 ./spam.py -> 9 > SPAM=9 ./spam.py --spam=12 -> 12 > > What do you think? > -- > Miki GMail decided to reply to python-ideas at googlegroups.com, but this was my response which I shamefully did not bottom post. From miki.tebeka at gmail.com Wed Feb 20 00:51:08 2013 From: miki.tebeka at gmail.com (Miki Tebeka) Date: Tue, 19 Feb 2013 15:51:08 -0800 (PST) Subject: [Python-ideas] Fwd: argparse - add support for environment variables In-Reply-To: References: <71869e7d-605c-4599-8010-0c195e86e982@googlegroups.com> Message-ID: > Why not: > > parser.add_argument('--spam', default=os.environ.get('SPAM', 7)) > Mainly because then you can't swap in different dictionaries to resolve SPAM (when specifying "env" to ArgumentParser). The other reason is that it's making common code shorter, which is a win IMO. -------------- next part -------------- An HTML attachment was scrubbed... URL: From moloney at ohsu.edu Wed Feb 20 02:54:58 2013 From: moloney at ohsu.edu (Brendan Moloney) Date: Tue, 19 Feb 2013 17:54:58 -0800 Subject: [Python-ideas] Fwd: argparse - add support for environment variables In-Reply-To: References: <71869e7d-605c-4599-8010-0c195e86e982@googlegroups.com> , Message-ID: <5E25C96030E66B44B9CFAA95D3DE59194675672FA1@EX-MB08.ohsu.edu> This is something I wanted before (as well as loading defaults from per user config files). A solution was proposed a while back, but it was never merged due to a lack of consensus on the API: http://code.google.com/p/argparse/issues/detail?id=35 Not sure if there is a more recent discussion. ________________________________________ From: Python-ideas [python-ideas-bounces+moloney=ohsu.edu at python.org] On Behalf Of Miki Tebeka [miki.tebeka at gmail.com] Sent: Tuesday, February 19, 2013 3:51 PM To: python-ideas at googlegroups.com Cc: python-ideas Subject: Re: [Python-ideas] Fwd: argparse - add support for environment variables Why not: parser.add_argument('--spam', default=os.environ.get('SPAM', 7)) Mainly because then you can't swap in different dictionaries to resolve SPAM (when specifying "env" to ArgumentParser). The other reason is that it's making common code shorter, which is a win IMO. From rovitotv at gmail.com Wed Feb 20 03:48:16 2013 From: rovitotv at gmail.com (Todd Rovito) Date: Tue, 19 Feb 2013 21:48:16 -0500 Subject: [Python-ideas] Introducing PEP 434 -- IDLE Enhancement Exception for All Branches Message-ID: Please see http://www.python.org/dev/peps/pep-0434/ for the complete PEP. The idea for the PEP was generated because a right click menu was added to IDLE in Python 2.7 this started a debate about bug fix VS enhancement see the PEP references for more information. IDLE has many patches that already exist but are difficult to get committed. This PEP is designed to make it easier to bring IDLE up to speed with modern GUI standards. The PEP is not perfect some issues that need to be solved: -How do we test IDLE patches to make sure they are high quality? Apparently the build bots don't test user interface issues. My philosophy is keep it simple so I was thinking that a simple procedure for testing on each of the major platforms should be performed before a commit. Creating a build bot that tests user interfaces seems difficult to me and could put IDLE improvements even more behind. -Does it make sense to separate IDLE from the stdlib for Python 3.4? I understand the batteries included argument, but if IDLE development is going to continue to proceed at a more accelerated pace than Python, it might make sense to separate it out into its own project or a sub-project. Please provide comments or other concerns. Thanks. From eliben at gmail.com Wed Feb 20 05:29:48 2013 From: eliben at gmail.com (Eli Bendersky) Date: Tue, 19 Feb 2013 20:29:48 -0800 Subject: [Python-ideas] Fwd: argparse - add support for environment variables In-Reply-To: References: <71869e7d-605c-4599-8010-0c195e86e982@googlegroups.com> Message-ID: On Tue, Feb 19, 2013 at 3:51 PM, Miki Tebeka wrote: > > Why not: >> >> parser.add_argument('--spam', default=os.environ.get('SPAM', 7)) >> > Mainly because then you can't swap in different dictionaries to resolve > SPAM (when specifying "env" to ArgumentParser). > Can you elaborate on this requirement? > The other reason is that it's making common code shorter, which is a win > IMO. > How common is such code? YMMV here... I don't think it's nearly common enough to warrant adding complexity to a stdlib module in order to implement a non-orthogonal feature. In general, I'm -1 on the idea. argparse is for parsing command line arguments. Python provides you convenient access to env vars via os.environ. Why mix up the two? Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Wed Feb 20 06:10:29 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 20 Feb 2013 16:10:29 +1100 Subject: [Python-ideas] Introducing PEP 434 -- IDLE Enhancement Exception for All Branches In-Reply-To: References: Message-ID: <20130220051028.GA22784@ando> On Tue, Feb 19, 2013 at 09:48:16PM -0500, Todd Rovito wrote: > -Does it make sense to separate IDLE from the stdlib for Python 3.4? > I understand the batteries included argument, but if IDLE development > is going to continue to proceed at a more accelerated pace than > Python, it might make sense to separate it out into its own project or > a sub-project. Personally, I don't use IDLE, but I know many people who do. I think it is important for them that IDLE remains in the std lib. We could have a stable version of IDLE in the std lib, and another version available on PyPI. The PyPI version could change more rapidly, since it doesn't have to fit in with the release schedule for the std lib. As the release schedule allows, the two branches could be merged. I have no idea if this has been tried before, but I think this would work better for an application like IDLE than for a library. -- Steven From ericsnowcurrently at gmail.com Wed Feb 20 06:34:53 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Tue, 19 Feb 2013 22:34:53 -0700 Subject: [Python-ideas] capturing the origin of objects Message-ID: With the removal of unbound methods in Python 3, introspecting the class on which a method was defined is not so simple. Python 3.3 gives us __qualname__ for classes and functions (see PEP 3155), which helps somewhat by giving us a string composed at compile-time. However, I was considering something a little more powerful. I propose that what would have formerly been an unbound method get a new attribute, __origin__. It will be bound to class where the method is defined. Motivation ========== One motivator for this proposal is http://bugs.python.org/issue15582, allowing inspect.getdoc() to "inherit" docstrings. Without a concrete connection between a method and the "origin" class, dynamically determining the docstring is basically a non-starter. The __origin__ attribute would help. Extended Idea ============= The name, __origin__, is a bit generic to accommodate the possible extension of the proposal to other objects: * f.__origin__ (the module or function (nested) where the function was defined) * cls.__origin__ (the module, class (nested), or function where the class was defined) * code.__origin__ (the function for which the code object was originally created) * module.__origin__ (a.k.a. module.__file__) For functions and classes defined in a function, __origin__ would be bound to the function rather than the locals or the code object. For modules, __origin__ is more accurate in the cases that the module was generated from something other than a file. It would still be a string, though. Also, we currently have a convention of setting __module__ to the name of the module rather than the module itself. Whether to break with that convention for __origin__ is an open question which relates to how __origin__ would be used. Conceivably, each use case for __origin__ could be covered by a name more specific to the object, e.g. module.__file__. However, having the one name would help make the purpose clear and consistent. Objections ========== The downside of binding the objects to __origin__ is in memory usage and, particularly, in ref-counts/reference-cycles. I expect that this is where the biggest objections will lie. However, my understanding is that this is less of an issue now than it used to be. If need be, weakref proxies could address some of the concerns. The status quo has a gap in the case of "unbound" methods and of code objects, though __qualname__ does offer an improvement for methods. The actual use case I have relates strictly to those methods. Having something like __origin__ for those methods, at least, would be helpful. -eric From vito.detullio at gmail.com Wed Feb 20 07:59:18 2013 From: vito.detullio at gmail.com (Vito De Tullio) Date: Wed, 20 Feb 2013 07:59:18 +0100 Subject: [Python-ideas] Fwd: argparse - add support for environment variables References: <71869e7d-605c-4599-8010-0c195e86e982@googlegroups.com> Message-ID: Eli Bendersky wrote: > In general, I'm -1 on the idea. argparse is for parsing command line > arguments. Python provides you convenient access to env vars via > os.environ. Why mix up the two? well, command line arguments, environmental variables and, I add, configuration files are commonly intertwined. While argparse focus on the first one, there are cases where you want some "layer" on top of it and on os.environ and on... ConfigParser?) to DRY. -- ZeD From chris.jerdonek at gmail.com Wed Feb 20 09:40:57 2013 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Wed, 20 Feb 2013 00:40:57 -0800 Subject: [Python-ideas] Fwd: argparse - add support for environment variables In-Reply-To: References: <71869e7d-605c-4599-8010-0c195e86e982@googlegroups.com> Message-ID: On Tue, Feb 19, 2013 at 3:51 PM, Miki Tebeka wrote: > >> Why not: >> >> parser.add_argument('--spam', default=os.environ.get('SPAM', 7)) > > Mainly because then you can't swap in different dictionaries to resolve > SPAM (when specifying "env" to ArgumentParser). Couldn't you achieve this with a simple helper function? Something along the lines of-- env = {'SPAM': 9} parser = argparse.ArgumentParser() def add_argument(key, *args, **kwargs): if key in env: kwargs['default'] = env[key] parser.add_argument(*args, **kwargs) add_argument('SPAM', '--spam', default=7) ... This also allows env to be an arbitrary dictionary. --Chris From barry at python.org Wed Feb 20 15:04:26 2013 From: barry at python.org (Barry Warsaw) Date: Wed, 20 Feb 2013 09:04:26 -0500 Subject: [Python-ideas] Introducing PEP 434 -- IDLE Enhancement Exception for All Branches References: <20130220051028.GA22784@ando> Message-ID: <20130220090426.66ebbe8e@anarchist.wooz.org> On Feb 20, 2013, at 04:10 PM, Steven D'Aprano wrote: >We could have a stable version of IDLE in the std lib, and another >version available on PyPI. The PyPI version could change more rapidly, >since it doesn't have to fit in with the release schedule for the std >lib. As the release schedule allows, the two branches could be merged. > >I have no idea if this has been tried before, but I think this would >work better for an application like IDLE than for a library. There's some precedence for this I think, e.g. mock, unittest2, email. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From ncoghlan at gmail.com Wed Feb 20 15:30:49 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 21 Feb 2013 00:30:49 +1000 Subject: [Python-ideas] Introducing PEP 434 -- IDLE Enhancement Exception for All Branches In-Reply-To: <20130220090426.66ebbe8e@anarchist.wooz.org> References: <20130220051028.GA22784@ando> <20130220090426.66ebbe8e@anarchist.wooz.org> Message-ID: On Thu, Feb 21, 2013 at 12:04 AM, Barry Warsaw wrote: > On Feb 20, 2013, at 04:10 PM, Steven D'Aprano wrote: > >>We could have a stable version of IDLE in the std lib, and another >>version available on PyPI. The PyPI version could change more rapidly, >>since it doesn't have to fit in with the release schedule for the std >>lib. As the release schedule allows, the two branches could be merged. >> >>I have no idea if this has been tried before, but I think this would >>work better for an application like IDLE than for a library. > > There's some precedence for this I think, e.g. mock, unittest2, email. This was the plan for packaging vs distutils2, and I'll be proposing something similar for distlib, too. It can be a little tricky to manage at times, but you can do truly experimental features in the PyPI version first with a warning that they're experimental (e.g the various incarnations contextlib.ExitStack went through when I first added it to contextlib2), while bug fixes and features you're more confident about can go directly into hg.python.org and then be published early through the downstream project. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From abarnert at yahoo.com Wed Feb 20 18:32:48 2013 From: abarnert at yahoo.com (Andrew Barnert) Date: Wed, 20 Feb 2013 09:32:48 -0800 Subject: [Python-ideas] Introducing PEP 434 -- IDLE Enhancement Exception for All Branches In-Reply-To: <20130220090426.66ebbe8e@anarchist.wooz.org> References: <20130220051028.GA22784@ando> <20130220090426.66ebbe8e@anarchist.wooz.org> Message-ID: On Feb 20, 2013, at 6:04, Barry Warsaw wrote: > On Feb 20, 2013, at 04:10 PM, Steven D'Aprano wrote: > >> We could have a stable version of IDLE in the std lib, and another >> version available on PyPI. The PyPI version could change more rapidly, >> since it doesn't have to fit in with the release schedule for the std >> lib. As the release schedule allows, the two branches could be merged. >> >> I have no idea if this has been tried before, but I think this would >> work better for an application like IDLE than for a library. > > There's some precedence for this I think, e.g. mock, unittest2, email. And sqlite3. Personally, I've found it easier to track down certain changes in sqlite3 than in other modules. The Python docs are very clear about when a new function was added, but not always as clear about, e.g., performance characteristics or internal behavior with indirect external effects. I've found it easier in some cases to find that I'm relying on, say, pysqlite >= 2.5.0 behavior than python >= 2.6. And then mapping pysqlite >= 2.5.0 to Python versions is easy. And this might be even more relevant for IDLE than for a module. And of course it gives me a way to deal with the changes. On a machine that has 2.5, I can just pip install pysqlite. And I can certainly see it being useful to pip-3.3 install idle to get IDLE 3.4.1 on a machine with python 3.3. I don't think these are _major_ selling points, but they're at least minor ones, which may help counter the obvious downside (merging isn't free). From dholth at gmail.com Wed Feb 20 18:58:24 2013 From: dholth at gmail.com (Daniel Holth) Date: Wed, 20 Feb 2013 12:58:24 -0500 Subject: [Python-ideas] Introducing PEP 434 -- IDLE Enhancement Exception for All Branches In-Reply-To: References: <20130220051028.GA22784@ando> <20130220090426.66ebbe8e@anarchist.wooz.org> Message-ID: Separate set of rules for features classified as "standard applications" versus "standard libraries"? -------------- next part -------------- An HTML attachment was scrubbed... URL: From miki.tebeka at gmail.com Wed Feb 20 19:10:33 2013 From: miki.tebeka at gmail.com (Miki Tebeka) Date: Wed, 20 Feb 2013 10:10:33 -0800 (PST) Subject: [Python-ideas] Fwd: argparse - add support for environment variables In-Reply-To: References: <71869e7d-605c-4599-8010-0c195e86e982@googlegroups.com> Message-ID: <5cca4189-c01d-44e6-9903-6be5f2bb58d7@googlegroups.com> On Tuesday, February 19, 2013 8:29:48 PM UTC-8, Eli Bendersky wrote: > > On Tue, Feb 19, 2013 at 3:51 PM, Miki Tebeka > > wrote: > >> >> Why not: >>> >>> parser.add_argument('--spam', default=os.environ.get('SPAM', 7)) >>> >> Mainly because then you can't swap in different dictionaries to resolve >> SPAM (when specifying "env" to ArgumentParser). >> > > Can you elaborate on this requirement? > Sure. The idea is the you have some kind of "parameter resolution" (very much like the interpreter is resolving variables). You first look into the command line, and if not found you query a dictionary. Most times, this dictionary will be os.environ. However you can pass your own dictionary to resolve the parameters (for testing, configuration files). You can even use something like Raymonds Nested Contextsfor that matter. -------------- next part -------------- An HTML attachment was scrubbed... URL: From fuzzyman at gmail.com Wed Feb 20 20:18:25 2013 From: fuzzyman at gmail.com (Michael Foord) Date: Wed, 20 Feb 2013 19:18:25 +0000 Subject: [Python-ideas] Fwd: argparse - add support for environment variables In-Reply-To: References: <71869e7d-605c-4599-8010-0c195e86e982@googlegroups.com> Message-ID: On 20 February 2013 06:59, Vito De Tullio wrote: > Eli Bendersky wrote: > > > In general, I'm -1 on the idea. argparse is for parsing command line > > arguments. Python provides you convenient access to env vars via > > os.environ. Why mix up the two? > > well, command line arguments, environmental variables and, I add, > configuration files are commonly intertwined. > > While argparse focus on the first one, there are cases where you want some > "layer" on top of it and on os.environ and on... ConfigParser?) to DRY. > > configglue combines support for config files (and hierarchies of config files) with command line arguments. https://pypi.python.org/pypi/configglue/ Michael > -- > ZeD > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Wed Feb 20 23:07:57 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 20 Feb 2013 17:07:57 -0500 Subject: [Python-ideas] Introducing PEP 434 -- IDLE Enhancement Exception for All Branches In-Reply-To: References: <20130220051028.GA22784@ando> <20130220090426.66ebbe8e@anarchist.wooz.org> Message-ID: On 2/20/2013 9:30 AM, Nick Coghlan wrote: > On Thu, Feb 21, 2013 at 12:04 AM, Barry Warsaw wrote: >> On Feb 20, 2013, at 04:10 PM, Steven D'Aprano wrote: >> >>> We could have a stable version of IDLE in the std lib, and another >>> version available on PyPI. The PyPI version could change more rapidly, >>> since it doesn't have to fit in with the release schedule for the std >>> lib. As the release schedule allows, the two branches could be merged. >>> >>> I have no idea if this has been tried before, but I think this would >>> work better for an application like IDLE than for a library. >> >> There's some precedence for this I think, e.g. mock, unittest2, email. > > This was the plan for packaging vs distutils2, and I'll be proposing > something similar for distlib, too. > > It can be a little tricky to manage at times, but you can do truly > experimental features in the PyPI version first with a warning that > they're experimental (e.g the various incarnations > contextlib.ExitStack went through when I first added it to > contextlib2), while bug fixes and features you're more confident about > can go directly into hg.python.org and then be published early through > the downstream project. Most of the actual issues that prompted this proposal are minor changes, such as to the find or replace dialog. Some might even be considered bug fixes rather than enhancements, depending on one's interpretation of the vague doc for such. This PEP is partly about whether we need to classify such changes, or just put them into whatever release we feel appropriate. Whatever the decision, those changes should directly go into whatever version they are going to go in. On the other hand, I agree that a PyPI preview release of a rewrite of IDLE to use the newer ttk widgets would be a good idea. But I personally would base such a project on 3.3 and consider whether it should require tcl/tk 8.6 rather than 8.5. And I would only put it in the stdlib in a new release (perhaps alongside the existing IDLE for at least one release). So this PEP is not relevance to such a project. -- Terry Jan Reedy From solipsis at pitrou.net Thu Feb 21 00:12:44 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 21 Feb 2013 00:12:44 +0100 Subject: [Python-ideas] Introducing PEP 434 -- IDLE Enhancement Exception for All Branches References: <20130220051028.GA22784@ando> <20130220090426.66ebbe8e@anarchist.wooz.org> Message-ID: <20130221001244.1b602cfd@pitrou.net> On Wed, 20 Feb 2013 17:07:57 -0500 Terry Reedy wrote: > > On the other hand, I agree that a PyPI preview release of a rewrite of > IDLE to use the newer ttk widgets would be a good idea. But I personally > would base such a project on 3.3 and consider whether it should require > tcl/tk 8.6 rather than 8.5. And I would only put it in the stdlib in a > new release (perhaps alongside the existing IDLE for at least one > release). So this PEP is not relevance to such a project. The only thing relevant to such a project is to find someone actually motivated to do it. What is IDLE's actual maintenance activity, exactly? I am sympathetic to the PEP but, really, thinking IDLE's development is hampered by Python's release process is *completely* ridiculous. If you want IDLE development to happen, please stop talking and start reviewing / committing patches. FTR, I'm personally +1 on yanking IDLE out of 3.4. Regards Antoine. From vinay_sajip at yahoo.co.uk Thu Feb 21 01:07:55 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 21 Feb 2013 00:07:55 +0000 (UTC) Subject: [Python-ideas] =?utf-8?q?Fwd=3A_argparse_-_add_support_for_enviro?= =?utf-8?q?nment=09variables?= References: <71869e7d-605c-4599-8010-0c195e86e982@googlegroups.com> Message-ID: Michael Foord writes: > > configglue combines support for config files (and hierarchies of config files) with command line arguments. > > https://pypi.python.org/pypi/configglue/ > So does the config module: http://www.red-dove.com/config-doc/ Regards, Vinay Sajip From eliben at gmail.com Thu Feb 21 01:18:50 2013 From: eliben at gmail.com (Eli Bendersky) Date: Wed, 20 Feb 2013 16:18:50 -0800 Subject: [Python-ideas] Fwd: argparse - add support for environment variables In-Reply-To: <5cca4189-c01d-44e6-9903-6be5f2bb58d7@googlegroups.com> References: <71869e7d-605c-4599-8010-0c195e86e982@googlegroups.com> <5cca4189-c01d-44e6-9903-6be5f2bb58d7@googlegroups.com> Message-ID: On Wed, Feb 20, 2013 at 10:10 AM, Miki Tebeka wrote: > > > On Tuesday, February 19, 2013 8:29:48 PM UTC-8, Eli Bendersky wrote: > >> On Tue, Feb 19, 2013 at 3:51 PM, Miki Tebeka wrote: >> >>> >>> Why not: >>>> >>>> parser.add_argument('--spam', default=os.environ.get('SPAM', 7)) >>>> >>> Mainly because then you can't swap in different dictionaries to resolve >>> SPAM (when specifying "env" to ArgumentParser). >>> >> >> Can you elaborate on this requirement? >> > > Sure. The idea is the you have some kind of "parameter resolution" (very > much like the interpreter is resolving variables). You first look into the > command line, and if not found you query a dictionary. Most times, this > dictionary will be os.environ. However you can pass your own dictionary to > resolve the parameters (for testing, configuration files). You can even use > something like Raymonds Nested Contextsfor that matter. > What about config files? Mix them into argparse too? Oh, and support for .ini/XML/YAML/JSON/sqlite configs, please! Sarcasm aside (don't take it personally ;-), my point is this: argparse is a nice, orthogonal library for dealing with command-line arguments. Let's keep it this way. Python gives you tools to *very easily* mix up different configuration paths, with really minimal effort. Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Feb 21 01:44:44 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 21 Feb 2013 10:44:44 +1000 Subject: [Python-ideas] Fwd: argparse - add support for environment variables In-Reply-To: References: <71869e7d-605c-4599-8010-0c195e86e982@googlegroups.com> <5cca4189-c01d-44e6-9903-6be5f2bb58d7@googlegroups.com> Message-ID: On Thu, Feb 21, 2013 at 10:18 AM, Eli Bendersky wrote: > What about config files? Mix them into argparse too? Oh, and support for > .ini/XML/YAML/JSON/sqlite configs, please! > > Sarcasm aside (don't take it personally ;-), my point is this: argparse is a > nice, orthogonal library for dealing with command-line arguments. Let's keep > it this way. Python gives you tools to *very easily* mix up different > configuration paths, with really minimal effort. Indeed - a layered architecture is the way to go here. argparse can be a *component* of a comprehensive configuration system, but it shouldn't try to *become* a comprehensive configuration system. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From tjreedy at udel.edu Thu Feb 21 02:41:29 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 20 Feb 2013 20:41:29 -0500 Subject: [Python-ideas] Introducing PEP 434 -- IDLE Enhancement Exception for All Branches In-Reply-To: <20130221001244.1b602cfd@pitrou.net> References: <20130220051028.GA22784@ando> <20130220090426.66ebbe8e@anarchist.wooz.org> <20130221001244.1b602cfd@pitrou.net> Message-ID: On 2/20/2013 6:12 PM, Antoine Pitrou wrote: > On Wed, 20 Feb 2013 17:07:57 -0500 > Terry Reedy wrote: >> >> On the other hand, I agree that a PyPI preview release of a rewrite of >> IDLE to use the newer ttk widgets would be a good idea. But I personally >> would base such a project on 3.3 and consider whether it should require >> tcl/tk 8.6 rather than 8.5. And I would only put it in the stdlib in a >> new release (perhaps alongside the existing IDLE for at least one >> release). So this PEP is not relevance to such a project. > > The only thing relevant to such a project is to find someone actually > motivated to do it. Right. That I what I was trying to say. > What is IDLE's actual maintenance activity, exactly? I am not sure whether you are asking about volume or focus. A year ago, focus was on bugs that caused IDLE to quit because of an uncaught exception. When IDLE is started on Windows from an icon, this looks like a crash because there is no visible traceback or exit message. Recently, other issues have been worked on. In the last 5 months, activity has picked up and there are about 40 issues in 3.4 misc/news whose commit message contains 'idle'. > I am sympathetic to the PEP Even though I am willing for the PEP to be rejected, great. I say this because I think it better for the community at large, even though a strict policy might be easier, on net, for developers. > but, really, thinking IDLE's development is hampered by Python's > release process is *completely* ridiculous. I do not believe I have said that and certainly have not meant to say that. What I have said or meant to say is that uncertainly and disagreement about how it does and should fit into the release process can be a hindrance. > If you want IDLE development > to happen, please stop talking and start reviewing / committing patches. I have, of course, done some of both in the past year, even if not currently. As I remember, the review process for at least one issue got hung up on whether a (small) change was a 'bugfix' or 'enhancement' and if the latter, whether it could go into all versions or just default. As we move from obvious bug issue to more ambiguous issues, this question would come up more often. > FTR, I'm personally +1 on yanking IDLE out of 3.4. That would tend to keep 3.4 out of some classrooms. See, for example, http://search.gmane.org/?query=&author=Jean-Paul.Roy%40unice.fr&group=gmane.comp.python.idle&sort=relevance&DEFAULTOP=and&query= from a French college teacher. (The issue was fixed by a tk fix last March.) But of course, this is a different subject. -- Terry Jan Reedy From andre.roberge at gmail.com Thu Feb 21 02:44:33 2013 From: andre.roberge at gmail.com (Andre Roberge) Date: Wed, 20 Feb 2013 21:44:33 -0400 Subject: [Python-ideas] Introducing PEP 434 -- IDLE Enhancement Exception for All Branches In-Reply-To: References: <20130220051028.GA22784@ando> <20130220090426.66ebbe8e@anarchist.wooz.org> <20130221001244.1b602cfd@pitrou.net> Message-ID: >From what I have read on edu-sig, I believe it would be a huge mistake to remove IDLE from the Python standard library. On Wed, Feb 20, 2013 at 9:41 PM, Terry Reedy wrote: > On 2/20/2013 6:12 PM, Antoine Pitrou wrote: > >> On Wed, 20 Feb 2013 17:07:57 -0500 >> Terry Reedy wrote: >> >>> >>> On the other hand, I agree that a PyPI preview release of a rewrite of >>> IDLE to use the newer ttk widgets would be a good idea. But I personally >>> would base such a project on 3.3 and consider whether it should require >>> tcl/tk 8.6 rather than 8.5. And I would only put it in the stdlib in a >>> new release (perhaps alongside the existing IDLE for at least one >>> release). So this PEP is not relevance to such a project. >>> >> >> The only thing relevant to such a project is to find someone actually >> motivated to do it. >> > > Right. That I what I was trying to say. > > What is IDLE's actual maintenance activity, exactly? >> > > I am not sure whether you are asking about volume or focus. A year ago, > focus was on bugs that caused IDLE to quit because of an uncaught > exception. When IDLE is started on Windows from an icon, this looks like a > crash because there is no visible traceback or exit message. Recently, > other issues have been worked on. In the last 5 months, activity has picked > up and there are about 40 issues in 3.4 misc/news whose commit message > contains 'idle'. > > > I am sympathetic to the PEP > > Even though I am willing for the PEP to be rejected, great. I say this > because I think it better for the community at large, even though a strict > policy might be easier, on net, for developers. > > but, really, thinking IDLE's development is hampered by Python's >> release process is *completely* ridiculous. >> > > I do not believe I have said that and certainly have not meant to say > that. What I have said or meant to say is that uncertainly and disagreement > about how it does and should fit into the release process can be a > hindrance. > > > If you want IDLE development > >> to happen, please stop talking and start reviewing / committing patches. >> > > I have, of course, done some of both in the past year, even if not > currently. As I remember, the review process for at least one issue got > hung up on whether a (small) change was a 'bugfix' or 'enhancement' and if > the latter, whether it could go into all versions or just default. As we > move from obvious bug issue to more ambiguous issues, this question would > come up more often. > > FTR, I'm personally +1 on yanking IDLE out of 3.4. >> > > That would tend to keep 3.4 out of some classrooms. See, for example, > http://search.gmane.org/?**query=&author=Jean-Paul.Roy%** > 40unice.fr&group=gmane.comp.**python.idle&sort=relevance&** > DEFAULTOP=and&query= > from a French college teacher. (The issue was fixed by a tk fix last > March.) But of course, this is a different subject. > > -- > Terry Jan Reedy > > ______________________________**_________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/**mailman/listinfo/python-ideas > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert2682 at verizon.net Thu Feb 21 03:11:01 2013 From: robert2682 at verizon.net (robert2682) Date: Wed, 20 Feb 2013 21:11:01 -0500 Subject: [Python-ideas] Functions and Unbound methods Message-ID: <51258235.3090802@verizon.net> Hi, I'm new; greetings all! I'm not sure if this is a bug or feature, but it confused me so I thought I'd raise the issue. class a: def b (self): pass foo = ('Hello', b) class c(a): def d(self): t = type (self. __class__. foo [1]) print t t = type (self. __class__. b) print t e = c () e. d() prints for the first print, and it seems to me it should be an instancemethod I'm trying to something like this class EditPage: additonal_buttons = () def __init__ (self): buts = [] for x in addional_butons: if isinstance (x [1], types. UnboundMethodType): # fails because type (x [1]) is function, not UnboundMethod buts. append ((x [0], types. MethodType (x [1], self))) else: buts. append (x) class TreePage(EditPage): def EditAsText (self): pass additional_buttons = (('EditAsText', EditAsText),) Thanks Robert Kaplan robert2682 at verizon.net From roger.serwy at gmail.com Thu Feb 21 03:45:04 2013 From: roger.serwy at gmail.com (serwy) Date: Wed, 20 Feb 2013 20:45:04 -0600 Subject: [Python-ideas] Introducing PEP 434 -- IDLE Enhancement Exception for All Branches In-Reply-To: <20130221001244.1b602cfd@pitrou.net> References: <20130221001244.1b602cfd@pitrou.net> Message-ID: <51258A30.8090809@illinois.edu> Hi Antoine, I am the developer of the IdleX project (http://idlex.sourceforge.net). It's not a fork of IDLE, but just a collection of extensions that can be easily merged into IDLE. I started the project because of the slow pace of IDLE development, especially when having several outstanding patches languishing in the bug tracker. With your help I'd like to push IDLE forward. - Roger From python at mrabarnett.plus.com Thu Feb 21 04:50:21 2013 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 21 Feb 2013 03:50:21 +0000 Subject: [Python-ideas] Functions and Unbound methods In-Reply-To: <51258235.3090802@verizon.net> References: <51258235.3090802@verizon.net> Message-ID: <5125997D.4090608@mrabarnett.plus.com> On 2013-02-21 02:11, robert2682 wrote: > > > Hi, > > I'm new; greetings all! > > I'm not sure if this is a bug or feature, but it confused me so I thought > I'd raise the issue. > > > class a: > def b (self): > pass > > foo = ('Hello', b) > > class c(a): > def d(self): > t = type (self. __class__. foo [1]) > print t > t = type (self. __class__. b) > print t > > e = c () > e. d() > > prints for the first print, > and it seems to me it should be an instancemethod > [snip] I think what's happening is that it's defining 'b' as a function in the class's namespace, storing a reference to that function in the tuple, and then, when the class definition ends, it's wrapping the function as a method. You'll find: >>> a.foo[1] >>> a.b >>> a.b.im_func is a.foo[1] True From tjreedy at udel.edu Thu Feb 21 05:31:01 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 20 Feb 2013 23:31:01 -0500 Subject: [Python-ideas] Introducing PEP 434 -- IDLE Enhancement Exception for All Branches In-Reply-To: References: Message-ID: On 2/19/2013 9:48 PM, Todd Rovito wrote: > Please see http://www.python.org/dev/peps/pep-0434/ for the complete > PEP. The idea for the PEP was generated because a right click menu > was added to IDLE in Python 2.7 this started a debate about bug fix VS > enhancement see the PEP references for more information. IDLE has > many patches that already exist but are difficult to get committed. > This PEP is designed to make it easier to bring IDLE up to speed with > modern GUI standards. The PEP is not perfect some issues that need to > be solved: In my view, the PEP is definitely about allowing minor enhancements to existing features in bugfix releases. It is maybe about backporting new features such as would require a new entry on the menus. The right-click menu does not fit either category but it is more like the latter. In any case, I think it is an innocuous addition and should be allowed if a committer wants to do the backport. I do not think the PEP should be a blank check to backport currently hypothetical major revisions such a themed widgets or tabbed windows. If and when such a thing is tested and ready to be put into the 'next' version, the possibility of backporting it should be a separate discussion. > -How do we test IDLE patches to make sure they are high quality? This is a separate issue from the PEP. Those of us who have and are working on IDLE issues are aware of the need to test on Windows, Linux, and Mac, and to test 2.7 separately from 3.x. There is a tracker issue for improving IDLE tests. > -Does it make sense to separate IDLE from the stdlib for Python 3.4? Not to me. > I understand the batteries included argument, but if IDLE development > is going to continue to proceed at a more accelerated pace than > Python, it might make sense to separate it out into its own project or > a sub-project. We typically release bugfixes about every 6-9 months and I think that is often enough to release changes. The subject of the PEP is what should go into those releases. -- Terry Jan Reedy From tjreedy at udel.edu Thu Feb 21 05:37:43 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 20 Feb 2013 23:37:43 -0500 Subject: [Python-ideas] Introducing PEP 434 -- IDLE Enhancement Exception for All Branches In-Reply-To: References: <20130220051028.GA22784@ando> <20130220090426.66ebbe8e@anarchist.wooz.org> <20130221001244.1b602cfd@pitrou.net> Message-ID: <5125A497.9040905@udel.edu> On 2/20/2013 8:41 PM, Terry Reedy wrote: > On 2/20/2013 6:12 PM, Antoine Pitrou wrote: >> but, really, thinking IDLE's development is hampered by Python's >> release process is *completely* ridiculous. > > I do not believe I have said that and certainly have not meant to say > that Re-reading Todd's original post, I see that he did suggest that as a possibility. I expressed my disagreement with this red herring in a direct response to his post. -- Terry Jan Reedy From tjreedy at udel.edu Thu Feb 21 05:39:38 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 20 Feb 2013 23:39:38 -0500 Subject: [Python-ideas] Introducing PEP 434 -- IDLE Enhancement Exception for All Branches In-Reply-To: References: <20130220051028.GA22784@ando> <20130220090426.66ebbe8e@anarchist.wooz.org> <20130221001244.1b602cfd@pitrou.net> Message-ID: On 2/20/2013 8:44 PM, Andre Roberge wrote: > From what I have read on edu-sig, I believe it would be a huge mistake > to remove IDLE from the Python standard library. The idea was proposed 2 1/2 years ago, partly because IDLE was then buggy and mostly not maintained. The idea was rejected in favor of improving IDLE. Since then, many bugs have been quashed and idle is getting active attention. Do you have any idea what education users think about backporting IDLE enhancements? Would they support the PEP? Or would they rather only get true bugfixes in bugfix releases, even for IDLE? -- Terry Jan Reedy From solipsis at pitrou.net Thu Feb 21 08:01:56 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 21 Feb 2013 08:01:56 +0100 Subject: [Python-ideas] Introducing PEP 434 -- IDLE Enhancement Exception for All Branches References: <20130220051028.GA22784@ando> <20130220090426.66ebbe8e@anarchist.wooz.org> <20130221001244.1b602cfd@pitrou.net> Message-ID: <20130221080156.0a3947af@pitrou.net> On Wed, 20 Feb 2013 20:41:29 -0500 Terry Reedy wrote: > > > What is IDLE's actual maintenance activity, exactly? > > I am not sure whether you are asking about volume or focus. A year ago, > focus was on bugs that caused IDLE to quit because of an uncaught > exception. When IDLE is started on Windows from an icon, this looks like > a crash because there is no visible traceback or exit message. Recently, > other issues have been worked on. In the last 5 months, activity has > picked up and there are about 40 issues in 3.4 misc/news whose commit > message contains 'idle'. Ah, fair enough. In this case IDLE should probably remain, at least as long as it's maintained anyway. Regards Antoine. From ubershmekel at gmail.com Thu Feb 21 08:20:54 2013 From: ubershmekel at gmail.com (Yuval Greenfield) Date: Thu, 21 Feb 2013 09:20:54 +0200 Subject: [Python-ideas] Fwd: argparse - add support for environment variables In-Reply-To: References: <71869e7d-605c-4599-8010-0c195e86e982@googlegroups.com> <5cca4189-c01d-44e6-9903-6be5f2bb58d7@googlegroups.com> Message-ID: On Thu, Feb 21, 2013 at 2:18 AM, Eli Bendersky wrote: > What about config files? Mix them into argparse too? Oh, and support for > .ini/XML/YAML/JSON/sqlite configs, please! > > Sarcasm aside (don't take it personally ;-), my point is this: argparse is > a nice, orthogonal library for dealing with command-line arguments. Let's > keep it this way. Python gives you tools to *very easily* mix up different > configuration paths, with really minimal effort. > Excellent point, Yuval Greenfield -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Thu Feb 21 10:21:45 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 21 Feb 2013 10:21:45 +0100 Subject: [Python-ideas] Introducing PEP 434 -- IDLE Enhancement Exception for All Branches References: <20130221001244.1b602cfd@pitrou.net> <51258A30.8090809@illinois.edu> Message-ID: <20130221102145.2b8edebd@pitrou.net> Hello Roger, Le Wed, 20 Feb 2013 20:45:04 -0600, serwy a ?crit : > > I am the developer of the IdleX project > (http://idlex.sourceforge.net). It's not a fork of IDLE, but just a > collection of extensions that can be easily merged into IDLE. > > I started the project because of the slow pace of IDLE development, > especially when having several outstanding patches languishing in the > bug tracker. With your help I'd like to push IDLE forward. I guess it's a good idea to try and merge improvements into IDLE, but IDLE developers such as Terry would be in a better position to answer. Regards Antoine. From jeanpierreda at gmail.com Thu Feb 21 12:01:19 2013 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Thu, 21 Feb 2013 06:01:19 -0500 Subject: [Python-ideas] Adding a safe alternative to pickle in the standard library Message-ID: I've been noticing a lot of security-related issues being discussed in the Python world since the Ruby YAML problemcame out. Is it time to consider adding an alternative to pickle that is safe(r) by default? Pickle is usable in situations few other things are, because it can handle cyclic references and virtually any python object. The only stdlib alternative I'm aware of is json, which can do neither of those things. (Or at least, not without significant extra serialization code.) I would imagine that any alternative supplied should be easy enough to use that pickle users would seriously consider switching, and include at least those features. The benefit of using a secure alternative to pickle is that it increases the difficulty of creating an insecure application, even for those that are aware of the risks of the pickle module. With the pickle module, you are one mistake away from an insecure program: all you need is to have a way for the attacker to influence input to pickle. With a secure alternative, even if you make that mistake, it doesn't immediately result in a compromised application. You would need another mistake on top of that that results in the deserialized input being used improperly. The only third party library I'm aware of that attempts to be a safe/usable pickle replacement is cerealizer[1]_. Would it be worth considering adding cerealizer, or something like it, to the stdlib? .. [1]: http://home.gna.org/oomadness/en/cerealizer/index.html -- Devin From solipsis at pitrou.net Thu Feb 21 12:11:35 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 21 Feb 2013 12:11:35 +0100 Subject: [Python-ideas] Adding a safe alternative to pickle in the standard library References: Message-ID: <20130221121135.5a1aabd6@pitrou.net> Le Thu, 21 Feb 2013 06:01:19 -0500, Devin Jeanpierre a ?crit : > I've been noticing a lot of security-related issues being discussed in > the Python world since the Ruby YAML problemcame out. Is it time to > consider adding an alternative to pickle that is safe(r) by default? There's already json. Is something else needed? Regards Antoine. From jeanpierreda at gmail.com Thu Feb 21 12:47:58 2013 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Thu, 21 Feb 2013 06:47:58 -0500 Subject: [Python-ideas] Adding a safe alternative to pickle in the standard library In-Reply-To: <20130221121135.5a1aabd6@pitrou.net> References: <20130221121135.5a1aabd6@pitrou.net> Message-ID: On Thu, Feb 21, 2013 at 6:11 AM, Antoine Pitrou wrote: >> I've been noticing a lot of security-related issues being discussed in >> the Python world since the Ruby YAML problemcame out. Is it time to >> consider adding an alternative to pickle that is safe(r) by default? > > There's already json. Is something else needed? json can't handle cyclic references, and can't handle arbitrary python types. Even if you pass in a custom default and object_pairs_hook to json.dump and json.load respectively, it is impossible to serialize a subclass of (e.g.) dict as anything except the way dict is serialized, which will generally be incorrect. Even if this is changed, creating custom hooks in default and object_pairs_hook is a lot of work compared to using pickle (or, indeed, cerealizer), which handles this automatically. In some circumstances using pickle is clearly the wrong choice (e.g. storing data in cookies), but at the same time it is easier to do the wrong thing than the right thing. -- Devin From jsbueno at python.org.br Thu Feb 21 13:38:23 2013 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Thu, 21 Feb 2013 09:38:23 -0300 Subject: [Python-ideas] Adding a safe alternative to pickle in the standard library In-Reply-To: References: <20130221121135.5a1aabd6@pitrou.net> Message-ID: On 21 February 2013 08:47, Devin Jeanpierre wrote: > On Thu, Feb 21, 2013 at 6:11 AM, Antoine Pitrou wrote: >>> I've been noticing a lot of security-related issues being discussed in >>> the Python world since the Ruby YAML problemcame out. Is it time to >>> consider adding an alternative to pickle that is safe(r) by default? >> >> There's already json. Is something else needed? > > json can't handle cyclic references, and can't handle arbitrary python > types. Even if you pass in a custom default and object_pairs_hook to > json.dump and json.load respectively, it is impossible to serialize a > subclass of (e.g.) dict as anything except the way dict is serialized, > which will generally be incorrect. > > Even if this is changed, creating custom hooks in default and > object_pairs_hook is a lot of work compared to using pickle (or, > indeed, cerealizer), which handles this automatically. > > In some circumstances using pickle is clearly the wrong choice (e.g. > storing data in cookies), but at the same time it is easier to do the > wrong thing than the right thing. Do you think a couple hleper functions to json could help? Funcitons that would translate a complex Python object into a dictionary, containing all type information, and object metadata - still yield a simple dictionary. (instance attributes of the root object would be under that dictionary's "__dict__" key, for example) Cyclic reference would require something more complex - but just these could allow one to json serialize arbitrary objects. Maybe these helper functions could be in the json module itself. > > -- Devin > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas From eric at trueblade.com Thu Feb 21 14:32:47 2013 From: eric at trueblade.com (Eric V. Smith) Date: Thu, 21 Feb 2013 08:32:47 -0500 Subject: [Python-ideas] Adding a safe alternative to pickle in the standard library In-Reply-To: <20130221121135.5a1aabd6@pitrou.net> References: <20130221121135.5a1aabd6@pitrou.net> Message-ID: <512621FF.6070606@trueblade.com> On 2/21/2013 6:11 AM, Antoine Pitrou wrote: > Le Thu, 21 Feb 2013 06:01:19 -0500, > Devin Jeanpierre > a ?crit : >> I've been noticing a lot of security-related issues being discussed in >> the Python world since the Ruby YAML problemcame out. Is it time to >> consider adding an alternative to pickle that is safe(r) by default? > > There's already json. Is something else needed? As stated elsewhere, it's cycles and especially arbitrary python objects that are the big draw for pickle. I've always wanted a version of pickle.loads() that takes a list of classes that are allowed to be instantiated. Often, when using pickle to serialize over say AMQP or some other transport, I know what classes I want to allow. Anything else is either a (not infrequent) logic error or an attack of some sort. I realize this isn't perfect, but it would certainly reduce the attack surface for many of my use cases. I'm already authenticating the sender, and when I'm really paranoid I also sign the pickles. Just a thought. -- Eric. From wolfgang.maier at biologie.uni-freiburg.de Thu Feb 21 14:35:17 2013 From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier) Date: Thu, 21 Feb 2013 13:35:17 +0000 (UTC) Subject: [Python-ideas] range Message-ID: Dear all, what do you think about allowing the use of operators on ranges? I thought of things like: a = range(1,10) b = range(5,12) intersect = a & b # equivalent to intersect = range(5,10) merge = a | b # equivalent to merge = range(1,12) to make this work in more complex cases, like: a = range(1,10) b = range(12,15) merge = a | b # equivalent to sth. like range(range(1,10),range(12,15)) maybe range nesting would be desirable, but the full sequence of values of such a nested range could still be calculated on demand. Such syntax would allow for the generation of more complex sequences, and, as a side-effect, ranges could also be checked for overlap easily: if a & b: do_something() this whole idea is reminiscent, of course, of what's implemented for sets already, so like there you could think of intersect = a & b as shorthand for intersect = a.intersection(b) Opinions? From mark.hackett at metoffice.gov.uk Thu Feb 21 14:39:02 2013 From: mark.hackett at metoffice.gov.uk (Mark Hackett) Date: Thu, 21 Feb 2013 13:39:02 +0000 Subject: [Python-ideas] Adding a safe alternative to pickle in the standard library In-Reply-To: <512621FF.6070606@trueblade.com> References: <20130221121135.5a1aabd6@pitrou.net> <512621FF.6070606@trueblade.com> Message-ID: <201302211339.02892.mark.hackett@metoffice.gov.uk> On Thursday 21 Feb 2013, Eric V. Smith wrote: > On 2/21/2013 6:11 AM, Antoine Pitrou wrote: > > Le Thu, 21 Feb 2013 06:01:19 -0500, > > Devin Jeanpierre > > > > a ?crit : > >> I've been noticing a lot of security-related issues being discussed in > >> the Python world since the Ruby YAML problemcame out. Is it time to > >> consider adding an alternative to pickle that is safe(r) by default? > > > > There's already json. Is something else needed? > > As stated elsewhere, it's cycles and especially arbitrary python objects > that are the big draw for pickle. > > I've always wanted a version of pickle.loads() that takes a list of > classes that are allowed to be instantiated. Often, when using pickle to > serialize over say AMQP or some other transport, I know what classes I > want to allow. Anything else is either a (not infrequent) logic error or > an attack of some sort. > > I realize this isn't perfect, but it would certainly reduce the attack > surface for many of my use cases. I'm already authenticating the sender, > and when I'm really paranoid I also sign the pickles. > > Just a thought. > Is this not better solved by other methods? I.e. wasteful, but effective would be to send it all by XML. From ubershmekel at gmail.com Thu Feb 21 14:50:18 2013 From: ubershmekel at gmail.com (Yuval Greenfield) Date: Thu, 21 Feb 2013 15:50:18 +0200 Subject: [Python-ideas] range In-Reply-To: References: Message-ID: On Thu, Feb 21, 2013 at 3:35 PM, Wolfgang Maier < wolfgang.maier at biologie.uni-freiburg.de> wrote: > Dear all, > what do you think about allowing the use of operators on ranges? > > I thought of things like: > a = range(1,10) > b = range(5,12) > intersect = a & b # equivalent to intersect = range(5,10) > merge = a | b # equivalent to merge = range(1,12) > > to make this work in more complex cases, like: > a = range(1,10) > b = range(12,15) > merge = a | b # equivalent to sth. like > range(range(1,10),range(12,15)) > > maybe range nesting would be desirable, but the full sequence of values of > such > a nested range could still be calculated on demand. > > Such syntax would allow for the generation of more complex sequences, and, > as a > side-effect, ranges could also be checked for overlap easily: > if a & b: > do_something() > > this whole idea is reminiscent, of course, of what's implemented for sets > already, so like there you could think of > intersect = a & b as shorthand for intersect = a.intersection(b) > > Opinions? > > > This has been suggested in the past. http://mail.python.org/pipermail/python-bugs-list/2012-June/173343.html I think each operation you implement can have a lot of different bikeshed details. Since these are all very simple to begin with, it's better to not burden the language and just let users implement whatever it is they need. Though, maybe you can show some popular and compelling use cases in the wild? Yuval Greenfield -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric at trueblade.com Thu Feb 21 14:53:13 2013 From: eric at trueblade.com (Eric V. Smith) Date: Thu, 21 Feb 2013 08:53:13 -0500 Subject: [Python-ideas] Adding a safe alternative to pickle in the standard library In-Reply-To: <201302211339.02892.mark.hackett@metoffice.gov.uk> References: <20130221121135.5a1aabd6@pitrou.net> <512621FF.6070606@trueblade.com> <201302211339.02892.mark.hackett@metoffice.gov.uk> Message-ID: <512626C9.6010105@trueblade.com> On 2/21/2013 8:39 AM, Mark Hackett wrote: > On Thursday 21 Feb 2013, Eric V. Smith wrote: >> On 2/21/2013 6:11 AM, Antoine Pitrou wrote: >>> Le Thu, 21 Feb 2013 06:01:19 -0500, >>> Devin Jeanpierre >>> >>> a ?crit : >>>> I've been noticing a lot of security-related issues being discussed in >>>> the Python world since the Ruby YAML problemcame out. Is it time to >>>> consider adding an alternative to pickle that is safe(r) by default? >>> >>> There's already json. Is something else needed? >> >> As stated elsewhere, it's cycles and especially arbitrary python objects >> that are the big draw for pickle. >> >> I've always wanted a version of pickle.loads() that takes a list of >> classes that are allowed to be instantiated. Often, when using pickle to >> serialize over say AMQP or some other transport, I know what classes I >> want to allow. Anything else is either a (not infrequent) logic error or >> an attack of some sort. >> >> I realize this isn't perfect, but it would certainly reduce the attack >> surface for many of my use cases. I'm already authenticating the sender, >> and when I'm really paranoid I also sign the pickles. >> >> Just a thought. >> > > Is this not better solved by other methods? I.e. wasteful, but effective would > be to send it all by XML. Sure. I could write a serializer (to XML or whatever) that handles graphs of arbitrary python objects, but then I'm duplicating most of what pickle does. I'd rather leverage all of the work that pickle represents. Maybe I'll write a patch just to see what's involved. -- Eric. From solipsis at pitrou.net Thu Feb 21 15:00:49 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 21 Feb 2013 15:00:49 +0100 Subject: [Python-ideas] Adding a safe alternative to pickle in the standard library References: <20130221121135.5a1aabd6@pitrou.net> <512621FF.6070606@trueblade.com> Message-ID: <20130221150049.2731682e@pitrou.net> Le Thu, 21 Feb 2013 08:32:47 -0500, "Eric V. Smith" a ?crit : > On 2/21/2013 6:11 AM, Antoine Pitrou wrote: > > Le Thu, 21 Feb 2013 06:01:19 -0500, > > Devin Jeanpierre > > a ?crit : > >> I've been noticing a lot of security-related issues being > >> discussed in the Python world since the Ruby YAML problemcame out. > >> Is it time to consider adding an alternative to pickle that is > >> safe(r) by default? > > > > There's already json. Is something else needed? > > As stated elsewhere, it's cycles and especially arbitrary python > objects that are the big draw for pickle. Of course, but it's being powerful which also makes pickle dangerous. > I've always wanted a version of pickle.loads() that takes a list of > classes that are allowed to be instantiated. Is the following enough for you: http://docs.python.org/3.4/library/pickle.html#restricting-globals ? Regards Antoine. From eric at trueblade.com Thu Feb 21 15:11:20 2013 From: eric at trueblade.com (Eric V. Smith) Date: Thu, 21 Feb 2013 09:11:20 -0500 Subject: [Python-ideas] Adding a safe alternative to pickle in the standard library In-Reply-To: <20130221150049.2731682e@pitrou.net> References: <20130221121135.5a1aabd6@pitrou.net> <512621FF.6070606@trueblade.com> <20130221150049.2731682e@pitrou.net> Message-ID: <51262B08.3030600@trueblade.com> On 2/21/2013 9:00 AM, Antoine Pitrou wrote: > Le Thu, 21 Feb 2013 08:32:47 -0500, > "Eric V. Smith" a ?crit : >> On 2/21/2013 6:11 AM, Antoine Pitrou wrote: >>> Le Thu, 21 Feb 2013 06:01:19 -0500, >>> Devin Jeanpierre >>> a ?crit : >>>> I've been noticing a lot of security-related issues being >>>> discussed in the Python world since the Ruby YAML problemcame out. >>>> Is it time to consider adding an alternative to pickle that is >>>> safe(r) by default? >>> >>> There's already json. Is something else needed? >> >> As stated elsewhere, it's cycles and especially arbitrary python >> objects that are the big draw for pickle. > > Of course, but it's being powerful which also makes pickle dangerous. > >> I've always wanted a version of pickle.loads() that takes a list of >> classes that are allowed to be instantiated. > > Is the following enough for you: > http://docs.python.org/3.4/library/pickle.html#restricting-globals > ? Indeed, it is. Thanks for pointing it out! I've never gotten past the module interface part of the docs. Maybe the warning at the top of the page could also mention that there are ways to mitigate the safety concerns, and point to #restricting-globals? -- Eric. From solipsis at pitrou.net Thu Feb 21 16:41:42 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 21 Feb 2013 16:41:42 +0100 Subject: [Python-ideas] Adding a safe alternative to pickle in the standard library References: <20130221121135.5a1aabd6@pitrou.net> <512621FF.6070606@trueblade.com> <20130221150049.2731682e@pitrou.net> <51262B08.3030600@trueblade.com> Message-ID: <20130221164142.09cbc5dd@pitrou.net> Le Thu, 21 Feb 2013 09:11:20 -0500, "Eric V. Smith" a ?crit : > On 2/21/2013 9:00 AM, Antoine Pitrou wrote: > > Le Thu, 21 Feb 2013 08:32:47 -0500, > > "Eric V. Smith" a ?crit : > >> On 2/21/2013 6:11 AM, Antoine Pitrou wrote: > >>> Le Thu, 21 Feb 2013 06:01:19 -0500, > >>> Devin Jeanpierre > >>> a ?crit : > >>>> I've been noticing a lot of security-related issues being > >>>> discussed in the Python world since the Ruby YAML problemcame > >>>> out. Is it time to consider adding an alternative to pickle that > >>>> is safe(r) by default? > >>> > >>> There's already json. Is something else needed? > >> > >> As stated elsewhere, it's cycles and especially arbitrary python > >> objects that are the big draw for pickle. > > > > Of course, but it's being powerful which also makes pickle > > dangerous. > > > >> I've always wanted a version of pickle.loads() that takes a list of > >> classes that are allowed to be instantiated. > > > > Is the following enough for you: > > http://docs.python.org/3.4/library/pickle.html#restricting-globals > > ? > > Indeed, it is. Thanks for pointing it out! I've never gotten past the > module interface part of the docs. Maybe the warning at the top of the > page could also mention that there are ways to mitigate the safety > concerns, and point to #restricting-globals? Yes, that would be a good idea :-) Regards Antoine. From bruce at leapyear.org Thu Feb 21 16:43:22 2013 From: bruce at leapyear.org (Bruce Leban) Date: Thu, 21 Feb 2013 07:43:22 -0800 Subject: [Python-ideas] range In-Reply-To: References: Message-ID: On Thu, Feb 21, 2013 at 5:50 AM, Yuval Greenfield wrote: > On Thu, Feb 21, 2013 at 3:35 PM, Wolfgang Maier < > wolfgang.maier at biologie.uni-freiburg.de> wrote: > >> a = range(1,10) >> b = range(5,12) >> intersect = a & b # equivalent to intersect = range(5,10) >> merge = a | b # equivalent to merge = range(1,12) >> >> ... > this whole idea is reminiscent, of course, of what's implemented for sets >> already, so like there you could think of >> intersect = a & b as shorthand for intersect = a.intersection(b) >> >> I think each operation you implement can have a lot of different bikeshed > details. Since these are all very simple to begin with, it's better to not > burden the language and just let users implement whatever it is they need. > > Ignoring performance issues, imagine instead a sorted set class that you could add ranges to (or remove them). Well, actually it's trivial to convert a range to a set; the interesting problem is performance. You could save the range parts unexpanded so you could add huge ranges and you could also support infinite ranges (itertools.count). I suggest this because once you consider doing the first set of operations, like r1 & r2, it's quite natural for someone to want some_non_unlucky_integers = range(100) - [13] The idea of a sorted set class is, of course, not original. See for example http://stutzbachenterprises.com/blist/sortedset.html. If SortedSet isn't in the stdlib, I can't imagine this enhancement being either. --- Bruce Latest blog post: Alice's Puzzle Page http://www.vroospeak.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From dustin at v.igoro.us Thu Feb 21 16:50:33 2013 From: dustin at v.igoro.us (Dustin J. Mitchell) Date: Thu, 21 Feb 2013 10:50:33 -0500 Subject: [Python-ideas] Adding a safe alternative to pickle in the standard library In-Reply-To: <51262B08.3030600@trueblade.com> References: <20130221121135.5a1aabd6@pitrou.net> <512621FF.6070606@trueblade.com> <20130221150049.2731682e@pitrou.net> <51262B08.3030600@trueblade.com> Message-ID: This conversation worries me. The security community has shown that safety isn't something you can add to a powerful tool. With great power comes great expressivity, and correspondingly more difficulty reasoning about it. Not to mention reasoning about yhe implementation. JSON is probably secure against code-execution exploits, but only probably. When you put something in the stdlib and call it "safe", even with caveats, people will make even more brazen mistakes than with a documented-unsafe tool like pickle. Dustin -------------- next part -------------- An HTML attachment was scrubbed... URL: From masklinn at masklinn.net Thu Feb 21 17:19:50 2013 From: masklinn at masklinn.net (Masklinn) Date: Thu, 21 Feb 2013 17:19:50 +0100 Subject: [Python-ideas] Adding a safe alternative to pickle in the standard library In-Reply-To: References: <20130221121135.5a1aabd6@pitrou.net> <512621FF.6070606@trueblade.com> <20130221150049.2731682e@pitrou.net> <51262B08.3030600@trueblade.com> Message-ID: <00C63493-1733-46CE-A37D-2D0477A00C54@masklinn.net> On 2013-02-21, at 16:50 , Dustin J. Mitchell wrote: > This conversation worries me. The security community has shown that safety isn't something you can add to a powerful tool. With great power comes great expressivity, and correspondingly more difficulty reasoning about it. Not to mention reasoning about yhe implementation. JSON is probably secure against code-execution exploits, but only probably. Considering there's no provision whatsoever in JSON itself for directing any kind of execution or programmatic-ish behavior (as opposed to YAML and ? from what I understand ? XML) why "only probably"? I could see JSON implementations having vulnerability and applications using JSON to do unsafe things (e.g. eval'ing JSON-sourced strings), but JSON itself? From ned at nedbatchelder.com Thu Feb 21 17:38:21 2013 From: ned at nedbatchelder.com (Ned Batchelder) Date: Thu, 21 Feb 2013 11:38:21 -0500 Subject: [Python-ideas] Adding a safe alternative to pickle in the standard library In-Reply-To: <00C63493-1733-46CE-A37D-2D0477A00C54@masklinn.net> References: <20130221121135.5a1aabd6@pitrou.net> <512621FF.6070606@trueblade.com> <20130221150049.2731682e@pitrou.net> <51262B08.3030600@trueblade.com> <00C63493-1733-46CE-A37D-2D0477A00C54@masklinn.net> Message-ID: <51264D7D.5010003@nedbatchelder.com> On 2/21/2013 11:19 AM, Masklinn wrote: > On 2013-02-21, at 16:50 , Dustin J. Mitchell wrote: >> This conversation worries me. The security community has shown that safety isn't something you can add to a powerful tool. With great power comes great expressivity, and correspondingly more difficulty reasoning about it. Not to mention reasoning about yhe implementation. JSON is probably secure against code-execution exploits, but only probably. > Considering there's no provision whatsoever in JSON itself for directing > any kind of execution or programmatic-ish behavior (as opposed to YAML > and ? from what I understand ? XML) why "only probably"? I was going to say, "YAML the format does not include execution," but then I went to read the YAML spec about the !! notation, and I honestly have no idea what it means. YAML is scary... --Ned. > I could see JSON implementations having vulnerability and applications > using JSON to do unsafe things (e.g. eval'ing JSON-sourced strings), but > JSON itself? > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > From jeanpierreda at gmail.com Thu Feb 21 18:18:50 2013 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Thu, 21 Feb 2013 12:18:50 -0500 Subject: [Python-ideas] Adding a safe alternative to pickle in the standard library In-Reply-To: References: <20130221121135.5a1aabd6@pitrou.net> <512621FF.6070606@trueblade.com> <20130221150049.2731682e@pitrou.net> <51262B08.3030600@trueblade.com> Message-ID: On Thu, Feb 21, 2013 at 10:50 AM, Dustin J. Mitchell wrote: > When you put something in the stdlib and call it "safe", even with caveats, > people will make even more brazen mistakes than with a documented-unsafe > tool like pickle. Then how do we improve on the status quo? The best situation can't possibly be one in which the standard serialization tool allows for code injection exploits out of the box, by default, and where there is no reasonable alternative in the stdlib without such problems. To my ears, this objection is like objecting to the inclusion of raw_input. Surely people will make even more brazen mistakes with a so-called "safe" input method like raw_input, than with a documented-unsafe tool like input()? -- Devin From abarnert at yahoo.com Thu Feb 21 18:20:28 2013 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 21 Feb 2013 09:20:28 -0800 Subject: [Python-ideas] range In-Reply-To: References: Message-ID: <1CD63256-1739-4998-B84A-97F82C198712@yahoo.com> On Feb 21, 2013, at 5:35, Wolfgang Maier wrote: > Dear all, > what do you think about allowing the use of operators on ranges? > > I thought of things like: > a = range(1,10) > b = range(5,12) > intersect = a & b # equivalent to intersect = range(5,10) > merge = a | b # equivalent to merge = range(1,12) > > to make this work in more complex cases, like: > a = range(1,10) > b = range(12,15) > merge = a | b # equivalent to sth. like range(range(1,10),range(12,15)) That means ranges are no longer constant size, no longer have constant- time contains-testing, etc.; it's linear in the number of sub ranges. (And so would intersect and union, for that matter.) That's more like a sparse set or an RLE-encoded set or something--which is a useful thing for numerical work (does pandas have an equivalent?), but it's not really a range anymore. Maybe build a new IntervalSet or something, which can interact with ranges (and itertools.count?) transparently (by treating them as a set of 1), put it on PyPI, and see how many users find themselves wishing that range(1, 10) | range(12, 15) would return an IntervalSet so they wouldn't have to do IntervalSet(range(1, 10)) | range(12, 15). That seems like the best way to find out whether this would be useful in practice. Also, I think I'd expect difference and symmetric difference if I wanted union, and if you're handling infinite ranges I'd expect complement as well. Plus interaction with single ints as well as ranges and other sets. But that may be excessive feature creep--since I don't have an obvious use for union, I may be imagining things wrong. > maybe range nesting would be desirable, but the full sequence of values of such > a nested range could still be calculated on demand. > > Such syntax would allow for the generation of more complex sequences, and, as a > side-effect, ranges could also be checked for overlap easily: > if a & b: > do_something() This side effect seems like it would be useful more often than the main effect. I've written a ranges_intersect function in at least two projects, but I've never written an intersect_ranges. (That is, I only need a boolean value.) > > this whole idea is reminiscent, of course, of what's implemented for sets > already, so like there you could think of > intersect = a & b as shorthand for intersect = a.intersection(b) > > Opinions? > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas From mark.hackett at metoffice.gov.uk Thu Feb 21 18:22:47 2013 From: mark.hackett at metoffice.gov.uk (Mark Hackett) Date: Thu, 21 Feb 2013 17:22:47 +0000 Subject: [Python-ideas] Adding a safe alternative to pickle in the standard library In-Reply-To: References: Message-ID: <201302211722.47380.mark.hackett@metoffice.gov.uk> On Thursday 21 Feb 2013, Devin Jeanpierre wrote: > On Thu, Feb 21, 2013 at 10:50 AM, Dustin J. Mitchell wrote: > > When you put something in the stdlib and call it "safe", even with > > caveats, people will make even more brazen mistakes than with a > > documented-unsafe tool like pickle. > > Then how do we improve on the status quo? The best situation can't > possibly be one in which the standard serialization tool allows for > code injection exploits out of the box, by default, and where there is > no reasonable alternative in the stdlib without such problems. By writing your application for its needs, not the needs of 10000 programs yet to be written and making the wrong assumption and putting it in a stdlib. If every problem could be solved with a stdlib call, there'd only have to be one programmer in the world... From solipsis at pitrou.net Thu Feb 21 18:24:52 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 21 Feb 2013 18:24:52 +0100 Subject: [Python-ideas] Adding a safe alternative to pickle in the standard library References: <201302211722.47380.mark.hackett@metoffice.gov.uk> Message-ID: <20130221182452.1ef90184@pitrou.net> Le Thu, 21 Feb 2013 17:22:47 +0000, Mark Hackett a ?crit : > On Thursday 21 Feb 2013, Devin Jeanpierre wrote: > > On Thu, Feb 21, 2013 at 10:50 AM, Dustin J. Mitchell > > > wrote: > > > When you put something in the stdlib and call it "safe", even with > > > caveats, people will make even more brazen mistakes than with a > > > documented-unsafe tool like pickle. > > > > Then how do we improve on the status quo? The best situation can't > > possibly be one in which the standard serialization tool allows for > > code injection exploits out of the box, by default, and where there > > is no reasonable alternative in the stdlib without such problems. > > By writing your application for its needs, not the needs of 10000 > programs yet to be written and making the wrong assumption and > putting it in a stdlib. > > If every problem could be solved with a stdlib call, there'd only > have to be one programmer in the world... You're forgetting the millions of stdlib programmers :-) Regards Antoine. From abarnert at yahoo.com Thu Feb 21 18:32:37 2013 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 21 Feb 2013 09:32:37 -0800 Subject: [Python-ideas] Adding a safe alternative to pickle in the standard library In-Reply-To: References: <20130221121135.5a1aabd6@pitrou.net> <512621FF.6070606@trueblade.com> <20130221150049.2731682e@pitrou.net> <51262B08.3030600@trueblade.com> Message-ID: <1E2DA813-A8AF-46CF-B7D4-5A1C1953B781@yahoo.com> How often have you needed either cyclic references or the ability to dynamically store arbitrary classes in something like a cookie or a cache file? It seems to me that when you're trying to do something that would be difficult to do with json (with the new helpers proposed by someone earlier in the thread), you're usually doing something too dangerous to store in a cookie anyway. If someone can construct instances of arbitrary classes, they can run code. If they can't, why did you need pickle? I realize the overlap is just small, not zero. But I think catering to it could be an attractive nuisance that would lead to more unsafe code rather than less. It's always easier to preserve safety while adding features to something simple, than to add safety while restricting features in something powerful. Sorry for the top post. Sent from a random iPhone On Feb 21, 2013, at 9:18, Devin Jeanpierre wrote: > On Thu, Feb 21, 2013 at 10:50 AM, Dustin J. Mitchell wrote: >> When you put something in the stdlib and call it "safe", even with caveats, >> people will make even more brazen mistakes than with a documented-unsafe >> tool like pickle. > > Then how do we improve on the status quo? The best situation can't > possibly be one in which the standard serialization tool allows for > code injection exploits out of the box, by default, and where there is > no reasonable alternative in the stdlib without such problems. > > To my ears, this objection is like objecting to the inclusion of > raw_input. Surely people will make even more brazen mistakes with a > so-called "safe" input method like raw_input, than with a > documented-unsafe tool like input()? > > -- Devin > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas From mark.hackett at metoffice.gov.uk Thu Feb 21 18:33:02 2013 From: mark.hackett at metoffice.gov.uk (Mark Hackett) Date: Thu, 21 Feb 2013 17:33:02 +0000 Subject: [Python-ideas] Adding a safe alternative to pickle in the standard library In-Reply-To: <20130221182452.1ef90184@pitrou.net> References: <201302211722.47380.mark.hackett@metoffice.gov.uk> <20130221182452.1ef90184@pitrou.net> Message-ID: <201302211733.02145.mark.hackett@metoffice.gov.uk> On Thursday 21 Feb 2013, Antoine Pitrou wrote: > Le Thu, 21 Feb 2013 17:22:47 +0000, > Mark Hackett a > > > > > If every problem could be solved with a stdlib call, there'd only > > have to be one programmer in the world... > > You're forgetting the millions of stdlib programmers :-) > > Regards > > Antoine. > But only because 10,000 different stdlib calls are needed! (I should shush now, I don't want to give away the Programmer's Secret. Even if it IS very similar to the Lawyers Secret)... Being serious, though, if your code requires a serious amount of security, you're better off writing your own parsing. It does mean the poor guy taking over has to use this "stdlib", but that's more work for programmers. Oh no, I said it...! AaaarrrgghhCARRIER LOST From abarnert at yahoo.com Thu Feb 21 18:35:14 2013 From: abarnert at yahoo.com (Andrew Barnert) Date: Thu, 21 Feb 2013 09:35:14 -0800 Subject: [Python-ideas] Adding a safe alternative to pickle in the standard library In-Reply-To: <20130221182452.1ef90184@pitrou.net> References: <201302211722.47380.mark.hackett@metoffice.gov.uk> <20130221182452.1ef90184@pitrou.net> Message-ID: On Feb 21, 2013, at 9:24, Antoine Pitrou wrote: > Le Thu, 21 Feb 2013 17:22:47 +0000, > Mark Hackett a > ?crit : > >> On Thursday 21 Feb 2013, Devin Jeanpierre wrote: >>> On Thu, Feb 21, 2013 at 10:50 AM, Dustin J. Mitchell >>> >> wrote: >>>> When you put something in the stdlib and call it "safe", even with >>>> caveats, people will make even more brazen mistakes than with a >>>> documented-unsafe tool like pickle. >>> >>> Then how do we improve on the status quo? The best situation can't >>> possibly be one in which the standard serialization tool allows for >>> code injection exploits out of the box, by default, and where there >>> is no reasonable alternative in the stdlib without such problems. >> >> By writing your application for its needs, not the needs of 10000 >> programs yet to be written and making the wrong assumption and >> putting it in a stdlib. >> >> If every problem could be solved with a stdlib call, there'd only >> have to be one programmer in the world... > > You're forgetting the millions of stdlib programmers :-) This is one of those "any sufficiently powerful language becomes lisp" things, isn't it. :) > Regards > > Antoine. > > > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas From Steve.Dower at microsoft.com Thu Feb 21 18:55:57 2013 From: Steve.Dower at microsoft.com (Steve Dower) Date: Thu, 21 Feb 2013 17:55:57 +0000 Subject: [Python-ideas] Adding a safe alternative to pickle in the standard library In-Reply-To: <1E2DA813-A8AF-46CF-B7D4-5A1C1953B781@yahoo.com> References: <20130221121135.5a1aabd6@pitrou.net> <512621FF.6070606@trueblade.com> <20130221150049.2731682e@pitrou.net> <51262B08.3030600@trueblade.com> <1E2DA813-A8AF-46CF-B7D4-5A1C1953B781@yahoo.com> Message-ID: <0f836a6d49664d2cb93c97bf496bab90@BLUPR03MB035.namprd03.prod.outlook.com> > From: Andrew Barnert > How often have you needed either cyclic references or the ability to > dynamically store arbitrary classes in something like a cookie or a cache file? In a past life I used pickle regularly to snapshot long-running (evolutionary) algorithms that used user-provided classes and all sorts of highly improper circular references. And there are plenty of researchers out there using Python for much crazier things than I ever did. There is a lot more to Python than web apps... Steve From stephen at xemacs.org Thu Feb 21 19:29:57 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 22 Feb 2013 03:29:57 +0900 Subject: [Python-ideas] Adding a safe alternative to pickle in the standard library In-Reply-To: <0f836a6d49664d2cb93c97bf496bab90@BLUPR03MB035.namprd03.prod.outlook.com> References: <20130221121135.5a1aabd6@pitrou.net> <512621FF.6070606@trueblade.com> <20130221150049.2731682e@pitrou.net> <51262B08.3030600@trueblade.com> <1E2DA813-A8AF-46CF-B7D4-5A1C1953B781@yahoo.com> <0f836a6d49664d2cb93c97bf496bab90@BLUPR03MB035.namprd03.prod.outlook.com> Message-ID: <87mwuxseoq.fsf@uwakimon.sk.tsukuba.ac.jp> Steve Dower writes: > In a past life I used pickle regularly to snapshot long-running > (evolutionary) algorithms that used user-provided classes And how do you propose to prevent user-provided exploits, then? Nobody wants to take away the power of pickle if it imposes only risks you're happy to bear. The question here is "is it possible to be *safer* than pickle without giving up any of the power?" From jeanpierreda at gmail.com Thu Feb 21 19:48:06 2013 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Thu, 21 Feb 2013 13:48:06 -0500 Subject: [Python-ideas] Adding a safe alternative to pickle in the standard library In-Reply-To: <87mwuxseoq.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20130221121135.5a1aabd6@pitrou.net> <512621FF.6070606@trueblade.com> <20130221150049.2731682e@pitrou.net> <51262B08.3030600@trueblade.com> <1E2DA813-A8AF-46CF-B7D4-5A1C1953B781@yahoo.com> <0f836a6d49664d2cb93c97bf496bab90@BLUPR03MB035.namprd03.prod.outlook.com> <87mwuxseoq.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Thu, Feb 21, 2013 at 1:29 PM, Stephen J. Turnbull wrote: > Steve Dower writes: > > > In a past life I used pickle regularly to snapshot long-running > > (evolutionary) algorithms that used user-provided classes > > And how do you propose to prevent user-provided exploits, then? Just because an application has one place where someone can inject new code, doesn't mean it should have another. You might trust the people that write these evolutionary algorithm classes, but not trust people that give you snapshots of the algorithms running. > Nobody wants to take away the power of pickle if it imposes only risks > you're happy to bear. The question here is "is it possible to be > *safer* than pickle without giving up any of the power?" I hope nobody is asking that question, because the answer is a strong no. Pickle's ability to call arbitrary objects accessible in any module, anywhere, is part of how powerful it is, but it is also a fundamental source of unsafety. That does not mean that we should not write or use safer alternatives. We have written and do use safer alternatives, like the json module. But it means we can't expect them to be usable exactly everywhere pickle is. I would've said the question is how far in that direction we should bother to go. How many features do you add before you're increasing risk from faulty code, rather than decreasing it by making it easier to use a secure-by-design library? -- Devin From Steve.Dower at microsoft.com Thu Feb 21 19:52:31 2013 From: Steve.Dower at microsoft.com (Steve Dower) Date: Thu, 21 Feb 2013 18:52:31 +0000 Subject: [Python-ideas] Adding a safe alternative to pickle in the standard library In-Reply-To: <87mwuxseoq.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20130221121135.5a1aabd6@pitrou.net> <512621FF.6070606@trueblade.com> <20130221150049.2731682e@pitrou.net> <51262B08.3030600@trueblade.com> <1E2DA813-A8AF-46CF-B7D4-5A1C1953B781@yahoo.com> <0f836a6d49664d2cb93c97bf496bab90@BLUPR03MB035.namprd03.prod.outlook.com> <87mwuxseoq.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <769b7cc7f941469d9a64c825cf9d5e1a@BLUPR03MB035.namprd03.prod.outlook.com> > From: Stephen J. Turnbull [mailto:stephen at xemacs.org] > Sent: Thursday, February 21, 2013 1030 > To: Steve Dower > Cc: Andrew Barnert; Devin Jeanpierre; Dustin J. Mitchell; Eric V. Smith; > Python-Ideas > Subject: Re: [Python-ideas] Adding a safe alternative to pickle in the standard > library > > Steve Dower writes: > > > In a past life I used pickle regularly to snapshot long-running > > (evolutionary) algorithms that used user-provided classes > > And how do you propose to prevent user-provided exploits, then? > > Nobody wants to take away the power of pickle if it imposes only risks you're > happy to bear. The question here is "is it possible to be > *safer* than pickle without giving up any of the power?" > I was only aiming to provide perspective, rather than a proposal. The existing ability to customise the unpickler suited my needs perfectly, not that I actually used it. To make it safer I would have a restricted unpickler as the default (for load/loads) and force people to override it if they want to loosen restrictions. To be really explicit, I would make load/loads only work with built-in types. For compatibility when reading earlier protocols we could add a type representing a class instance and its members that doesn't actually construct anything. (Maybe we override __getattr__ to make it lazily construct the instance when the program actually wants to use it?) For convenience, I'd add a parameter to Unpickler to let the user provide a set of types that are allowed to be constructed (or a mapping from names to callables as used in find_class()). Finally, I'd give greater exposure to overriding find_class(), as has already been suggested. People who want to unpickle any arbitrary type have to do this to opt-in. (Maybe that's a little aggressive?) I'd expect this would come under a new protocol version, but I wouldn't be opposed to old data being unpickled under these rules, especially if the non-constructing type is actually a lazy-constructing type. Cheers, Steve From foogod at gmail.com Thu Feb 21 20:18:55 2013 From: foogod at gmail.com (Alex Stewart) Date: Thu, 21 Feb 2013 11:18:55 -0800 Subject: [Python-ideas] PEP for enum library type? Message-ID: Hi all, Sorry to jump into this discussion so late, but I just finished reading through this thread and had a few thoughts.. (Sorry this message is a bit long. TL;DR: Please check the list of required/desired/undesired properties at the end and let me know if I've gotten anything seriously wrong with my interpretation of the discussion thus far) It seems to me that this particular thread started out as a call to step away from existing implementations and take a look at this issue from the direction of "what do we want/need" instead, but then it quickly got sidetracked back into discussing all the details of various existing/proposed implementations. I'd like to try to take a step back (again) for a minute and raise the question: What do we actually want to get out of this whole endeavor? First of all, as I see it, there are two main (and fairly distinct) use cases for enums in Python: 1. Predefined "unique values" for passing around within Python code 2. API-defined constants for interacting with non-Python libraries, etc (i.e. C defines/enums that need to be represented in Python, or database field values, etc) In non-Python code, typically enums have always been represented under the covers as ints, and therefore must be passed back and forth as numbers. The fact that they have an integer value, however, is purely an implementation artifact. It comes from the fact that C and some other languages don't have a rich enough type system to properly make enums their own distinct types, but Python does not have this limitation, and I think we should be careful not to constrain the way we do things within Python just because of the limitations of other languages. Where possible I believe we should conceptually be thinking of enums not as "sequences of ints" but more as "collections of singletons". That is, they are simply objects, with a defined name and type, which compare equal to themselves but not to others, and are generally related to others by some sort of grouping mechanism (and the same name always maps to the same object). In this context, the idea of assigning a "value" to an enum makes little sense and is arguably completely unnecessary. (and if we eliminate this aspect, it mitigates many of the issues that have been brought up about evaluation order and accidental duplication, in addition to potentially making the base design a lot simpler) Obviously, the second use case does require an association between enums and (typically int) values, but that could be viewed as simply a special case of the larger definition of "enums", rather than the other way around. I do think one thing worth noting, however, is that (at least in my experience) the cases which require associating names with values pretty much also always require that every name has a specific value, so the value for each and every enum within the group should probably be being defined explicitly anyway (I have encountered very few cases where it's actually useful to mix-and-match "I care about this value but I don't care about those ones"). It doesn't seem unreasonable, therefore, to define two different categories of enums: one that has no concept of "value" (for pure-Python), and one which does have associated values but all values have to be specified explicitly (for the "mapping constants" case). On a related note, to be honest, I'm not really sure I can think of any realistic use cases for "string enums" (or really anything other than ints in general). Does anybody have an example of where this would actually be useful as opposed to just using "pure" (valueless) enums (which would already have string names)? Anyway, in the interest of trying to get the discussion back onto more theoretical ground, I also wanted to try to summarize the more general thoughts/impressions I've gleaned from the discussions up to this point. From what I can tell, these are some of the properties that there seems to be some general consensus enums probably should or shouldn't have: Required properties (without these, any implementation is not generally useful, or is at least something different than an "enum"): 1. Enums must be groupable in some way (i.e. "Colors", or "Error values") 2. Enums within the same group must not compare equal to each other (unless two names are intentionally defined to map to the same enum (i.e. "aliases")) 3. (Within reason and the limitations of Python) Once defined, an enum's properties (i.e. its name, identity, group membership, relationships to other objects, etc) must be treated as immutable (i.e. not change out from under the programmer unexpectedly). Conceptually they should be considered to be "constants". Desirable properties (which ones are more or less desirable will vary for different people, but from what I've heard I think everybody sorta agrees that all of these could be good things as long as they don't cause other problems): 1. Enums should represent themselves (str/repr) by symbolic names, not as ints, etc. 2. Enums from different groups should preferably not compare equal to each other (even if they have the same associated int value). 3. It should be possible to determine what group an enum belongs to. 4. Enums/groups should be definable inline using a fairly simple Python syntax. 5. It should also be relatively easy to define enums/groups programmatically. 6. By default, enums should be referenceable as relatively simple identifiers (i.e. no need for quoting, function-calls, etc, just variables/attributes/etc) 7. If the programmer doesn't care about the value of an enum, she shouldn't have to explicitly state a meaningless value. 8. (If an enum does have an associated value) it should be easy to compare with and/or convert back and forth between enums and values (so that they can be used with existing APIs). 9. It would be nice to be able to associate docstrings, and possibly other metadata with enums. Undesirable properties: 1. Enum syntax should not be "too magic". (In particular, it's pretty clear at this point that creating new enums as a side-effect of name lookups (even as convenient as it does make the syntax) is ultimately not gonna fly) 2. The syntax for defining enums should not be so onerous or verbose that it's significantly harder to use than the existing idioms people are already using. 3. The syntax for defining enums should not be so alien that it will completely baffle programmers who are already used to typical Python constructs. 4. It shouldn't be necessary to quote enum names when defining them (since they won't be quoted when they're used) I want to check: Is this a valid summary of things? Anything I missed, or do people have substantial objections to any of the required/desirable/undesirable points I mentioned? Obviously, it may not be possible to achieve all of the desirable properties at the same time, but I think it's useful to start with an idea of what we'd ideally like, and then we can sit down and see how close we can actually get to it.. (Actually, on pondering these properties, I've started to put together a slightly different enum implementation which I think has some potential (it's somewhat a cross between Barry's and Tim's with a couple of ideas of my own). I think I'll flesh it out a little more and then put it up for comment as a separate thread, if people don't mind..) --Alex -------------- next part -------------- An HTML attachment was scrubbed... URL: From miki.tebeka at gmail.com Thu Feb 21 20:34:46 2013 From: miki.tebeka at gmail.com (Miki Tebeka) Date: Thu, 21 Feb 2013 11:34:46 -0800 (PST) Subject: [Python-ideas] Fwd: argparse - add support for environment variables In-Reply-To: References: <71869e7d-605c-4599-8010-0c195e86e982@googlegroups.com> <5cca4189-c01d-44e6-9903-6be5f2bb58d7@googlegroups.com> Message-ID: > What about config files? Mix them into argparse too? Oh, and support for > .ini/XML/YAML/JSON/sqlite configs, please! > Well, baby steps toward configuration solution :) The other thing is that unlike config files which have many formats, environment has one uniform access to it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsbueno at python.org.br Thu Feb 21 20:54:49 2013 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Thu, 21 Feb 2013 16:54:49 -0300 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: References: Message-ID: On 21 February 2013 16:18, Alex Stewart wrote: ... On a related note, to be honest, I'm not really sure I can think of any realistic use cases for "string enums" (or really anything other than ints in general). Does anybody have an example of where this would actually be useful as opposed to just using "pure" (valueless) enums (which would already have string names)? Backwards compatibility. Of course, although you have restricted your wording to "enums" we need constants. And it would be nice to replace magic string contants in the Python stdlib itself, by real -"unquoted" constants - and if the new constants are equivalent to the strings existing in current code, there is no breaking. Think of """ my_text.encode("utf--8", errors=str.IGNORE) """, instead of """ my_text.encode("utf--8", errors="ignore") """, for example. ... > Anyway, in the interest of trying to get the discussion back onto more > theoretical ground, I also wanted to try to summarize the more general > thoughts/impressions I've gleaned from the discussions up to this point. > From what I can tell, these are some of the properties that there seems to > be some general consensus enums probably should or shouldn't have: > > Required properties (without these, any implementation is not generally > useful, or is at least something different than an "enum"): > >1. Enums must be groupable in some way (i.e. "Colors", or "Error values") > 2. Enums within the same group must not compare equal to each other (unless two > names are intentionally defined to map to the same enum (i.e. "aliases")) >3. (Within reason and the limitations of Python) Once defined, an enum's > properties (i.e. its name, identity, group membership, relationships to > other objects, etc) must be treated as immutable (i.e. not change out from > under the programmer unexpectedly). Conceptually they should be considered > to be "constants". 4. They should subclass INTs in way it is possible to interact with low level libraries already using C Enums (only aplicable to those that need an int value) - their use should be as transparent as True and False are today. > > Desirable properties (which ones are more or less desirable will vary for > different people, but from what I've heard I think everybody sorta agrees > that all of these could be good things as long as they don't cause other > problems): > >1. Enums should represent themselves (str/repr) by symbolic names, not as ints, > etc. I would say that, while not "required", as low level languages lack this, the whole idea is simply not worth implementing without this. This is the real motivation - and NB. also for "Constants", not only "Enums". >2. Enums from different groups should preferably not compare equal to each > other (even if they have the same associated int value). I am not shure about this requirement. I agree about "valueless" Enums, but as far as they have values associated, they should behave just like their values. I did not hear of anyone injured by " True == 1" in Python. >3. It should be possible to determine what group an enum belongs to. > Enums/groups should be definable inline using a fairly simple Python syntax. >4. It should also be relatively easy to define enums/groups programmatically. >5. By default, enums should be referenceable as relatively simple identifiers > (i.e. no need for quoting, function-calls, etc, just > variables/attributes/etc) >6. If the programmer doesn't care about the value of an enum, she shouldn't > have to explicitly state a meaningless value. >7. (If an enum does have an associated value) it should be easy to compare with > and/or convert back and forth between enums and values (so that they can be > used with existing APIs). >8. It would be nice to be able to associate docstrings, and possibly other > metadata with enums. > > Undesirable properties: > >1. Enum syntax should not be "too magic". (In particular, it's pretty clear at > this point that creating new enums as a side-effect of name lookups (even as > convenient as it does make the syntax) is ultimately not gonna fly) > 2.The syntax for defining enums should not be so onerous or verbose that it's > significantly harder to use than the existing idioms people are already > using. >3. The syntax for defining enums should not be so alien that it will completely > baffle programmers who are already used to typical Python constructs. >4. It shouldn't be necessary to quote enum names when defining them (since they > won't be quoted when they're used) > > I want to check: Is this a valid summary of things? Anything I missed, or > do people have substantial objections to any of the > required/desirable/undesirable points I mentioned? > > Obviously, it may not be possible to achieve all of the desirable properties > at the same time, but I think it's useful to start with an idea of what we'd > ideally like, and then we can sit down and see how close we can actually get > to it.. > > (Actually, on pondering these properties, I've started to put together a > slightly different enum implementation which I think has some potential > (it's somewhat a cross between Barry's and Tim's with a couple of ideas of > my own). I think I'll flesh it out a little more and then put it up for > comment as a separate thread, if people don't mind..) Ok - I added there what I think are the most important. What lead me to ressurect this subject was a period when I was dealing with a couple of Image packages related packages - PIL, GIMP-Python and Pygame -- all of them make intense use of Constants and Enums...all of which print as "int"s. All the three have the common trait that they can't rely on other 3rd party Python packages besides the stdlib - since there is a relatively large codebase depending only on them and Python. js -><- > --Alex From vinay_sajip at yahoo.co.uk Thu Feb 21 21:16:46 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 21 Feb 2013 20:16:46 +0000 (UTC) Subject: [Python-ideas] =?utf-8?q?Adding_a_safe_alternative_to_pickle_in_t?= =?utf-8?q?he_standard=09library?= References: Message-ID: Devin Jeanpierre writes: > Pickle is usable in situations few other things are, because it can > handle cyclic references and virtually any python object. The only > stdlib alternative I'm aware of is json, which can do neither of those > things. (Or at least, not without significant extra serialization > code.) I would imagine that any alternative supplied should be easy > enough to use that pickle users would seriously consider switching, > and include at least those features. What I'm about to mention may or may not meet your use case, but there is a way in which JSON can support cyclic references and virtually any Python object. Not without *some* work and *some* limitations, obviously, but then, TANSTAAFL :-) In logging.dictConfig, we use a dict to configure logging, but the underlying mechanism is more general. It allows you to construct objects outside the dict using a syntax such as 'ext://sys.stderr', and objects inside the dict using syntax such as 'cfg://xyz.abc[1]'. To illustrate, consider the following JSON: { "v1": "ext://sys.stderr", "v2": "cfg://v3.some_list[1]", "v3": { "some_list": [1, 2, 3], "some_value": "cfg://v4.key" }, "v4": { "key": "value", "some_value": "cfg://v3.some_value" } } You can see that sys.stderr is the external object, and that there are various cyclic references. Now, using Python 2.7 (where dictConfig was introduced), you can do this: >>> import json >>> with open("example.json") as f: ... d = json.load(f) ... >>> d {u'v1': u'ext://sys.stderr', u'v2': u'cfg://v3.some_list[1]', u'v3': {u'some_lis t': [1, 2, 3], u'some_value': u'cfg://v4.key'}, u'v4': {u'some_value': u'cfg://v 3.some_value', u'key': u'value'}} >>> import logging.config >>> bc = logging.config.BaseConfigurator(d) >>> bc.config['v1'] ', mode 'w' at 0x004750D0> >>> bc.config['v2'] 2 >>> bc.config['v3'] {u'some_list': [1, 2, 3], u'some_value': u'cfg://v4.key'} >>> bc.config['v3']['some_value'] u'value' >>> bc.config['v4']['some_value'] u'value' The values are lazily evaluated (as you can see from the above example for bc.config['v3'], but that isn't necessarily a problem. Of course, one can build on this using keys to define classes, kwargs, etc., which is how logging's DictConfigurator (derived from BaseConfigurator) works. The handlers for ext://, cfg:// are customisable, and additional schemes can be added; it allows for fairly flexible configuration, but within bounds you can set. I probably should write a post about the BaseConfigurator, it is useful outside the context of logging and I have used it for other things, but it's hidden away under the covers a bit. Regards, Vinay Sajip From foogod at gmail.com Thu Feb 21 21:34:25 2013 From: foogod at gmail.com (Alex Stewart) Date: Thu, 21 Feb 2013 12:34:25 -0800 (PST) Subject: [Python-ideas] PEP for enum library type? In-Reply-To: References: Message-ID: <35679e42-3cd1-4fa4-b5ef-99e3fad35b79@googlegroups.com> On Thursday, February 21, 2013 11:54:49 AM UTC-8, Joao S. O. Bueno wrote: > 4. They should subclass INTs in way it is possible to interact with low > level > libraries already using C Enums (only aplicable to those that need an > int value) - > their use should be as transparent as True and False are today. > Ok, I can see that would be a desirable trait (and personally I'm with you on that one), though I'm not sure it really qualifies as "required" (there are potential uses for int-enums which would still be useful/valid even if it couldn't do that automatically), but I'll definitely add that to my "desirable" list.. > >1. Enums should represent themselves (str/repr) by symbolic names, not as > ints, > > etc. > I would say that, while not "required", as low level languages lack this, > the whole idea is simply not worth implementing without this. This is the > real motivation - and NB. also for "Constants", not only "Enums". Frankly, I think this one is so easy it's really not worth fussing about that much. It's pretty much a given that any solution we come up with will have this functionality anyway.. >2. Enums from different groups should preferably not compare equal to each > > other (even if they have the same associated int value). > I am not shure about this requirement. I agree about "valueless" Enums, > but > as far as they have values associated, they should behave just like > their values. I did not hear of anyone injured by " True == 1" in Python. Well, it's explicitly not a requirement (that's why it's under "desirable" instead of "required"). Here's the question, though: Do you believe if a solution was proposed which had this property it would be a bad thing? Note that this isn't really exactly the same as the "True == 1" case. In my opinion the argument in favor of this is along the lines of: class FooOptions (Enum): YES = 1 MAYBE = 2 NO = 3 class BarOptions (Enum): NO = 1 MAYBE = 2 YES = 3 def foofunc(choice): if choice == FooOptions.YES: do_something() Now what if somebody calls foofunc(BarOptions.NO)? Admittedly, this is a somewhat exaggerated case, but I still think it would be better if in these sorts of cases FooOptions.YES != BarOptions.NO (if one explicitly did want to test if the values were the same (regardless of enum type) they could do something like "if int(choice) == FooOptions.YES" instead) --Alex -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Fri Feb 22 00:24:50 2013 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 22 Feb 2013 12:24:50 +1300 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: References: Message-ID: <5126ACC2.3080904@canterbury.ac.nz> On 22/02/13 08:18, Alex Stewart wrote: > In non-Python code, typically enums have always been represented under the > covers as ints, and therefore must be passed back and forth as numbers. The > fact that they have an integer value, however, is purely an implementation > artifact. It comes from the fact that C and some other languages don't have > a rich enough type system to properly make enums their own distinct type I don't think that's true for all languages. For example, enums in Pascal are definitely distinct types from integers, yet the language explicitly assigns them ordinal values and defines an ordering for them. Wirth must have thought there was a benefit in doing that. > It doesn't seem > unreasonable, therefore, to define two different categories of enums: one that > has no concept of "value" (for pure-Python), and one which does have > associated values but all values have to be specified explicitly That sounds reasonable. However, I'm wondering if there isn't a third case: where you don't care about the values, but you do want them to have a defined ordering? > 1. Enum syntax should not be "too magic". (In particular, it's pretty clear > at this point that creating new enums as a side-effect of name lookups > (even as convenient as it does make the syntax) is ultimately not gonna fly) > 4. It shouldn't be necessary to quote enum names when defining them (since > they won't be quoted when they're used) Satisfying both of these constraints without new syntax seems to be pretty much impossible. That seems to be the main sticking point in these discussions. > I want to check: Is this a valid summary of things? It looks pretty comprehensive to me! -- Greg From bruce at leapyear.org Fri Feb 22 00:39:19 2013 From: bruce at leapyear.org (Bruce Leban) Date: Thu, 21 Feb 2013 15:39:19 -0800 Subject: [Python-ideas] Adding a safe alternative to pickle in the standard library In-Reply-To: References: <20130221121135.5a1aabd6@pitrou.net> <512621FF.6070606@trueblade.com> <20130221150049.2731682e@pitrou.net> <51262B08.3030600@trueblade.com> Message-ID: On Thu, Feb 21, 2013 at 3:01 AM, Devin Jeanpierre wrote: > I've been noticing a lot of security-related issues being discussed in > the Python world since the Ruby YAML problemcame out. Is it time to > consider adding an alternative to pickle that is safe(r) by default? > > Pickle is usable in situations few other things are, because it can > handle cyclic references and virtually any python object. The only > stdlib alternative I'm aware of is json, which can do neither of those > things. (Or at least, not without significant extra serialization > code.) I would imagine that any alternative supplied should be easy > enough to use that pickle users would seriously consider switching, > and include at least those features. > Pickle is unsafe if you give it untrusted input. It's safe if you pickle something yourself and then unpickle it. If the problem is that you want to pickle something and store it in some unsafe place (like a cookie or a db under user control) and then read it back in later and unpickle it, then you can mitigate the risk by using an HMAC or some other mechanism to prevent tampering and may want to consider encrypting it too. That said, there is one risk in pickling something yourself and unpickling it later that you need to watch out for. If your objects change, then unpickling might produce unexpected and even potentially unsafe results. You can mitigate this by adding object versions to your objects (as long as you don't forget to update that when the object changes). There's another problem - pickling is not guaranteed to work across Python versions. So you may find yourself having to read pickles that are no longer readable in a future python version. Not a problem for cookies, but a potential headache with long-lived pickles. All of this leads me to suggest using a better format for this problem. Json is a reasonable choice (I've used it myself) although I would still use an HMAC. If you encrypt it then that makes attacking the object that much harder. I'd advise against using your own format. I wrote a tutorial on hacking web sites called Gruyere . I suggest reading the section on cookies http://j.mp/learn-state-manipulation (although to be honest, I recommend reading the whole thing :-) Aside from security, using a format like json encourages you to think about what belongs in the persisted object and what doesn't. Suppose your object includes a url. If you pickle it, you may end up persisting the parsed url with a dictionary of parameters and other unnecessary overhead. When you convert to json, you're going to just copy the url. On Thu, Feb 21, 2013 at 7:50 AM, Dustin J. Mitchell wrote: > This conversation worries me. The security community has shown that > safety isn't something you can add to a powerful tool. With great power > comes great expressivity, and correspondingly more difficulty reasoning > about it. Not to mention reasoning about yhe implementation. JSON is > probably secure against code-execution exploits, but only probably. > > When you put something in the stdlib and call it "safe", even with > caveats, people will make even more brazen mistakes than with a > documented-unsafe tool like pickle. > Yes indeed. --- Bruce Latest blog post: Alice's Puzzle Page http://www.vroospeak.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Fri Feb 22 01:39:06 2013 From: python at mrabarnett.plus.com (MRAB) Date: Fri, 22 Feb 2013 00:39:06 +0000 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: <5126ACC2.3080904@canterbury.ac.nz> References: <5126ACC2.3080904@canterbury.ac.nz> Message-ID: <5126BE2A.70804@mrabarnett.plus.com> On 2013-02-21 23:24, Greg Ewing wrote: > On 22/02/13 08:18, Alex Stewart wrote: >> In non-Python code, typically enums have always been represented under the >> covers as ints, and therefore must be passed back and forth as numbers. The >> fact that they have an integer value, however, is purely an implementation >> artifact. It comes from the fact that C and some other languages don't have > > a rich enough type system to properly make enums their own distinct type > > I don't think that's true for all languages. For example, enums in > Pascal are definitely distinct types from integers, yet the language > explicitly assigns them ordinal values and defines an ordering for > them. Wirth must have thought there was a benefit in doing that. > [snip] It means that sets of enums can be implemented using bitsets. From rob.cliffe at btinternet.com Fri Feb 22 02:58:47 2013 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Fri, 22 Feb 2013 01:58:47 +0000 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: <5126ACC2.3080904@canterbury.ac.nz> References: <5126ACC2.3080904@canterbury.ac.nz> Message-ID: <5126D0D7.7040302@btinternet.com> On 21/02/2013 23:24, Greg Ewing wrote: > On 22/02/13 08:18, Alex Stewart wrote: >> In non-Python code, typically enums have always been represented >> under the >> covers as ints, and therefore must be passed back and forth as >> numbers. The >> fact that they have an integer value, however, is purely an >> implementation >> artifact. It comes from the fact that C and some other languages >> don't have > > a rich enough type system to properly make enums their own distinct > type > > I don't think that's true for all languages. For example, enums in > Pascal are definitely distinct types from integers, yet the language > explicitly assigns them ordinal values and defines an ordering for > them. Wirth must have thought there was a benefit in doing that. > Not necessarily. With respect to the great man, he may not have given the matter as much thought, or considered (or been interested in) as many use cases as the contributors to this thread have. >> It doesn't seem >> unreasonable, therefore, to define two different categories of enums: >> one that >> has no concept of "value" (for pure-Python), and one which does have >> associated values but all values have to be specified explicitly > > That sounds reasonable. However, I'm wondering if there isn't a > third case: where you don't care about the values, but you do > want them to have a defined ordering? The same thought occurred to me, although I could not think of a use case on the spot. But I think that there surely are such. > >> 1. Enum syntax should not be "too magic". (In particular, it's >> pretty clear >> at this point that creating new enums as a side-effect of name >> lookups >> (even as convenient as it does make the syntax) is ultimately >> not gonna fly) > >> 4. It shouldn't be necessary to quote enum names when defining >> them (since >> they won't be quoted when they're used) > > Satisfying both of these constraints without new syntax seems > to be pretty much impossible. That seems to be the main > sticking point in these discussions. > >> I want to check: Is this a valid summary of things? > > It looks pretty comprehensive to me! > Apart from the point about maybe wanting to order enums, I agree. I think Alex is right when he says you want to create singleton enums that don't equate to anything else. We don't think of RED, GREEN, BLUE as being identical with the ints 1,2,3, and so don't want our implementation of them to compare equal to those ints. Rob Cliffe From tjreedy at udel.edu Fri Feb 22 03:05:25 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 21 Feb 2013 21:05:25 -0500 Subject: [Python-ideas] Introducing PEP 434 -- IDLE Enhancement Exception for All Branches In-Reply-To: <20130221102145.2b8edebd@pitrou.net> References: <20130221001244.1b602cfd@pitrou.net> <51258A30.8090809@illinois.edu> <20130221102145.2b8edebd@pitrou.net> Message-ID: On 2/21/2013 4:21 AM, Antoine Pitrou wrote: > Le Wed, 20 Feb 2013 20:45:04 -0600, > serwy a ?crit : >> >> I am the developer of the IdleX project >> (http://idlex.sourceforge.net). It's not a fork of IDLE, but just a >> collection of extensions that can be easily merged into IDLE. >> >> I started the project because of the slow pace of IDLE development, >> especially when having several outstanding patches languishing in the >> bug tracker. With your help I'd like to push IDLE forward. > I guess it's a good idea to try and merge improvements into IDLE, but > IDLE developers such as Terry would be in a better position to answer. The pace of reviewing and applying patches to the default branch is a functions of people's time. I admit that I have been slower than I would like to be. This PEP is strictly about the question of which patches applied to tip should also be applied to older branches. A clearer answer, whatever it is, will help me, at least, to get more done. If some enhancements are backported to bug-fix releases, they may appear in the wild sooner. On the other hand, not backporting might mean that more total patches get applied to default before the next release. On the third hand, it can be easier to get a patch widely tested on existing releases than on newly built default. (This is true on Windows where _tkinter is not compiled without considerable extra work.) -- Terry Jan Reedy From ethan at stoneleaf.us Fri Feb 22 02:59:00 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 21 Feb 2013 17:59:00 -0800 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: References: Message-ID: <5126D0E4.3090402@stoneleaf.us> On 02/21/2013 11:18 AM, Alex Stewart wrote: > On a related note, to be honest, I'm not really sure I can think of any realistic use cases for "string enums" (or really anything other than ints in general). Does anybody have an example of where this would actually be useful as opposed to just using "pure" (valueless) enums (which would already have string names)? The tk library is a good example of where string enums would be useful; they also provide an easy "valueless" entity (at least as far as numbers go). > I want to check: Is this a valid summary of things? Anything I missed, or do people have substantial objections to any of the required/desirable/undesirable points I mentioned? Looks good. -- ~Ethan~ From ethan at stoneleaf.us Fri Feb 22 03:15:55 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 21 Feb 2013 18:15:55 -0800 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: <5126D0D7.7040302@btinternet.com> References: <5126ACC2.3080904@canterbury.ac.nz> <5126D0D7.7040302@btinternet.com> Message-ID: <5126D4DB.5020007@stoneleaf.us> On 02/21/2013 05:58 PM, Rob Cliffe wrote: > > On 21/02/2013 23:24, Greg Ewing wrote: >> On 22/02/13 08:18, Alex Stewart wrote: >>> It doesn't seem >>> unreasonable, therefore, to define two different categories of enums: one that >>> has no concept of "value" (for pure-Python), and one which does have >>> associated values but all values have to be specified explicitly >> >> That sounds reasonable. However, I'm wondering if there isn't a >> third case: where you don't care about the values, but you do >> want them to have a defined ordering? > > The same thought occurred to me, although I could not think of a use case on the spot. But I think that there surely > are such. If you want order, but don't care about values, let them be ints and assign names in the order you want. >>> 1. Enum syntax should not be "too magic". (In particular, it's pretty clear >>> at this point that creating new enums as a side-effect of name lookups >>> (even as convenient as it does make the syntax) is ultimately not gonna fly) >> >>> 4. It shouldn't be necessary to quote enum names when defining them (since >>> they won't be quoted when they're used) >> >> Satisfying both of these constraints without new syntax seems >> to be pretty much impossible. That seems to be the main >> sticking point in these discussions. >> >>> I want to check: Is this a valid summary of things? >> >> It looks pretty comprehensive to me! >> > Apart from the point about maybe wanting to order enums, I agree. I think Alex is right when he says you want to create > singleton enums that don't equate to anything else. We don't think of RED, GREEN, BLUE as being identical with the ints > 1,2,3, and so don't want our implementation of them to compare equal to those ints. That depends entirely an what you are interfacing with: if it's a graphical library then you surely will want to compare equal with an int; if something else where the int value is meaningless, base the enum on a string. ;) -- ~Ethan~ From robert2682 at verizon.net Fri Feb 22 03:28:02 2013 From: robert2682 at verizon.net (robert2682) Date: Thu, 21 Feb 2013 21:28:02 -0500 Subject: [Python-ideas] Functions and Unbound methods Message-ID: <5126D7B2.3050903@verizon.net> Hi, > Hi, > > I'm new; greetings all! > > I'm not sure if this is a bug or feature, but it confused me so I thought > I'd raise the issue. > > > class a: > def b (self): > pass > > foo = ('Hello', b) > > class c(a): > def d(self): > t = type (self. __class__. foo [1]) > print t > t = type (self. __class__. b) > print t > > e = c () > e. d() > > prints for the first print, > and it seems to me it should be an instancemethod > > [snip] > I think what's happening is that it's defining 'b' as a function in the > class's namespace, storing a reference to that function in the tuple, > and then, when the class definition ends, it's wrapping the function as > a method. > > You'll find: > > >>> a.foo[1] > > >>> a.b > > >>> a.b.im_func is a.foo[1] > True > I'm afraid this won't do what I want. As illustrated below, I want to store the reference from withing the same class, so as to be able to access in from a superclass. (sorry if subclass and superclass is not the correct terminology for Python) > I'm trying to something like this > > class EditPage: > > additonal_buttons = () > def __init__ (self): > buts = [] > for x in addional_butons: > if isinstance (x [1], types. UnboundMethodType): # fails > because type (x [1]) is function, not UnboundMethod > buts. append ((x [0], types. MethodType (x [1], self))) > else: > buts. append (x) > > class TreePage(EditPage): > def EditAsText (self): > pass > > additional_buttons = (('EditAsText', EditAsText),) > > It might be that Python simple doesn't do exactly what I want. This is really unusual; mostly I think it's great and highly intuitive. No biggie; I'm sure I can find a workaround! Bob From greg.ewing at canterbury.ac.nz Fri Feb 22 05:59:10 2013 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 22 Feb 2013 17:59:10 +1300 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: <5126BE2A.70804@mrabarnett.plus.com> References: <5126ACC2.3080904@canterbury.ac.nz> <5126BE2A.70804@mrabarnett.plus.com> Message-ID: <5126FB1E.8030504@canterbury.ac.nz> MRAB wrote: > On 2013-02-21 23:24, Greg Ewing wrote: > >> For example, enums in >> Pascal are definitely distinct types from integers, yet the language >> explicitly assigns them ordinal values and defines an ordering for >> them. Wirth must have thought there was a benefit in doing that. > > It means that sets of enums can be implemented using bitsets. That could still have been done without exposing the ordinal values. Yet, Pascal provides an ord() function. I suppose one viewpoint might be that as long as there is a defined ordering, there's always going to be at least an implied mapping to natural numbers, so there's not much point in trying to hide it. -- Greg From steve at pearwood.info Fri Feb 22 07:00:47 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 22 Feb 2013 17:00:47 +1100 Subject: [Python-ideas] Functions and Unbound methods In-Reply-To: <51258235.3090802@verizon.net> References: <51258235.3090802@verizon.net> Message-ID: <5127098F.6090503@pearwood.info> On 21/02/13 13:11, robert2682 wrote: > > > Hi, > > I'm new; greetings all! > > I'm not sure if this is a bug or feature, but it confused me so I thought > I'd raise the issue. Not a bug, but a feature, and in fact a fundamental way that Python works. The def statement creates functions. Always, without exception. So when you define a method in a class, you're actually defining a function. It doesn't get converted to a method until later, when it is retrieved via attribute access from the class or instance. You can google on "descriptor protocol" to learn more about it, although that is considered fairly advanced. This mailing list is actually intended for discussing new ideas and future changes to the Python language, standard library, and compiler, not for generic questions like this. For general questions, you should subscribe to the "python-list at python.org" mailing list, or "tutor@" if you are a beginner. If you repeat your question there, someone will reply with a more detailed response and some suggestions for your code. -- Steven From steve at pearwood.info Fri Feb 22 07:55:12 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 22 Feb 2013 17:55:12 +1100 Subject: [Python-ideas] Adding a safe alternative to pickle in the standard library In-Reply-To: References: <20130221121135.5a1aabd6@pitrou.net> <512621FF.6070606@trueblade.com> <20130221150049.2731682e@pitrou.net> <51262B08.3030600@trueblade.com> Message-ID: <51271650.2030808@pearwood.info> On 22/02/13 10:39, Bruce Leban wrote: > There's another problem - pickling is not guaranteed to work across Python > versions. I think you are confusing pickle with marshal. Obviously you cannot guarantee unpickling in older versions of Python, e.g. I can't unpickle a Python 2.7 set in Python 2.1, since sets didn't exist back then. Or a pickle created using version N of the protocol requires a Python version recent enough to understand version N, e.g. protocol 3 exists only in Python 3.x. But going forward should not be a problem, and the pickle documentation promises backwards-compatibility. Actually I believe that *forward* compatibility is a better description, but either way, the compatibility is guaranteed: Python version X will be able to unpickle anything pickled by an older version of Python; furthermore, anything pickled by version X will be able to be unpickled by future versions. (Modulo removal of the pickled classes, etc.) -- Steven From bruce at leapyear.org Fri Feb 22 08:03:40 2013 From: bruce at leapyear.org (Bruce Leban) Date: Thu, 21 Feb 2013 23:03:40 -0800 Subject: [Python-ideas] Adding a safe alternative to pickle in the standard library In-Reply-To: <51271650.2030808@pearwood.info> References: <20130221121135.5a1aabd6@pitrou.net> <512621FF.6070606@trueblade.com> <20130221150049.2731682e@pitrou.net> <51262B08.3030600@trueblade.com> <51271650.2030808@pearwood.info> Message-ID: On Thu, Feb 21, 2013 at 10:55 PM, Steven D'Aprano wrote: > On 22/02/13 10:39, Bruce Leban wrote: > > There's another problem - pickling is not guaranteed to work across Python >> versions. >> > > I think you are confusing pickle with marshal. > My bad. Should have rtfm. Thanks. --- Bruce Follow me: http://www.twitter.com/Vroo http://www.vroospeak.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Fri Feb 22 08:04:05 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 22 Feb 2013 18:04:05 +1100 Subject: [Python-ideas] Adding a safe alternative to pickle in the standard library In-Reply-To: <201302211733.02145.mark.hackett@metoffice.gov.uk> References: <201302211722.47380.mark.hackett@metoffice.gov.uk> <20130221182452.1ef90184@pitrou.net> <201302211733.02145.mark.hackett@metoffice.gov.uk> Message-ID: <51271865.4040900@pearwood.info> On 22/02/13 04:33, Mark Hackett wrote: > Being serious, though, if your code requires a serious amount of security, > you're better off writing your own parsing. If you're serious about security, you don't want amateurs trying to build security from scratch. And that includes yourself, if you are not a security expert. A programmer ought to be aware of their own limitations. I am not a security expert, and I don't have the time or inclination to become one. I want, no, I *need*, solutions for common problems to be safe by default, or at least for their vulnerabilities to be documented clearly and obviously in language I can understand, so I can write code with reasonable levels of security instead of inventing my own insecure, unsafe solutions. I know enough not to call eval() on data retrieved from untrusted sources. Not everyone even knows that much. I've seen code that literally downloaded content from a website, then eval'ed it without even a token attempt to sanitize it. Do you expect this person to write his own secure data serialiser? Anyone can write code with no security vulnerabilities that *they* can see. And frequently do. -- Steven From mark.hackett at metoffice.gov.uk Fri Feb 22 12:24:26 2013 From: mark.hackett at metoffice.gov.uk (Mark Hackett) Date: Fri, 22 Feb 2013 11:24:26 +0000 Subject: [Python-ideas] Adding a safe alternative to pickle in the standard library In-Reply-To: <51271865.4040900@pearwood.info> References: <201302211733.02145.mark.hackett@metoffice.gov.uk> <51271865.4040900@pearwood.info> Message-ID: <201302221124.26452.mark.hackett@metoffice.gov.uk> On Friday 22 Feb 2013, Steven D'Aprano wrote: > On 22/02/13 04:33, Mark Hackett wrote: > > Being serious, though, if your code requires a serious amount of > > security, you're better off writing your own parsing. > > If you're serious about security, you don't want amateurs trying to build > security from scratch. And if your code needs to be secure and you aren't capable of doing so, then pay someone to do it who is capable. You know, get a security expert. From mark.hackett at metoffice.gov.uk Fri Feb 22 12:26:55 2013 From: mark.hackett at metoffice.gov.uk (Mark Hackett) Date: Fri, 22 Feb 2013 11:26:55 +0000 Subject: [Python-ideas] Fwd: argparse - add support for environment variables In-Reply-To: References: <71869e7d-605c-4599-8010-0c195e86e982@googlegroups.com> Message-ID: <201302221126.55127.mark.hackett@metoffice.gov.uk> On Thursday 21 Feb 2013, Miki Tebeka wrote: > > What about config files? Mix them into argparse too? Oh, and support for > > .ini/XML/YAML/JSON/sqlite configs, please! > > Well, baby steps toward configuration solution :) > The other thing is that unlike config files which have many formats, > environment has one uniform access to it. > In the interests of doing it the UNIX way, would this be better done as a parser for the .ini/XML/YAML (they're ALL YAML, though. Yet ANOTHER Meta Language..!) that results in $ENV changing in python. Send the result that to argparse. Bada-bum, bada-bing, done. From stephen at xemacs.org Fri Feb 22 12:58:34 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 22 Feb 2013 20:58:34 +0900 Subject: [Python-ideas] Adding a safe alternative to pickle in the standard library In-Reply-To: <51271865.4040900@pearwood.info> References: <201302211722.47380.mark.hackett@metoffice.gov.uk> <20130221182452.1ef90184@pitrou.net> <201302211733.02145.mark.hackett@metoffice.gov.uk> <51271865.4040900@pearwood.info> Message-ID: <87liagsgph.fsf@uwakimon.sk.tsukuba.ac.jp> Steven D'Aprano writes: > A programmer ought to be aware of their own limitations. I am not a security > expert, and I don't have the time or inclination to become one. I want, no, > I *need*, solutions for common problems to be safe by default, or at least for > their vulnerabilities to be documented clearly and obviously in language I > can understand, so I can write code with reasonable levels of security instead > of inventing my own insecure, unsafe solutions. Sure. So just use JSON where it will do, and avoid pickle. No? Sure, you can make a case that "restricted pickle" would give you a trivial upgrade path if you find you really need it later. But it seems to me that if you think you need a protocol that executes serialized code automatically, you've got a heck of a lot of security work to do, beside which the effort to port from JSON API to pickle API is tiny. From stephen at xemacs.org Fri Feb 22 13:19:33 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 22 Feb 2013 21:19:33 +0900 Subject: [Python-ideas] Adding a safe alternative to pickle in the standard library In-Reply-To: <769b7cc7f941469d9a64c825cf9d5e1a@BLUPR03MB035.namprd03.prod.outlook.com> References: <20130221121135.5a1aabd6@pitrou.net> <512621FF.6070606@trueblade.com> <20130221150049.2731682e@pitrou.net> <51262B08.3030600@trueblade.com> <1E2DA813-A8AF-46CF-B7D4-5A1C1953B781@yahoo.com> <0f836a6d49664d2cb93c97bf496bab90@BLUPR03MB035.namprd03.prod.outlook.com> <87mwuxseoq.fsf@uwakimon.sk.tsukuba.ac.jp> <769b7cc7f941469d9a64c825cf9d5e1a@BLUPR03MB035.namprd03.prod.outlook.com> Message-ID: <87k3q0sfqi.fsf@uwakimon.sk.tsukuba.ac.jp> Steve Dower writes: > I was only aiming to provide perspective, rather than a proposal. Sure, but I didn't think there was a need for more "general" perspective. Pickle is a well-established protocol with certain risks and certain benefits, of which the python-dev community is basically well-aware. This is coming up *now* because recent events (Ruby/YAML has been mentioned) are causing some people to reevaluate the risks. This is a "quantitative" issue, and needs concrete proposals. > To be really explicit, I would make load/loads only work with > built-in types. For compatibility when reading earlier protocols we > could add a type representing a class instance and its members that > doesn't actually construct anything. (Maybe we override __getattr__ > to make it lazily construct the instance when the program actually > wants to use it?) I am not a security expert, but it seems to me that's going in the wrong direction. Unpickler would *still* run constructor code automatically under some circumstances -- but those circumstances become murkier. > For convenience, I'd add a parameter to Unpickler to let the user > provide a set of types that are allowed to be constructed (or a > mapping from names to callables as used in find_class()). And this is secure, why? There's no way to decorate the allowed types to add nasty stuff to the pickled class definitions (including built-in types), right? There are no bugs that allow a back door, right? Is the API sufficiently well-designed that users will easily figure out how to do what they need, and *only* what they need, and therefore won't be tempted to simply turn on permission to do *everything*? And they won't give up, and write their own? Isn't it better just to give users the advice to use JSON where it will do? Perhaps the difference in APIs will give them pause to think again if they're starting to think about unpickling classes? Granted, I don't have answers to those questions (except for myself!) But I think some thought should be given to them before trying to create a restricted pickle protocol and make it default. Restricted modes/protocols/sublanguages are hard to get right. From stephen at xemacs.org Fri Feb 22 13:29:38 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 22 Feb 2013 21:29:38 +0900 Subject: [Python-ideas] Adding a safe alternative to pickle in the standard library In-Reply-To: References: <20130221121135.5a1aabd6@pitrou.net> <512621FF.6070606@trueblade.com> <20130221150049.2731682e@pitrou.net> <51262B08.3030600@trueblade.com> <1E2DA813-A8AF-46CF-B7D4-5A1C1953B781@yahoo.com> <0f836a6d49664d2cb93c97bf496bab90@BLUPR03MB035.namprd03.prod.outlook.com> <87mwuxseoq.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <87ip5ksf9p.fsf@uwakimon.sk.tsukuba.ac.jp> Devin Jeanpierre writes: > That does not mean that we should not write or use safer alternatives. > We have written and do use safer alternatives, like the json module. Then why do we need a "safe alternative to pickle" when json is already in the standard library? > But it means we can't expect them to be usable exactly everywhere > pickle is. I would've said the question is how far in that direction > we should bother to go. OK, that is a better way to put what I have in mind. Well, we've already gone as far as json, which is pretty powerful (but still subject to attacks using "relatively secure" json to transport "insecure" data!) Why do we need an alternative *between* pickle and json? Maybe we should advocate that users think seriously about securing channels, and validating the pickles before doing anything with them, if they think they need more features than json offers? From jsbueno at python.org.br Fri Feb 22 13:53:27 2013 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Fri, 22 Feb 2013 09:53:27 -0300 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: <5126D0D7.7040302@btinternet.com> References: <5126ACC2.3080904@canterbury.ac.nz> <5126D0D7.7040302@btinternet.com> Message-ID: On 21 February 2013 22:58, Rob Cliffe wrote: > Apart from the point about maybe wanting to order enums, I agree. I think > Alex is right when he says you want to create singleton enums that don't > equate to anything else. We don't think of RED, GREEN, BLUE as being > identical with the ints 1,2,3, and so don't want our implementation of them > to compare equal to those ints. Again, I thinnkt hat my 2002 code that expected a (1) working when it got a (True) instead was a great, great thing. And I think that lots of places where magical strings today ("str.decode" errors argument, "compile" mode argument) should be lifted up to expect constants and still keep working if one pass them the strings "replace" or "eval". Requiring that things compare differently, IMHO, is too much statically-typed-minded - it is not the way of Python. As for Alex's surreal example of someone defining "NO" and "YES" with different values in different Enums and passing the wrong one to a given call - this person will have it comming. Think of this case instead: >>> disk_io.IDDLE == "iddle" True >>> network.IDDLE == "iddle" True >>> disk_io.IDDLE == network.IDDLE False So, if the compared enums do have an explicit value assigned to them (and I agree there are some cases the value is not needed) , they should compare just as that value - anything else is more close to C++ than Python. And no one is preventing people from using decorators or use other mecanisms for certain function calls to accept only values of a given Enum, just as no one ever was prohibited from statically verifying any other values in a call. From jeanpierreda at gmail.com Fri Feb 22 14:26:06 2013 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Fri, 22 Feb 2013 08:26:06 -0500 Subject: [Python-ideas] Adding a safe alternative to pickle in the standard library In-Reply-To: <87ip5ksf9p.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20130221121135.5a1aabd6@pitrou.net> <512621FF.6070606@trueblade.com> <20130221150049.2731682e@pitrou.net> <51262B08.3030600@trueblade.com> <1E2DA813-A8AF-46CF-B7D4-5A1C1953B781@yahoo.com> <0f836a6d49664d2cb93c97bf496bab90@BLUPR03MB035.namprd03.prod.outlook.com> <87mwuxseoq.fsf@uwakimon.sk.tsukuba.ac.jp> <87ip5ksf9p.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Fri, Feb 22, 2013 at 7:29 AM, Stephen J. Turnbull wrote: > > That does not mean that we should not write or use safer alternatives. > > We have written and do use safer alternatives, like the json module. > > Then why do we need a "safe alternative to pickle" when json is > already in the standard library? json can't handle cycles or new types -- at least, not without a rather significant amount of preprocessing and postprocessing that is little less work than writing your own serialization library from scratch. I do believe that support for these things is important. People do want to store frozensets of tuples, and objects which reference themselves, and so on. The easy way to do this is with pickle, and that encourages people to use pickle even where it is a bad idea. The easier it is to do something safer, the more likely it is people will do something safer. I've seen people use pickle even when advised that it was a security risk, because they didn't want to go through the effort of using the json module. To me that signals that something is wrong. > Well, we've already gone as far as json, which is pretty powerful (but > still subject to attacks using "relatively secure" json to transport > "insecure" data!) Of course a serialization library can't protect against eval(deserialize(foo)) running arbitrary code. That doesn't mean we shouldn't bother with security. > Why do we need an alternative *between* pickle and > json? Maybe we should advocate that users think seriously about > securing channels, and validating the pickles before doing anything > with them, if they think they need more features than json offers? Signed pickles and secured channels and so on don't solve the problem of trying to get data from an an untrusted user. Moreover I think it isn't an uncommon opinion that, even in that case, it'd be better to use json or some other nominally secure library instead of pickle, since it increases the number of mistakes necessary to compromise your security. -- Devin From solipsis at pitrou.net Fri Feb 22 15:05:57 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 22 Feb 2013 15:05:57 +0100 Subject: [Python-ideas] range References: Message-ID: <20130222150557.36884c3f@pitrou.net> Le Thu, 21 Feb 2013 13:35:17 +0000 (UTC), Wolfgang Maier a ?crit : > Dear all, > what do you think about allowing the use of operators on ranges? > > I thought of things like: > a = range(1,10) > b = range(5,12) > intersect = a & b # equivalent to intersect = range(5,10) > merge = a | b # equivalent to merge = range(1,12) > > to make this work in more complex cases, like: > a = range(1,10) > b = range(12,15) > merge = a | b # equivalent to sth. like > range(range(1,10),range(12,15)) -1. range() is a contiguous sequence, not an arbitrary container. Simplicity is part of its virtues. Regards Antoine. From ncoghlan at gmail.com Fri Feb 22 16:57:36 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 23 Feb 2013 01:57:36 +1000 Subject: [Python-ideas] range In-Reply-To: <20130222150557.36884c3f@pitrou.net> References: <20130222150557.36884c3f@pitrou.net> Message-ID: On Sat, Feb 23, 2013 at 12:05 AM, Antoine Pitrou wrote: > Le Thu, 21 Feb 2013 13:35:17 +0000 (UTC), > Wolfgang Maier > a > ?crit : >> Dear all, >> what do you think about allowing the use of operators on ranges? >> >> I thought of things like: >> a = range(1,10) >> b = range(5,12) >> intersect = a & b # equivalent to intersect = range(5,10) >> merge = a | b # equivalent to merge = range(1,12) >> >> to make this work in more complex cases, like: >> a = range(1,10) >> b = range(12,15) >> merge = a | b # equivalent to sth. like >> range(range(1,10),range(12,15)) > > -1. range() is a contiguous sequence, not an arbitrary container. > Simplicity is part of its virtues. -1 from me as well. We don't even implement concatenation and repetition for ranges in order to keep them simple - we have to caveat the range docs to point out they don't *quite* implement the full sequence API due to this restriction. However, I will note that starting in Python 3.3, range objects expose their "start", "stop" and "step" attributes. That makes it much easier for third party software to manipulate them arithmetically. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From Steve.Dower at microsoft.com Fri Feb 22 17:59:32 2013 From: Steve.Dower at microsoft.com (Steve Dower) Date: Fri, 22 Feb 2013 16:59:32 +0000 Subject: [Python-ideas] Adding a safe alternative to pickle in the standard library In-Reply-To: <87k3q0sfqi.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20130221121135.5a1aabd6@pitrou.net> <512621FF.6070606@trueblade.com> <20130221150049.2731682e@pitrou.net> <51262B08.3030600@trueblade.com> <1E2DA813-A8AF-46CF-B7D4-5A1C1953B781@yahoo.com> <0f836a6d49664d2cb93c97bf496bab90@BLUPR03MB035.namprd03.prod.outlook.com> <87mwuxseoq.fsf@uwakimon.sk.tsukuba.ac.jp> <769b7cc7f941469d9a64c825cf9d5e1a@BLUPR03MB035.namprd03.prod.outlook.com> <87k3q0sfqi.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <065d6242ab4940638811c9f3071e68b7@BLUPR03MB035.namprd03.prod.outlook.com> > From: Stephen J. Turnbull [mailto:stephen at xemacs.org] > Steve Dower writes: > > To be really explicit, I would make load/loads only work with > > built-in types. For compatibility when reading earlier protocols we > > could add a type representing a class instance and its members that > > doesn't actually construct anything. (Maybe we override __getattr__ > > to make it lazily construct the instance when the program actually > > wants to use it?) > > I am not a security expert, but it seems to me that's going in the wrong > direction. Unpickler would *still* run constructor code automatically under > some circumstances -- but those circumstances become murkier. Agreed on the bit in parentheses, that's probably questionable enough to ignore. However, if it only works with built-in types then there is no user code that will run. IIUC we already have a C implementation of pickle that is immune to users redefining builtins (if not, we should do this too). Pickled objects would be unpickled as (effectively) a tuple of the members - ('system', ("echo Hello World")) does not execute any code. (And yeah, wrap that tuple up in a type that can be tested.) > > For convenience, I'd add a parameter to Unpickler to let the user > > provide a set of types that are allowed to be constructed (or a > > mapping from names to callables as used in find_class()). > > And this is secure, why? There's no way to decorate the allowed types to > add nasty stuff to the pickled class definitions (including built-in types), right? Code is only pickled by name. Unpickler resolves the names and returns the class or function reference in the current environment. If it can't find the module or name in its current environment, it raises an error. > There are no bugs that allow a back door, right? Of course not. That's why we never see security patches or updates for operating systems or platforms. This is a silly argument. > Is the API sufficiently well-designed that users will easily figure out how to > do what they need, and *only* what they need, and therefore won't be > tempted to simply turn on permission to do *everything*? All we can ever do is provide instructions to keep the developer safe and make it clear that ignoring those rules will reduce the security of their program. It's up to the developer to make the right decisions. > And they won't give up, and write their own? In my experience, people more often write their own out of ignorance rather than frustration (same for unnecessarily using XML). Or they'll switch to an earlier version of Python that doesn't have this change in it. Again, we can encourage, but not dictate. > Isn't it better just to give users the advice to use JSON where it will do? > Perhaps the difference in APIs will give them pause to think again if they're > starting to think about unpickling classes? Maybe, though since pickle is literally the Python equivalent (should it have been called Python Object Notation (PON)? Probably not...) we should be ensuring that it is the best it can be. > Granted, I don't have answers to those questions (except for myself!) But I > think some thought should be given to them before trying to create a > restricted pickle protocol and make it default. Restricted > modes/protocols/sublanguages are hard to get right. Agreed. I don't think we need a new protocol though, just a less permissive default implementation of Unpickler.find_class(). From abarnert at yahoo.com Fri Feb 22 18:07:32 2013 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 22 Feb 2013 09:07:32 -0800 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: <5126ACC2.3080904@canterbury.ac.nz> References: <5126ACC2.3080904@canterbury.ac.nz> Message-ID: On Feb 21, 2013, at 15:24, Greg Ewing wrote: > On 22/02/13 08:18, Alex Stewart wrote: >> >> It doesn't seem >> unreasonable, therefore, to define two different categories of enums: one that >> has no concept of "value" (for pure-Python), and one which does have >> associated values but all values have to be specified explicitly > > That sounds reasonable. However, I'm wondering if there isn't a > third case: where you don't care about the values, but you do > want them to have a defined ordering? And a fourth case: You don't care about the values, but you want to be able to | them together into a set. For example, consider file open flags, mmap access flags, chmod flags, etc. You need some way to pass READ | TEXT, WRITE | SHARED, etc. occasionally, but most of the time you just pass READ or WRITE. It wouldn't be too onerous to require an explicit {READ, SHARED} set for the uncommon case, but set(READ) for the common case would be horrible. And you don't want every function that takes enums to typeswitch so it can handle a single value as if it were a set. The obvious answer to this is the one every other language uses: take READ | SHARED. This doesn't have to mean flag enums have integral values. However, the only alternative I can think of is that enums have set syntax, so READ | SHARED returns something like READ.__class__({READ, SHARED}), which of course also means that set has to automatically be a valid value for the type. Which is pretty magical and complex. Of course the answer could be not to allow that. Call mmap(file, access=WRITE, contention=SHARED), not mmap(WRITE | SHARED). But if we're talking about changing the stdlib and hopefully most popular third party libs, that would be a pretty drastic change. Finally, the answer could be: if you want that, you have to give them int values, and accept ints instead of enums in your APIs, at which point I think the benefit of valueless enums has gone way down, because many places you'd think you want them (including all over the stdlib), you don't. From g.rodola at gmail.com Fri Feb 22 18:15:43 2013 From: g.rodola at gmail.com (=?ISO-8859-1?Q?Giampaolo_Rodol=E0?=) Date: Fri, 22 Feb 2013 18:15:43 +0100 Subject: [Python-ideas] My wishlist for Tulip Message-ID: I understand it's still beta but anyways, here's my little wish list for Tulip. * provide a 'demo' / 'example' directory containing very simple scripts showing the most basic usages such as: - echo_tcp_client.py - echo_tcp_server.py - echo_tcp_server_w_timeout.py (same as echo_server.py but also disconnects the client after a certain time of inactivity) - echo_tcp_ssl_client.py - echo_tcp_ssl_server.py - echo_udp_client.py - echo_udp_server.py * move all *test*.py scripts in a separate 'test' directory * if it's not part of the API intended to be public move tulip/http_client.py elsewhere ('examples'/'demo' or a brand new 'scripts'/'tools' directory) * (minor) same for check.py, crawl.py, curl.py, sslsrv.py which looks like they belong elsewhere * write a simple benchmark framework testing (at least) sending, receiving and the internal scheduler (I'd like to help with this one) --- Giampaolo http://code.google.com/p/pyftpdlib/ http://code.google.com/p/psutil/ http://code.google.com/p/pysendfile/ From abarnert at yahoo.com Fri Feb 22 18:29:36 2013 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 22 Feb 2013 09:29:36 -0800 Subject: [Python-ideas] Adding a safe alternative to pickle in the standard library In-Reply-To: <0f836a6d49664d2cb93c97bf496bab90@BLUPR03MB035.namprd03.prod.outlook.com> References: <20130221121135.5a1aabd6@pitrou.net> <512621FF.6070606@trueblade.com> <20130221150049.2731682e@pitrou.net> <51262B08.3030600@trueblade.com> <1E2DA813-A8AF-46CF-B7D4-5A1C1953B781@yahoo.com> <0f836a6d49664d2cb93c97bf496bab90@BLUPR03MB035.namprd03.prod.outlook.com> Message-ID: On Feb 21, 2013, at 9:55, Steve Dower wrote: >> From: Andrew Barnert >> How often have you needed either cyclic references or the ability to >> dynamically store arbitrary classes in something like a cookie or a cache file? > > In a past life I used pickle regularly to snapshot long-running (evolutionary) algorithms that used user-provided classes and all sorts of highly improper circular references. And there are plenty of researchers out there using Python for much crazier things than I ever did. > > There is a lot more to Python than web apps... But you're not storing those pickles in a cookie, which is exactly my point. There are many cases where you need the power of pickle. There are also many cases where you need safe serialization. But there's not much overlap. There are plenty of cases where you need safety, and also need a little more power than JSON--but you still don't usually need the full power of pickle for those cases, and making it easier to extend the json lib is a much cleaner way forward than making it easier to restrict pickle. It's true that "not much overlap" != "no overlap". But you can't cover everything. If you're building, say, an online interactive python interpreter that saves and restores its state between sessions, you're going to have to think through the security implications. That doesn't mean someone who wants to just store scientific data and doesn't have untested sources, or someone building a web app who doesn't care about storing arbitrary dynamically defined types, should have the same burden. From wolfgang.maier at biologie.uni-freiburg.de Fri Feb 22 18:40:29 2013 From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier) Date: Fri, 22 Feb 2013 17:40:29 +0000 (UTC) Subject: [Python-ideas] range References: <20130222150557.36884c3f@pitrou.net> Message-ID: Nick Coghlan writes: > -1 from me as well. We don't even implement concatenation and > repetition for ranges in order to keep them simple - we have to caveat > the range docs to point out they don't *quite* implement the full > sequence API due to this restriction. > > However, I will note that starting in Python 3.3, range objects expose > their "start", "stop" and "step" attributes. That makes it much easier > for third party software to manipulate them arithmetically. > > Cheers, > Nick. > Any plans to make range a valid base class to build upon in future releases? Best, Wolfgang From abarnert at yahoo.com Fri Feb 22 18:41:02 2013 From: abarnert at yahoo.com (Andrew Barnert) Date: Fri, 22 Feb 2013 09:41:02 -0800 Subject: [Python-ideas] Adding a safe alternative to pickle in the standard library In-Reply-To: References: <20130221121135.5a1aabd6@pitrou.net> <512621FF.6070606@trueblade.com> <20130221150049.2731682e@pitrou.net> <51262B08.3030600@trueblade.com> <1E2DA813-A8AF-46CF-B7D4-5A1C1953B781@yahoo.com> <0f836a6d49664d2cb93c97bf496bab90@BLUPR03MB035.namprd03.prod.outlook.com> <87mwuxseoq.fsf@uwakimon.sk.tsukuba.ac.jp> <87ip5ksf9p.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Feb 22, 2013, at 5:26, Devin Jeanpierre wrote: >> Well, we've already gone as far as json, which is pretty powerful (but >> still subject to attacks using "relatively secure" json to transport >> "insecure" data!) > > Of course a serialization library can't protect against > eval(deserialize(foo)) running arbitrary code. That doesn't mean we > shouldn't bother with security. The difference is that json.loads is just deserialize(foo), which pickle.loads inherently has some eval mixed in. That's why I think for most use cases, the answer is making json easier to extend, not making pickle easier to secure. The biggest problem people have with the json library isn't that you have to do the extending explicitly and externally, but that it's a huge pain to do so. There was a suggestion earlier in this thread (I forget the author) that would go a long way toward relieving that pain. Some people also want it to be implicitly extensible, to have some way to create an instance of a new empty class named Foo with given attributes (but not an existing builtin or user-defined class named Foo). I'm not sure what their use case is, and I'm not sure it's a good idea--but if it is, there was also a suggestion for that. From jeanpierreda at gmail.com Fri Feb 22 19:14:16 2013 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Fri, 22 Feb 2013 13:14:16 -0500 Subject: [Python-ideas] Adding a safe alternative to pickle in the standard library In-Reply-To: References: <20130221121135.5a1aabd6@pitrou.net> <512621FF.6070606@trueblade.com> <20130221150049.2731682e@pitrou.net> <51262B08.3030600@trueblade.com> <1E2DA813-A8AF-46CF-B7D4-5A1C1953B781@yahoo.com> <0f836a6d49664d2cb93c97bf496bab90@BLUPR03MB035.namprd03.prod.outlook.com> <87mwuxseoq.fsf@uwakimon.sk.tsukuba.ac.jp> <87ip5ksf9p.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Fri, Feb 22, 2013 at 12:41 PM, Andrew Barnert wrote: > The difference is that json.loads is just deserialize(foo), which pickle.loads inherently has some eval mixed in. > > That's why I think for most use cases, the answer is making json easier to extend, not making pickle easier to secure. My original suggestion was to add a third thing, such as cerealizer, not to restrict pickle or extend json. Some others have talked about restricting pickle, but I don't know how one could do that and still be confident in the safety of the end product. You usually build things to be safe from the ground up, not as some afterthought with a few restrictions. > The biggest problem people have with the json library isn't that you have to do the extending explicitly and externally, but that it's a huge pain to do so. There was a suggestion earlier in this thread (I forget the author) that would go a long way toward relieving that pain. I feel that'd be very helpful, yes. Obviously not as helpful as something that can handle cyclic references, but those aren't really as important. Besides which, a yaml module could synthesize something more complete out of these pieces (YAML is like JSON, but with support for cyclic references and some extra syntax). My issue is making safe serialization easier, so that not using pickle is a viable option. As you say, we can go a long way towards this using the json module. -- Devin From foogod at gmail.com Fri Feb 22 19:21:57 2013 From: foogod at gmail.com (Alex Stewart) Date: Fri, 22 Feb 2013 10:21:57 -0800 (PST) Subject: [Python-ideas] PEP for enum library type? In-Reply-To: References: <5126ACC2.3080904@canterbury.ac.nz> <5126D0D7.7040302@btinternet.com> Message-ID: <2a0e7db0-2cae-4b62-867f-0f6919d7d29c@googlegroups.com> On Friday, February 22, 2013 4:53:27 AM UTC-8, Joao S. O. Bueno wrote: > Again, I thinnkt hat my 2002 code that expected a (1) working when it > got a (True) instead was a great, great thing. And I think that lots > of places where magical strings today ("str.decode" errors argument, > "compile" mode argument) should be lifted up to expect constants and > still keep working if one pass them the strings "replace" or "eval". > Ok, first of all, nobody's suggesting that wouldn't work. I'm not talking about comparing enums with base types (which I have no problem working like this), I'm talking specifically about comparing enums directly with other enums from a different category. Requiring that things compare differently, IMHO, is too much > statically-typed-minded - it is not the way of Python. As for Alex's > surreal example of someone defining "NO" and "YES" with different > values in different Enums and passing the wrong one to a given call - > this person will have it comming. I think this is unnecessarily dismissive. Obviously, my example was simplified, but what if these two enums were created by two completely different people in different modules that just happen to be used in the same program? Does the hapless programmer "have it coming" then? You say that my suggestion is more like C++ than Python, but I counter that your suggestion seems to me a lot more like Perl than Python. It smacks of one of the things I dislike most about Perl, which also happens to be, in my opinion, one of the language features most responsible for (often subtle) bugs in Perl code even by experienced programmers: The contents of variables can have very different semantic meanings depending solely on the context in which they're evaluated (scalar vs array, etc). In this case what you're saying is that just because entirely unrelated programmers happened to use the same int value underneath, that Align.TOP and Errno.EPERM and Choice.OK should all be considered to mean exactly the same thing everywhere (even if none of the programmers actually intended or foresaw that possibility), but what that meaning is actually depends entirely on what context they happen to be evaluated in. Another way of looking at this is that what you're proposing is effectively implicit type-casting between enums of different types. Python does support implicit casting in a very few well-defined cases, but not most of the time, and only when doing so does not discard important semantic information (i.e. "meaning"). For ints, floats, etc, they really only have one piece of semantic information ("a number value"). Enums, however, have more than that. They have (potentially) a number value, but they also have names, and they (preferably) have information about "what type of thing am I". Automatically throwing out that extra information on a comparison where both objects have the extra info and it's not the same on both objects is, in my opinion, just wrong. If there were no extra semantic meaning to enums, then this wouldn't be an issue, but the symbolic information they carry is clearly important, because it's the whole reason we're having this discussion in the first place instead of just using ints everywhere. (Indeed, it could easily be argued that the entire point to things like int-enums is that the programmer isn't supposed to have to care what the underlying value is, but should be able to think of things entirely symbolically.) In pretty much all the cases I can imagine (and that have been suggested thus far), it seems to me that passing a different class of enum than the other side is testing against is almost always going to be an accident, or at the very least sloppy code (if there are cases where it is intentional, it's unlikely to be common, and the programmer should be able to explicitly cast between them somehow). Given this, I think that the Principle of Least Surprise enters into things here as well. If somebody makes a coding mistake and uses the wrong type of enum somewhere, which is likely to be the least surprising outcome?: 1. If you use an enum of the wrong type, it won't be considered equivalent to any of the expected options. 2. If you use an enum of the wrong type, it might not be considered to be equivalent to any of the expected options, or it might match one of them and do something either the same or completely different than you expect, but you have no way to tell what will happen without looking at an underlying constant which you're not supposed to have to care about. --Alex -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsbueno at python.org.br Fri Feb 22 19:32:28 2013 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Fri, 22 Feb 2013 15:32:28 -0300 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: <2a0e7db0-2cae-4b62-867f-0f6919d7d29c@googlegroups.com> References: <5126ACC2.3080904@canterbury.ac.nz> <5126D0D7.7040302@btinternet.com> <2a0e7db0-2cae-4b62-867f-0f6919d7d29c@googlegroups.com> Message-ID: On 22 February 2013 15:21, Alex Stewart wrote: > On Friday, February 22, 2013 4:53:27 AM UTC-8, Joao S. O. Bueno wrote: >> >> Again, I thinnkt hat my 2002 code that expected a (1) working when it >> got a (True) instead was a great, great thing. And I think that lots >> of places where magical strings today ("str.decode" errors argument, >> "compile" mode argument) should be lifted up to expect constants and >> still keep working if one pass them the strings "replace" or "eval". > > > Ok, first of all, nobody's suggesting that wouldn't work. I'm not talking > about comparing enums with base types (which I have no problem working like > this), I'm talking specifically about comparing enums directly with other > enums from a different category. (big snip) Ok. I do agree with your argumentation. And in fact, my previous example of ---------- >>> disk_io.IDDLE == "iddle" True >>> network.IDDLE == "iddle" True >>> disk_io.IDDLE == network.IDDLE False ----- If the final behavior in some cases is this, no one will get harmed. We alread hav e NaN == Nan -> False and the language does not explode. :-) So, indeed - values in distinct enums should compare differently, even if they compare equal to their underlying value. (sorry for making you write the extensive e-mail) js -><- From foogod at gmail.com Fri Feb 22 19:43:41 2013 From: foogod at gmail.com (Alex Stewart) Date: Fri, 22 Feb 2013 10:43:41 -0800 (PST) Subject: [Python-ideas] PEP for enum library type? In-Reply-To: References: <5126ACC2.3080904@canterbury.ac.nz> <5126D0D7.7040302@btinternet.com> <2a0e7db0-2cae-4b62-867f-0f6919d7d29c@googlegroups.com> Message-ID: > (sorry for making you write the extensive e-mail) > No worries.. Having to write it all out actually helped me clarify my own thoughts and make sure that my intuitive position did actually make logical sense to me, so it was probably a good thing to do anyway :) --Alex -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Fri Feb 22 19:46:27 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 22 Feb 2013 10:46:27 -0800 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: References: <5126ACC2.3080904@canterbury.ac.nz> Message-ID: <5127BD03.5030703@stoneleaf.us> On 02/22/2013 09:07 AM, Andrew Barnert wrote: > On Feb 21, 2013, at 15:24, Greg Ewing wrote: > >> On 22/02/13 08:18, Alex Stewart wrote: >>> >>> It doesn't seem >>> unreasonable, therefore, to define two different categories of enums: one that >>> has no concept of "value" (for pure-Python), and one which does have >>> associated values but all values have to be specified explicitly >> >> That sounds reasonable. However, I'm wondering if there isn't a >> third case: where you don't care about the values, but you do >> want them to have a defined ordering? > > And a fourth case: You don't care about the values, but you want to be able to | them together into a set. > > For example, consider file open flags, mmap access flags, chmod flags, etc. You need some way to pass READ | TEXT, WRITE | SHARED, etc. occasionally, but most of the time you just pass READ or WRITE. It wouldn't be too onerous to require an explicit {READ, SHARED} set for the uncommon case, but set(READ) for the common case would be horrible. And you don't want every function that takes enums to typeswitch so it can handle a single value as if it were a set. The obvious answer to this is the one every other language uses: take READ | SHARED. > > This doesn't have to mean flag enums have integral values. However, the only alternative I can think of is that enums have set syntax, so READ | SHARED returns something like READ.__class__({READ, SHARED}), which of course also means that set has to automatically be a valid value for the type. Which is pretty magical and complex. > > Of course the answer could be not to allow that. Call mmap(file, access=WRITE, contention=SHARED), not mmap(WRITE | SHARED). But if we're talking about changing the stdlib and hopefully most popular third party libs, that would be a pretty drastic change. > > Finally, the answer could be: if you want that, you have to give them int values, and accept ints instead of enums in your APIs, at which point I think the benefit of valueless enums has gone way down, because many places you'd think you want them (including all over the stdlib), you don't. We shouldn't get hung up on "don't care about the values" == "no value assigned". I personally don't see any value in a valueless enum; I do see value in three other types: sequence (based on int), bitmask (based on int), and unique (or string -- based on str). It seems to me that we're throwing around "valueless" because we don't want some enums to support math operations -- so use the string version for that type. As for your example above: Python 3.2.3 (default, Oct 19 2012, 19:53:16) [GCC 4.7.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. -->from yaenum import BitMaskEnum, enum -->BitMaskEnum.create('MMap', 'READ WRITE TEXT SHARED', export=globals()) -->READ, WRITE, TEXT, SHARED (MMap('READ', value=1), MMap('WRITE', value=2), MMap('TEXT', value=4), MMap('SHARED', value=8)) -->READ | TEXT MMap('READ|TEXT', value=5) -->WRITE | SHARED MMap('WRITE|SHARED', value=10) From ethan at stoneleaf.us Fri Feb 22 19:33:29 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 22 Feb 2013 10:33:29 -0800 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: <2a0e7db0-2cae-4b62-867f-0f6919d7d29c@googlegroups.com> References: <5126ACC2.3080904@canterbury.ac.nz> <5126D0D7.7040302@btinternet.com> <2a0e7db0-2cae-4b62-867f-0f6919d7d29c@googlegroups.com> Message-ID: <5127B9F9.9020704@stoneleaf.us> On 02/22/2013 10:21 AM, Alex Stewart wrote: > On Friday, February 22, 2013 4:53:27 AM UTC-8, Joao S. O. Bueno wrote: > Requiring that things compare differently, IMHO, is too much > statically-typed-minded - it is not the way of Python. As for Alex's > surreal example of someone defining "NO" and "YES" with different > values in different Enums and passing the wrong one to a given call - > this person will have it comming. > > > I think this is unnecessarily dismissive. Obviously, my example was simplified, but what if these two enums were > created by two completely different people in different modules that just happen to be used in the same program? Does > the hapless programmer "have it coming" then? +1 > You say that my suggestion is more like C++ than Python, but I counter that your suggestion seems to me a lot more like > Perl than Python. It smacks of one of the things I dislike most about Perl, which also happens to be, in my opinion, > one of the language features most responsible for (often subtle) bugs in Perl code even by experienced programmers: The > contents of variables can have very different semantic meanings depending solely on the context in which they're > evaluated (scalar vs array, etc). In this case what you're saying is that just because entirely unrelated programmers > happened to use the same int value underneath, that Align.TOP and Errno.EPERM and Choice.OK should all be considered to > mean exactly the same thing everywhere (even if none of the programmers actually intended or foresaw that possibility), > but what that meaning is actually depends entirely on what context they happen to be evaluated in. +1 > Another way of looking at this is that what you're proposing is effectively implicit type-casting between enums of > different types. Python does support implicit casting in a very few well-defined cases, but not most of the time, and > only when doing so does not discard important semantic information (i.e. "meaning"). For ints, floats, etc, they really > only have one piece of semantic information ("a number value"). Enums, however, have more than that. They have > (potentially) a number value, but they also have names, and they (preferably) have information about "what type of thing > am I". Automatically throwing out that extra information on a comparison where both objects have the extra info and > it's not the same on both objects is, in my opinion, just wrong. If there were no extra semantic meaning to enums, then > this wouldn't be an issue, but the symbolic information they carry is clearly important, because it's the whole reason > we're having this discussion in the first place instead of just using ints everywhere. +1 > In pretty much all the cases I can imagine (and that have been suggested thus far), it seems to me that passing a > different class of enum than the other side is testing against is almost always going to be an accident, or at the very > least sloppy code (if there are cases where it is intentional, it's unlikely to be common, and the programmer should be > able to explicitly cast between them somehow). Given this, I think that the Principle of Least Surprise enters into > things here as well. If somebody makes a coding mistake and uses the wrong type of enum somewhere, which is likely to > be the least surprising outcome?: > > 1. If you use an enum of the wrong type, it won't be considered equivalent to any of the expected options. > 2. If you use an enum of the wrong type, it might not be considered to be equivalent to any of the expected options, or > it might match one of them and do something either the same or completely different than you expect, but you have no > way to tell what will happen without looking at an underlying constant which you're not supposed to have to care about. +1000 Just to be clear, option 1 is what should happen. -- ~Ethan~ From solipsis at pitrou.net Fri Feb 22 19:43:10 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 22 Feb 2013 19:43:10 +0100 Subject: [Python-ideas] range References: <20130222150557.36884c3f@pitrou.net> Message-ID: <20130222194310.75129c1f@pitrou.net> On Fri, 22 Feb 2013 17:40:29 +0000 (UTC) Wolfgang Maier wrote: > Nick Coghlan writes: > > > -1 from me as well. We don't even implement concatenation and > > repetition for ranges in order to keep them simple - we have to caveat > > the range docs to point out they don't *quite* implement the full > > sequence API due to this restriction. > > > > However, I will note that starting in Python 3.3, range objects expose > > their "start", "stop" and "step" attributes. That makes it much easier > > for third party software to manipulate them arithmetically. > > > > Cheers, > > Nick. > > > > Any plans to make range a valid base class to build upon in future releases? I suppose it wouldn't very difficult to make range subclassable. You can try writing a patch if you want: http://docs.python.org/devguide/ Regards Antoine. From bruce at leapyear.org Fri Feb 22 20:08:34 2013 From: bruce at leapyear.org (Bruce Leban) Date: Fri, 22 Feb 2013 11:08:34 -0800 Subject: [Python-ideas] Adding a safe alternative to pickle in the standard library In-Reply-To: References: <20130221121135.5a1aabd6@pitrou.net> <512621FF.6070606@trueblade.com> <20130221150049.2731682e@pitrou.net> <51262B08.3030600@trueblade.com> <1E2DA813-A8AF-46CF-B7D4-5A1C1953B781@yahoo.com> <0f836a6d49664d2cb93c97bf496bab90@BLUPR03MB035.namprd03.prod.outlook.com> <87mwuxseoq.fsf@uwakimon.sk.tsukuba.ac.jp> <87ip5ksf9p.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: To take this back to the ideas stage, one idea might be to integrate hmac into pickle. At a minimum, provide some sample code showing how to wrap an hmac around a pickled object. Related to this I note that it would be helpful if the hmac docs gave some advice on key generation (e.g., suggested length). Or to be a little more convenient, add new methods like these: pickle.set_hmac_key([key]) Sets the key used when pickle hmacs are generated. The key is as expected by hmac.new. If key is not provided or this function is not called before using an hmac, a random key is generated that will vary each time the program is run. This is useful if you do not want pickles to be reusable between different runs of the program. pickle.dump_hmac(obj, file[, protocol]) Same as pickle.dump except that it attaches an hmac to the pickled data. pickle.dumps_hmac(obj[, protocol]) Same as pickle.dumps except that it attaches an hmac to the pickled data. pickle.load_hmac(file) Same as pickle.load except that it verifies and removes an hmac as attached by dump_hmac. Raises UnpicklingHmacError if the hmac cannot be verified. pickle.loads_hmac(string) Same as pickle.loads except that it (1) verifies and removes an hmac as attached by dump_hmac and (2) it does not ignore extra characters. Raises UnpicklingHmacError if the hmac cannot be verified. The reason I suggest setting the hmac key at a global level rather than in each call is that this eliminates the need for either passing around the key or generating keys at multiple points in the code. If a key were passed in each call, it would have the benefit that a program could use multiple keys to ensure that pickles from one part of the program were not unpickled in other parts. This seems like a heavy-handed use of the feature. The reason I suggest using new methods rather than adding a keyword arg to the current methods is that this facilitates wholesale replacement of pickle.dump with pickle.dump_hmac and I don't envision an explosion of variations. Usually I'm an advocate of doing it the other way around. :-) --- Bruce Latest blog post: Alice's Puzzle Page http://www.vroospeak.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Fri Feb 22 20:09:12 2013 From: dholth at gmail.com (Daniel Holth) Date: Fri, 22 Feb 2013 14:09:12 -0500 Subject: [Python-ideas] range In-Reply-To: <20130222194310.75129c1f@pitrou.net> References: <20130222150557.36884c3f@pitrou.net> <20130222194310.75129c1f@pitrou.net> Message-ID: On Fri, Feb 22, 2013 at 1:43 PM, Antoine Pitrou wrote: > On Fri, 22 Feb 2013 17:40:29 +0000 (UTC) > Wolfgang Maier > > wrote: > > Nick Coghlan writes: > > > > > -1 from me as well. We don't even implement concatenation and > > > repetition for ranges in order to keep them simple - we have to caveat > > > the range docs to point out they don't *quite* implement the full > > > sequence API due to this restriction. > > > > > > However, I will note that starting in Python 3.3, range objects expose > > > their "start", "stop" and "step" attributes. That makes it much easier > > > for third party software to manipulate them arithmetically. > > > > > > Cheers, > > > Nick. > > > > > > > Any plans to make range a valid base class to build upon in future > releases? > > I suppose it wouldn't very difficult to make range subclassable. > You can try writing a patch if you want: > http://docs.python.org/devguide/ > > Regards I would be very happy to see a similar data structure available (on pypi). I was writing an HTTP-backed file object that used partial GET requests to just fetch the parts of the file that were actually read; it would have benefited from an efficient set-of-ranges (start+stop but not step) implementation. -------------- next part -------------- An HTML attachment was scrubbed... URL: From foogod at gmail.com Fri Feb 22 20:14:52 2013 From: foogod at gmail.com (Alex Stewart) Date: Fri, 22 Feb 2013 11:14:52 -0800 (PST) Subject: [Python-ideas] PEP for enum library type? In-Reply-To: References: <5126ACC2.3080904@canterbury.ac.nz> Message-ID: <892ca40c-474b-4e34-ad94-94051a0b4182@googlegroups.com> On Friday, February 22, 2013 9:07:32 AM UTC-8, Andrew Barnert wrote: > > On Feb 21, 2013, at 15:24, Greg Ewing > > wrote: > > That sounds reasonable. However, I'm wondering if there isn't a > > third case: where you don't care about the values, but you do > > want them to have a defined ordering? > This is a valid point. I think there are a lot of cases where enums are useful without any ordering, but there are arguably some cases where order would be useful (the immediate example that comes to mind is "logging levels", where DEBUG, INFO, and ERROR don't necessarily need to have any numerical meaning, but it would be nice to be able to say that DEBUG < INFO < ERROR).. I think it should be possible to build this into an existing enum implementation which doesn't inherently rely on underlying numbers, though.. I'll look into it.. And a fourth case: You don't care about the values, but you want to be able > to | them together into a set. > Heh.. I was kinda hoping nobody was going to bring this up yet :) Good point, though, and FWIW, I had considered this too but didn't bring it up initially because I was afraid it would muddy the discussion too much too early. I do however agree that this is a use case that will ultimately be important to support, and it's been in the back of my mind. This actually applies both to the "valueless" case and the "valued" case, but in somewhat different ways. As you mentioned, in the "valueless" case, the most obvious way to deal with this is with sets. I think we could probably ameliorate the single-value case a bit by just having enum objects also behave like single-item sets if used with set operations (which I don't think is too magic), and then we can define the '|' operator to just create (or add to) an enum-set.. The valued case is actually more complicated, because ideally if READ = enum(1) and SHARED = enum(4), then saying "READ | SHARED" should produce something that has an int() value of the or'd values (5), but it would also be nice if it still represented itself symbolically (as "READ | SHARED", for example, instead of "5"), and though not strictly required, it would probably also be nice if they could be inspected using the same set-type operations as valueless enums/sets ("if READ in openmode", for example). This would probably require some sort of new "enum-set" class/type which supported amalgomated valued enums, but I think it would still be doable without too much magic (I hope). Then of course there's extra issues like: if we also support string-enums, what does 'or' mean for string constants? (etc, etc..) But in summary, I think the valueless case is actually pretty easy to implement, but doing it well with valued-enums is much more work, which once again reinforces my opinion that valueless enums are useful to have, and preferable to use when lowlevel-type-compatibility is not explicitly required by some API.. --Alex -------------- next part -------------- An HTML attachment was scrubbed... URL: From foogod at gmail.com Fri Feb 22 20:33:41 2013 From: foogod at gmail.com (Alex Stewart) Date: Fri, 22 Feb 2013 11:33:41 -0800 (PST) Subject: [Python-ideas] iterable.__unpack__ method Message-ID: So when putting together my enum implementation I came up against something I've run into a couple of times in a few different situations, and I thought warranted some discussion as its own proposal: There are times where it would be extremely convenient if there was a way when unpacking an object (using the "a, b, c = foo" notation) for that object to know how many values it needed to provide. I would like to propose the following: When the unpack notation is used, the interpreter will automatically look for an __unpack__ method on the rvalue, and if it exists will call it as: rvalue.__unpack__(min_count, max_count) ... and use the returned value as the iterable to unpack into the variables instead of the original object. (If no __unpack__ method exists, it will just use the rvalue itself, as it does currently) In the case of simple unpacking ("a, b, c = foo"), min_count and max_count will be the same value (3). In the case of extended unpacking ("a, b, c, *d = foo"), min_count would be the smallest length that will not cause an unpacking exception (3, in this case), and max_count would be None. By effectively separating the notion of "unpackable" from "iterable", this would allow for a few useful things: 1. It would allow otherwise non-iterable objects to support the unpack notation if it actually makes sense to do so 2. It would allow for "smart" unpackable objects (such as __ in my enum example), which would "just work" no matter how many values they're required to produce. 3. It would allow (properly designed) infinite (or unknown-possibly-large-size) iterators to be unpackable into a finite sequence of variables if desired (which can only be done with extra special-case code currently) 4. It could make backwards-compatibility easier for APIs which return iterables (i.e. if you want to change a function that used to return a 3-tuple to return a 5-tuple, instead of inherently breaking all existing code that chose to unpack the return value (assuming only 3 items would ever be returned), you could return an unpackable object which will work correctly with both old and new code) Thoughts? --Alex -------------- next part -------------- An HTML attachment was scrubbed... URL: From foogod at gmail.com Fri Feb 22 20:25:08 2013 From: foogod at gmail.com (Alex Stewart) Date: Fri, 22 Feb 2013 11:25:08 -0800 (PST) Subject: [Python-ideas] Yet another enum proposal :) Message-ID: <2ec14b4b-2617-4563-8022-eec921b96fd1@googlegroups.com> Ok, so at the risk of muddying the waters even more, I've put together yet another possible way to do enums, and would be interested to hear comments.. Based on the list of required/desired/undesired properties laid out in the enum PEP thread, I believe a lot of the back and forth to date has been due to the fact that all of the proposed implementations fall short of fulfilling at least some of the desired properties for a general-purpose enum implementation in different ways. I've put together something based on ideas from Tim's, Barry's, other things thrown out in discussion, and a few of my own which I think comes closer to ticking off most of the boxes. (The code and some examples are available at https://github.com/foogod/pyenum) Enums/groups are defined by creating subclasses of Enum, similarly to Barry's implementation. The distinction is that the base "Enum" class does not have any associated values (they're just singleton objects with names). At a basic level, the values themselves are defined like so: class Color (Enum): RED = __ GREEN = __ BLUE = __ As has been (quite correctly) pointed out before, the single-underscore (_) has a well-established meaning in most circles related to gettext/etc, so for this application I picked the next best thing, the double-underscore (__) instead. I think this is reasonably mnemonic as a "fill in the blank" placeholder, and also not unduly verbose. One advantage of using __ over, say, ellipsis (...) is that since it involves a name resolution, we can add (just a little!) magic to generate distinct __ objects on each reference, so, for example, the following actually works as the user probably expects it to: class Color (Enum): RED = CRIMSON = __ BLUE = __ (RED and CRIMSON actually become aliases referring to the same enum value, but BLUE is a different enum value) One other rather notable advantage to __ is that we can define a special multiplication behavior for it which allows us to make the syntax much more compact: class Color (Enum): RED, GREEN, BLUE, ORANGE, VIOLET, BEIGE, ULTRAVIOLET, ULTRABEIGE = __ * 8 (as an aside, I have an idea about how we might be able to get rid of the "* 8" altogether, but it requires a different (I think generally useful) change to the language which I will propose separately) Each enum value is actually an instance of its defining class, so you can determine what type of enum something is with simple inheritance checks: >>> isinstance(Color.RED, Color) True For enums which need to have int values, we can use IntEnum instead of Enum: class Errno (IntEnum): EPERM = 1 ENOENT = 2 ESRCH = 3 EINTR = 4 (Technically, IntEnum is just a special case of the more generic TypeEnum class: class IntEnum (TypeEnum, basetype=int): pass ..which means that theoretically you could define enums based on (almost) any base type (examples.py has an example using floats)) Anyway, as expected, enums have reasonable strs/reprs, and can be easily translated to/from base values using traditional "casting" syntax: >>> Errno.EPERM <__main__.Errno.EPERM (1)> >>> str(Errno.EPERM) 'EPERM' >>> int(Errno.EPERM) 1 >>> Errno(1) <__main__.Errno.EPERM (1)> You can also lookup enums by name using index notation: >>> Errno['EPERM'] <__main__.Errno.EPERM (1)> It's actually worth noting here that TypeEnums are actually subclasses of their basetype, so IntEnums are also ints, and can be used as drop-in replacements for any existing code which is expecting an int argument. They can also be compared directly as if they were ints: if exc.errno == Errno.EPERM: do_something() TypeEnums enforce uniqueness: >>> class Foo (IntEnum): ... A = 1 ... B = 1 ... Traceback (most recent call last): File "", line 1, in ... ValueError: Duplicate enum value: 1 However, you can define "aliases" for enums, if you want to: >>> class Foo (IntEnum): ... A = 1 ... B = A ... >>> Foo.A <__main__.Foo.A (1)> >>> Foo.B <__main__.Foo.A (1)> Of course, pure (valueless) Enum instances are logical singletons (i.e. they only compare equal to themselves), so there's no potential for duplication there. Finally, a special feature of __ is that it can also be called with parameters to create enum values with docstrings or other metadata: >>> class Errno (IntEnum): ... EPERM = __(1, doc="Permission denied") ... >>> Errno.EPERM.__doc__ 'Permission denied' >>> class Color (Enum): ... RED = __(doc="The color red", rgb=(1., 0., 0.)) ... >>> Color.RED.__doc__ 'The color red' >>> Color.RED.rgb (1.0, 0.0, 0.0) A few other notable properties: - Enum values are hashable, so they can be used as dict keys, etc. - Though enum values will compare equal to their associated basetype values (Errno.EPERM == 1), enum values from different classes never compare equal to each other (Foo.A != Errno.EPERM). This prevents enum-aware code from accidentally mistaking one enum for a completely different enum (with a completely different meaning) simply because they map to the same int value. - Using the class constructor does not create new enums, it just returns the existing singleton enum associated with the passed value (Errno(1) returns Errno.EPERM, not a new Errno). If there is no predefined enum matching that value, a ValueError is raised, thus using the constructor is basically a "cast" operation, and only works within the supported range of values. (New enum values can be created using the .new() method on the class, however, if desired) So... thoughts? --Alex -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri Feb 22 22:17:59 2013 From: guido at python.org (Guido van Rossum) Date: Fri, 22 Feb 2013 13:17:59 -0800 Subject: [Python-ideas] My wishlist for Tulip In-Reply-To: References: Message-ID: Those are all good ideas! Now I just have to find time to implement them... On Fri, Feb 22, 2013 at 9:15 AM, Giampaolo Rodol? wrote: > I understand it's still beta but anyways, here's my little wish list for Tulip. > > * provide a 'demo' / 'example' directory containing very simple > scripts showing the most basic usages such as: > > - echo_tcp_client.py > - echo_tcp_server.py > - echo_tcp_server_w_timeout.py (same as echo_server.py but also > disconnects the client after a certain time of inactivity) > - echo_tcp_ssl_client.py > - echo_tcp_ssl_server.py > - echo_udp_client.py > - echo_udp_server.py > > * move all *test*.py scripts in a separate 'test' directory > > * if it's not part of the API intended to be public move > tulip/http_client.py elsewhere ('examples'/'demo' or a brand new > 'scripts'/'tools' directory) > > * (minor) same for check.py, crawl.py, curl.py, sslsrv.py which looks > like they belong elsewhere > > * write a simple benchmark framework testing (at least) sending, > receiving and the internal scheduler (I'd like to help with this one) > > > --- Giampaolo > http://code.google.com/p/pyftpdlib/ > http://code.google.com/p/psutil/ > http://code.google.com/p/pysendfile/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas -- --Guido van Rossum (python.org/~guido) From fafhrd91 at gmail.com Fri Feb 22 23:37:44 2013 From: fafhrd91 at gmail.com (Nikolay Kim) Date: Fri, 22 Feb 2013 14:37:44 -0800 Subject: [Python-ideas] My wishlist for Tulip In-Reply-To: References: Message-ID: I'm planing to work on udp support sometime next week. i'll implement echo examples except udp, later today. On Feb 22, 2013, at 1:17 PM, Guido van Rossum wrote: > Those are all good ideas! Now I just have to find time to implement them... > > On Fri, Feb 22, 2013 at 9:15 AM, Giampaolo Rodol? wrote: >> I understand it's still beta but anyways, here's my little wish list for Tulip. >> >> * provide a 'demo' / 'example' directory containing very simple >> scripts showing the most basic usages such as: >> >> - echo_tcp_client.py >> - echo_tcp_server.py >> - echo_tcp_server_w_timeout.py (same as echo_server.py but also >> disconnects the client after a certain time of inactivity) >> - echo_tcp_ssl_client.py >> - echo_tcp_ssl_server.py >> - echo_udp_client.py >> - echo_udp_server.py >> >> * move all *test*.py scripts in a separate 'test' directory >> >> * if it's not part of the API intended to be public move >> tulip/http_client.py elsewhere ('examples'/'demo' or a brand new >> 'scripts'/'tools' directory) >> >> * (minor) same for check.py, crawl.py, curl.py, sslsrv.py which looks >> like they belong elsewhere >> >> * write a simple benchmark framework testing (at least) sending, >> receiving and the internal scheduler (I'd like to help with this one) >> we can test performance on gunicorn with 3 different workers (sync, gevent, gtulip) From ethan at stoneleaf.us Sat Feb 23 00:16:45 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 22 Feb 2013 15:16:45 -0800 Subject: [Python-ideas] Yet another enum proposal :) In-Reply-To: <2ec14b4b-2617-4563-8022-eec921b96fd1@googlegroups.com> References: <2ec14b4b-2617-4563-8022-eec921b96fd1@googlegroups.com> Message-ID: <5127FC5D.3040609@stoneleaf.us> On 02/22/2013 11:25 AM, Alex Stewart wrote: > Ok, so at the risk of muddying the waters even more, I've put together yet another possible way to do enums, and would > be interested to hear comments.. Largely similar to my own implementation -- so of course I like it! :) > A few other notable properties: > > * Enum values are hashable, so they can be used as dict keys, etc. Problem here: should we have our enums hash the same as the underlying value? Consider: --> import yaenum --> class Color(yaenum.Enum): ... black ... red ... green ... blue ... --> class Literature(yaenum.Enum): ... scifi ... fantasy ... mystery ... pop ... --> Color.black Color('black', value=0) --> Literature.scifi Literature('scifi', value=0) --> black = Color.black --> scifi = Literature.scifi --> black == 0 True --> hash(black) 0 --> scifi == 0 True --> hash(scifi) 0 --> black == scifi False --> hash(0) 0 --> huh = dict() --> huh[black] = 9 --> huh {Color('black', value=0): 9} --> huh[0] 9 --> huh[scifi] Traceback (most recent call last): File "", line 1, in KeyError: Literature('scifi', value=0) --> huh[scifi] = 11 --> huh {Color('black', value=0): 9, Literature('scifi', value=0): 11} --> huh[0] 9 --> del huh[0] --> huh[0] 11 From a practicality standpoint the question is: How likely is it to use different enum classes as keys? -- ~Ethan~ From alexandre.zani at gmail.com Sat Feb 23 00:45:32 2013 From: alexandre.zani at gmail.com (Alexandre Zani) Date: Fri, 22 Feb 2013 15:45:32 -0800 Subject: [Python-ideas] Yet another enum proposal :) In-Reply-To: <5127FC5D.3040609@stoneleaf.us> References: <2ec14b4b-2617-4563-8022-eec921b96fd1@googlegroups.com> <5127FC5D.3040609@stoneleaf.us> Message-ID: On Fri, Feb 22, 2013 at 3:16 PM, Ethan Furman wrote: > On 02/22/2013 11:25 AM, Alex Stewart wrote: > >> Ok, so at the risk of muddying the waters even more, I've put together >> yet another possible way to do enums, and would >> be interested to hear comments.. >> > > Largely similar to my own implementation -- so of course I like it! :) > > > A few other notable properties: >> >> * Enum values are hashable, so they can be used as dict keys, etc. >> > > Problem here: should we have our enums hash the same as the underlying > value? Consider: > One feature of the implementation is that enum values from different enum classes will never equal each other. It would be very weird for them to have identical hashes. > > --> import yaenum > > --> class Color(yaenum.Enum): > ... black > ... red > ... green > ... blue > ... > > --> class Literature(yaenum.Enum): > ... scifi > ... fantasy > ... mystery > ... pop > ... > > --> Color.black > Color('black', value=0) > > --> Literature.scifi > Literature('scifi', value=0) > > --> black = Color.black > > --> scifi = Literature.scifi > > --> black == 0 > True > > --> hash(black) > 0 > > --> scifi == 0 > True > > --> hash(scifi) > 0 > > --> black == scifi > False > > --> hash(0) > 0 > > --> huh = dict() > --> huh[black] = 9 > --> huh > {Color('black', value=0): 9} > > --> huh[0] > 9 > --> huh[scifi] > > Traceback (most recent call last): > File "", line 1, in > KeyError: Literature('scifi', value=0) > > --> huh[scifi] = 11 > --> huh > {Color('black', value=0): 9, Literature('scifi', value=0): 11} > > --> huh[0] > 9 > > --> del huh[0] > --> huh[0] > 11 > > From a practicality standpoint the question is: How likely is it to use > different enum classes as keys? > > -- > ~Ethan~ > ______________________________**_________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/**mailman/listinfo/python-ideas > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bruce at leapyear.org Sat Feb 23 00:57:25 2013 From: bruce at leapyear.org (Bruce Leban) Date: Fri, 22 Feb 2013 15:57:25 -0800 Subject: [Python-ideas] Yet another enum proposal :) In-Reply-To: <5127FC5D.3040609@stoneleaf.us> References: <2ec14b4b-2617-4563-8022-eec921b96fd1@googlegroups.com> <5127FC5D.3040609@stoneleaf.us> Message-ID: On Fri, Feb 22, 2013 at 3:16 PM, Ethan Furman wrote: > > Problem here: should we have our enums hash the same as the underlying > value? Consider: > > > > From a practicality standpoint the question is: How likely is it to use > different enum classes as keys? > > Having the same hash value isn't the problem. >>> hash(-1) == hash(-2) True >>> d={} >>> d[-1] = 'm1' >>> d[-2] = 'm2' >>> d {-2: 'm2', -1: 'm1'} >>> d[-2] 'm2' >>> del d[-2] >>> d[-2] Traceback (most recent call last): File "", line 1, in KeyError: -2 >>> The problem is that black == 0 and scifi == 0. So when the hash values collide, it chains them and then uses == to compare the 0 to find the matching value in the table. To avoid this problem ensure that hash(enum) != hash(int(enum)) [or whatever the base type of the enum is]. Actually, I'm not sure that works without looking at the implementation of dict. When there are multiple values in a bucket, does it compare hash values first or does it jump right to ==? --- Bruce -------------- next part -------------- An HTML attachment was scrubbed... URL: From foogod at gmail.com Sat Feb 23 01:01:30 2013 From: foogod at gmail.com (Alex Stewart) Date: Fri, 22 Feb 2013 16:01:30 -0800 Subject: [Python-ideas] Yet another enum proposal :) In-Reply-To: References: <2ec14b4b-2617-4563-8022-eec921b96fd1@googlegroups.com> <5127FC5D.3040609@stoneleaf.us> Message-ID: On Fri, Feb 22, 2013 at 3:45 PM, Alexandre Zani wrote: > On Fri, Feb 22, 2013 at 3:16 PM, Ethan Furman wrote: > >> * Enum values are hashable, so they can be used as dict keys, etc. >>> >> >> Problem here: should we have our enums hash the same as the underlying >> value? Consider: >> > > One feature of the implementation is that enum values from different enum > classes will never equal each other. It would be very weird for them to > have identical hashes. > Actually, not really. The only requirement in Python is that if two objects compare equal, they need to have the same hash value, but there is no requirement the other direction (indeed, that would be a problem in many cases as the domain of hash values is inherently much smaller than the domain of all possible object data, so it must be expected that you will get hash duplication from time to time, no matter what method you use for calculating the hashes). In the case of pure Enums, they're just singleton object instances, so they hash the same as any other object, which works pretty much as one would expect. In the case of TypeEnums, since they do compare equal to the underlying value, they must also hash the same. However hashes are just a "hint": Two objects can hash the same but not compare equal, in which case they will be considered to be different objects when used as dictionary keys, etc. There is actually an example of this in the examples.py I put up in github with the reference code. It creates a dictionary with two keys, which are two different Enums, which both use the same int value. Because they do not compare equal, they are treated as different keys by Python, as I think most people would expect.. --Alex -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexandre.zani at gmail.com Sat Feb 23 01:09:32 2013 From: alexandre.zani at gmail.com (Alexandre Zani) Date: Fri, 22 Feb 2013 16:09:32 -0800 Subject: [Python-ideas] Yet another enum proposal :) In-Reply-To: References: <2ec14b4b-2617-4563-8022-eec921b96fd1@googlegroups.com> <5127FC5D.3040609@stoneleaf.us> Message-ID: On Fri, Feb 22, 2013 at 4:01 PM, Alex Stewart wrote: > On Fri, Feb 22, 2013 at 3:45 PM, Alexandre Zani wrote: > >> On Fri, Feb 22, 2013 at 3:16 PM, Ethan Furman wrote: >> >>> * Enum values are hashable, so they can be used as dict keys, etc. >>>> >>> >>> Problem here: should we have our enums hash the same as the underlying >>> value? Consider: >>> >> >> One feature of the implementation is that enum values from different enum >> classes will never equal each other. It would be very weird for them to >> have identical hashes. >> > > Actually, not really. The only requirement in Python is that if two > objects compare equal, they need to have the same hash value, but there is > no requirement the other direction (indeed, that would be a problem in many > cases as the domain of hash values is inherently much smaller than the > domain of all possible object data, so it must be expected that you will > get hash duplication from time to time, no matter what method you use for > calculating the hashes). > My bad. Please forget I ever mentioned it. > > In the case of pure Enums, they're just singleton object instances, so > they hash the same as any other object, which works pretty much as one > would expect. > > In the case of TypeEnums, since they do compare equal to the underlying > value, they must also hash the same. However hashes are just a "hint": Two > objects can hash the same but not compare equal, in which case they will be > considered to be different objects when used as dictionary keys, etc. > There is actually an example of this in the examples.py I put up in github > with the reference code. It creates a dictionary with two keys, which are > two different Enums, which both use the same int value. Because they do > not compare equal, they are treated as different keys by Python, as I think > most people would expect.. > > --Alex > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bruce at leapyear.org Sat Feb 23 01:38:45 2013 From: bruce at leapyear.org (Bruce Leban) Date: Fri, 22 Feb 2013 16:38:45 -0800 Subject: [Python-ideas] Yet another enum proposal :) In-Reply-To: References: <2ec14b4b-2617-4563-8022-eec921b96fd1@googlegroups.com> <5127FC5D.3040609@stoneleaf.us> Message-ID: On Fri, Feb 22, 2013 at 3:57 PM, Bruce Leban wrote: > To avoid this problem ensure that hash(enum) != hash(int(enum)) [or > whatever the base type of the enum is]. > Never mind. That's a bad idea. As others pointed out, if two objects compare equal, they have to have the same hash value. This weird behavior is a side effect of having non-transitive equality (a == b, b == c but a != c) and objects like dicts are not designed to work with objects with non-transitive equality. Not having transitivity when it's expected leads to weird behavior. For example, if you have non-transitive inequality (a < b < c < a, as is the case in PHP, for example) then sorting may not work and some sorting code can even get into infinite loops. --- Bruce Latest blot post: Alice's Puzzle Page http://www.vroospeak.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Sat Feb 23 02:12:49 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 22 Feb 2013 17:12:49 -0800 Subject: [Python-ideas] Yet another enum proposal :) In-Reply-To: References: <2ec14b4b-2617-4563-8022-eec921b96fd1@googlegroups.com> <5127FC5D.3040609@stoneleaf.us> Message-ID: <51281791.4040000@stoneleaf.us> On 02/22/2013 04:38 PM, Bruce Leban wrote: > > On Fri, Feb 22, 2013 at 3:57 PM, Bruce Leban > wrote: > > To avoid this problem ensure that hash(enum) != hash(int(enum)) [or whatever the base type of the enum is]. > > > Never mind. That's a bad idea. As others pointed out, if two objects compare equal, they have to have the same hash > value. This weird behavior is a side effect of having non-transitive equality (a == b, b == c but a != c) and objects > like dicts are not designed to work with objects with non-transitive equality. Not having transitivity when it's > expected leads to weird behavior. For example, if you have non-transitive inequality (a < b < c < a, as is the case in > PHP, for example) then sorting may not work and some sorting code can even get into infinite loops. Yeah, I can easily see this issue keeping an enum implementation out of the stdlib; perhaps an acceptable breakage would be to have enum instances not be hashable? -- ~Ethan~ From ethan at stoneleaf.us Sat Feb 23 01:41:00 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 22 Feb 2013 16:41:00 -0800 Subject: [Python-ideas] Yet another enum proposal :) In-Reply-To: References: <2ec14b4b-2617-4563-8022-eec921b96fd1@googlegroups.com> <5127FC5D.3040609@stoneleaf.us> Message-ID: <5128101C.3070803@stoneleaf.us> On 02/22/2013 03:57 PM, Bruce Leban wrote: > > On Fri, Feb 22, 2013 at 3:16 PM, Ethan Furman > wrote: > > > Problem here: should we have our enums hash the same as the underlying value? Consider: > > > > > From a practicality standpoint the question is: How likely is it to use different enum classes as keys? > > > Having the same hash value isn't the problem. True -- the problem is using two different enum classes with the same underlying int value as keys and then using ints instead of the enums to try to access the values. > The problem is that black == 0 and scifi == 0. So when the hash values collide, it chains them and then uses == to > compare the 0 to find the matching value in the table. To avoid this problem ensure that hash(enum) != hash(int(enum)) > [or whatever the base type of the enum is]. If we do this, then a plain int won't be able to look up an enum key, which would mean that int enums are not drop-in replacements for int's currently be used in enum-type contexts. I'm inclined to leave the problem as is, though, unless somebody has a use case where they are using two different enum classes in a single dictionary? > Actually, I'm not sure that works without looking at the implementation of > dict. When there are multiple values in a bucket, does it compare hash values first or does it jump right to ==? dict uses hash first -- after all, it's a hash table. :) -- ~Ethan~ From g.rodola at gmail.com Sat Feb 23 03:14:15 2013 From: g.rodola at gmail.com (=?ISO-8859-1?Q?Giampaolo_Rodol=E0?=) Date: Sat, 23 Feb 2013 03:14:15 +0100 Subject: [Python-ideas] My wishlist for Tulip In-Reply-To: References: Message-ID: Also, I had some troubles figuring out how to run tests because setup.py doesn't actually install Tulip and "import tulip" can be used only from within the root directory. > I'm planing to work on udp support sometime next week. > i'll implement echo examples except udp, later today. Thanks. --- Giampaolo http://code.google.com/p/pyftpdlib/ http://code.google.com/p/psutil/ http://code.google.com/p/pysendfile/ From foogod at gmail.com Sat Feb 23 03:57:36 2013 From: foogod at gmail.com (Alex Stewart) Date: Fri, 22 Feb 2013 18:57:36 -0800 (PST) Subject: [Python-ideas] Yet another enum proposal :) In-Reply-To: <5128101C.3070803@stoneleaf.us> References: <2ec14b4b-2617-4563-8022-eec921b96fd1@googlegroups.com> <5127FC5D.3040609@stoneleaf.us> <5128101C.3070803@stoneleaf.us> Message-ID: <378431d8-7927-4da4-9ca0-280a1733d36f@googlegroups.com> On Friday, February 22, 2013 4:41:00 PM UTC-8, stoneleaf wrote: > > I'm inclined to leave the problem as is, though, unless somebody has a use > case where they are using two different > enum classes in a single dictionary? I'll admit the behavior in this case is a little unconventional, but to be honest I think it's something that most people would find made a reasonable amount of sense if they stopped and thought about it for a minute, so I don't think it's that horrible (and seems like it'd be pretty rare anyway).. In my opinion it's far less disturbing than some of the behavior we already have in the language for certain types of keys: >>> x = {True: 'foo', 1: 'bar'} >>> x {True: 'bar'} --Alex -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sat Feb 23 06:17:15 2013 From: guido at python.org (Guido van Rossum) Date: Fri, 22 Feb 2013 21:17:15 -0800 Subject: [Python-ideas] My wishlist for Tulip In-Reply-To: References: Message-ID: Personally I think you'd be crazy to install this version of tulip (it's not beta, it's way before alpha!). The setup.py file only exists to build the C extension for Windows. (How to run tests is explained in README BTW. There is no boilerplate there. :-) From tjreedy at udel.edu Sat Feb 23 10:14:52 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 23 Feb 2013 04:14:52 -0500 Subject: [Python-ideas] iterable.__unpack__ method In-Reply-To: References: Message-ID: On 2/22/2013 2:33 PM, Alex Stewart wrote: > So when putting together my enum implementation I came up against > something I've run into a couple of times in a few different situations, > and I thought warranted some discussion as its own proposal: There are > times where it would be extremely convenient if there was a way when > unpacking an object (using the "a, b, c = foo" notation) for that object > to know how many values it needed to provide. Such statements always have a definite number of targets, so you can always arrange for the iterable to produce the required number of items -- unless it cannot. The count can be used as a multiplier, to slice or islice, or as an argument to an iterator class or function (such as generator functions). > I would like to propose the following: > > When the unpack notation is used, the interpreter will automatically > look for an __unpack__ method on the rvalue, and if it exists will call > it as: > > rvalue.__unpack__(min_count, max_count) > > ... and use the returned value as the iterable to unpack into the > variables instead of the original object. (If no __unpack__ method > exists, it will just use the rvalue itself, as it does currently) It would be much easier, and have much the same effect, if unpacking simply requested the minumum number of items and stopped raising a ValueError if the iterator has more items. No new protocol is needed. Guido rejected this as silently masking errors. -- Terry Jan Reedy From tjreedy at udel.edu Sat Feb 23 10:21:08 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 23 Feb 2013 04:21:08 -0500 Subject: [Python-ideas] range In-Reply-To: <20130222194310.75129c1f@pitrou.net> References: <20130222150557.36884c3f@pitrou.net> <20130222194310.75129c1f@pitrou.net> Message-ID: On 2/22/2013 1:43 PM, Antoine Pitrou wrote: > On Fri, 22 Feb 2013 17:40:29 +0000 (UTC) > Wolfgang Maier > >> Any plans to make range a valid base class to build upon in future releases? > > I suppose it wouldn't very difficult to make range subclassable. > You can try writing a patch if you want: > http://docs.python.org/devguide/ I was just about to suggest that making range subclassable would be an alternative to extending the current class. As far as I know, there is no principled reason why not, unlike Nonetype and bool. -- Terry Jan Reedy From rosuav at gmail.com Sat Feb 23 11:18:49 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 23 Feb 2013 21:18:49 +1100 Subject: [Python-ideas] iterable.__unpack__ method In-Reply-To: References: Message-ID: On Sat, Feb 23, 2013 at 8:14 PM, Terry Reedy wrote: > It would be much easier, and have much the same effect, if unpacking simply > requested the minumum number of items and stopped raising a ValueError if > the iterator has more items. No new protocol is needed. Guido rejected this > as silently masking errors. What if it were done explicitly, though? Currently: >>> a,b,c=range(3) >>> a,b,c=range(4) Traceback (most recent call last): File "", line 1, in a,b,c=range(4) ValueError: too many values to unpack (expected 3) >>> a,b,c,*d=range(6) >>> a,b,c,* =range(4) SyntaxError: invalid syntax Suppose the last notation were interpreted as "request values for a, b, and c, and then ignore the rest". This would support infinite iterators, and would be an explicit statement from the caller that it's not an error to have more elements. This doesn't solve the backward-compatibility issue, but it would allow a client to specifically engage a forward-compatibility mode (by putting ,* after any unpack that might introduce more elements - of course, you'd need to be confident that you won't care about those elements). It can be done with current code, of course. >>> a,b,c=itertools.islice(range(6),3) But this has significant overhead in verbiage, where a simple ",*" would carry the same sort of meaning it does in other contexts. ChrisA From ncoghlan at gmail.com Sat Feb 23 12:11:11 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 23 Feb 2013 21:11:11 +1000 Subject: [Python-ideas] range In-Reply-To: References: <20130222150557.36884c3f@pitrou.net> <20130222194310.75129c1f@pitrou.net> Message-ID: On Sat, Feb 23, 2013 at 7:21 PM, Terry Reedy wrote: > On 2/22/2013 1:43 PM, Antoine Pitrou wrote: >> >> On Fri, 22 Feb 2013 17:40:29 +0000 (UTC) >> Wolfgang Maier >> > > >>> Any plans to make range a valid base class to build upon in future >>> releases? >> >> >> I suppose it wouldn't very difficult to make range subclassable. >> You can try writing a patch if you want: >> http://docs.python.org/devguide/ > > > I was just about to suggest that making range subclassable would be an > alternative to extending the current class. As far as I know, there is no > principled reason why not, unlike Nonetype and bool. I wasn't even aware you *couldn't* subclass it (I'd never tried). As far as I am aware, that's just a quirk inherited from the old xrange implementation rather than a deliberate design decision. On the other hand, I suspect for many cases involving more advanced range variants, containment would be a better option than inheritance, particular if you want to implement a type that can't be reliably described through a (start, stop, step) triple. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sat Feb 23 12:58:32 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 23 Feb 2013 21:58:32 +1000 Subject: [Python-ideas] iterable.__unpack__ method In-Reply-To: References: Message-ID: On Sat, Feb 23, 2013 at 8:18 PM, Chris Angelico wrote: > On Sat, Feb 23, 2013 at 8:14 PM, Terry Reedy wrote: >> It would be much easier, and have much the same effect, if unpacking simply >> requested the minumum number of items and stopped raising a ValueError if >> the iterator has more items. No new protocol is needed. Guido rejected this >> as silently masking errors. > > What if it were done explicitly, though? > > Currently: >>>> a,b,c=range(3) >>>> a,b,c=range(4) > Traceback (most recent call last): > File "", line 1, in > a,b,c=range(4) > ValueError: too many values to unpack (expected 3) >>>> a,b,c,*d=range(6) >>>> a,b,c,* =range(4) > SyntaxError: invalid syntax > > Suppose the last notation were interpreted as "request values for a, > b, and c, and then ignore the rest". It highly unlikely it will ever be interpreted that way, because it contradicts the way the unnamed star is used in function headers (that is, to indicate the start of the keyword only arguments *without* accepting an arbitrary number of positional arguments). If you want to ignore excess values in unpacking, a double-underscore is the best currently available option (even though it has the downside of not working well with infinite iterators or large sequences): >>> a,b,c,*__ =range(4) However, Alex's original idea of an "unpack protocol" (distinct from, but falling back to, the ordinary iterable protocol) definitely has at least a few points in its favour. 1. Iterating over heterogeneous types doesn't really make sense, but unpacking them does. A separate protocol lets a type support unpacking assignment without supporting normal iteration. 2. The protocol could potentially be designed to allow an *iterable* to be assigned to the star target rather than requiring it to be unpacked into a tuple. This could be used not only to make unpacking assignment safe to use with infinite iterators, but also to make it cheaper with large sequences (through appropriate use of itertools.islice in a custom container's __unpack__ implementation) and with virtual sequences like range() objects. 3. As Alex noted, if a function that previously returned a 2-tuple wants to start returning a 3-tuple, that's currently a backwards incompatible change because it will break unpacking assignment. With an unpack protocol, such a function can return an object that behaves like a 3-tuple most of the time, but also transparently supports unpacking assignment to two targets rather than only supporting three. I would suggest a different signature for "__unpack__", though, built around the fact the star target can be used at most once, but in an arbitrary location: def __unpack__(target_len, star_index=None): """Unpack values into an iterable of the specified length. If star_index is not None, the value at that index will be an iterable representing the remaining contents that were not unpacked. """ ... While I'm somewhere between +0 and +1 on the idea myself, there are some open questions/potential downsides to the idea: - this change increases the complexity of the language, by explicitly separating the concept of heterogeneous unpacking from homogeneous iteration. Those *are* two distinct concepts though, so the increased complexity may be justifiable. - for function parameter unpacking, would we allow "*args" to be an arbitrary iterable rather than promising that it is always a tuple? We would probably want to do so for consistency with unpacking assignment under this scheme, which may lead to some confusing error messages thrown by existing functions which expect it to always be a tuple. - we would want to document very clearly that it remains possible to bypass custom unpacking by explicitly calling iter(obj) rather than using obj directly. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From stephen at xemacs.org Sat Feb 23 13:37:31 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 23 Feb 2013 21:37:31 +0900 Subject: [Python-ideas] Adding a safe alternative to pickle in the standard library In-Reply-To: References: <20130221121135.5a1aabd6@pitrou.net> <512621FF.6070606@trueblade.com> <20130221150049.2731682e@pitrou.net> <51262B08.3030600@trueblade.com> <1E2DA813-A8AF-46CF-B7D4-5A1C1953B781@yahoo.com> <0f836a6d49664d2cb93c97bf496bab90@BLUPR03MB035.namprd03.prod.outlook.com> <87mwuxseoq.fsf@uwakimon.sk.tsukuba.ac.jp> <87ip5ksf9p.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <87bobbryt0.fsf@uwakimon.sk.tsukuba.ac.jp> Devin Jeanpierre writes: > > Well, we've already gone as far as json, which is pretty powerful (but > > still subject to attacks using "relatively secure" json to transport > > "insecure" data!) > > Of course a serialization library can't protect against > eval(deserialize(foo)) running arbitrary code. That doesn't mean we > shouldn't bother with security. Nobody's saying we shouldn't bother with security. Any answer needs to be informed by the recognition that nothing we can design is proof against the Sufficiently Stupid/Lazy User, that's all I'm trying to say. But security probably does have a cost in terms of inconvenience and restriction on capabilities. My question is "given that people can and will do stupid things with relatively safe libraries like json, what is the point of providing something intermediate between json and pickle?" In more detail, what features can we provide that don't involve the known risks of pickle that would be sufficiently attractive to users that they don't go to pickle anyway? You mention handling cycles, which adds minimal risk (unprepared code could infloop on the unpacked data, but that's not the serializer's fault), but also "new" types which isn't clear to me. If you mean new built-in types, can't the json module be extended? (That would apply to cycles as well, since we know it's possible it should be automatable.) If you mean user-defined types, we're back where we started, with merely unpacking data running code whose provenance we don't know. > > Why do we need an alternative *between* pickle and json? Maybe > > we should advocate that users think seriously about securing > > channels, and validating the pickles before doing anything with > > them, if they think they need more features than json offers? > > Signed pickles and secured channels and so on don't solve the problem > of trying to get data from an an untrusted user. Yeah yeah, sorry about the red herring. It's the first question that matters. From stephen at xemacs.org Sat Feb 23 14:11:25 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 23 Feb 2013 22:11:25 +0900 Subject: [Python-ideas] Adding a safe alternative to pickle in the standard library In-Reply-To: <065d6242ab4940638811c9f3071e68b7@BLUPR03MB035.namprd03.prod.outlook.com> References: <20130221121135.5a1aabd6@pitrou.net> <512621FF.6070606@trueblade.com> <20130221150049.2731682e@pitrou.net> <51262B08.3030600@trueblade.com> <1E2DA813-A8AF-46CF-B7D4-5A1C1953B781@yahoo.com> <0f836a6d49664d2cb93c97bf496bab90@BLUPR03MB035.namprd03.prod.outlook.com> <87mwuxseoq.fsf@uwakimon.sk.tsukuba.ac.jp> <769b7cc7f941469d9a64c825cf9d5e1a@BLUPR03MB035.namprd03.prod.outlook.com> <87k3q0sfqi.fsf@uwakimon.sk.tsukuba.ac.jp> <065d6242ab4940638811c9f3071e68b7@BLUPR03MB035.namprd03.prod.outlook.com> Message-ID: <87a9qvrx8i.fsf@uwakimon.sk.tsukuba.ac.jp> Steve Dower writes: > > There are no bugs that allow a back door, right? > > Of course not. That's why we never see security patches or updates > for operating systems or platforms. This is a silly argument. "Of course not." The right answer is that "it's been audited and we're as sure as we ever are." The problem is that (as Devin Jeanpierre wrote IIRC) pickle was not designed for security from the ground up. Removing execution of untrusted code from it may not be as easy as just saying so. Maybe it is. > > Is the API sufficiently well-designed that users will easily > > figure out how to do what they need, and *only* what they need, > > and therefore won't be tempted to simply turn on permission to do > > *everything*? > > All we can ever do is provide instructions to keep the developer > safe and make it clear that ignoring those rules will reduce the > security of their program. It's up to the developer to make the > right decisions. But that's not enough. What others have said here is that json doesn't get used when it would be perfectly suitable (and as secure as serialization gets!) because the API doesn't provide convenient access to the features they need. > Agreed. I don't think we need a new protocol though, just a less > permissive default implementation of Unpickler.find_class(). The proof of the pudding is in the auditing, I guess. From jeanpierreda at gmail.com Sat Feb 23 15:14:24 2013 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Sat, 23 Feb 2013 09:14:24 -0500 Subject: [Python-ideas] Adding a safe alternative to pickle in the standard library In-Reply-To: <87bobbryt0.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20130221121135.5a1aabd6@pitrou.net> <512621FF.6070606@trueblade.com> <20130221150049.2731682e@pitrou.net> <51262B08.3030600@trueblade.com> <1E2DA813-A8AF-46CF-B7D4-5A1C1953B781@yahoo.com> <0f836a6d49664d2cb93c97bf496bab90@BLUPR03MB035.namprd03.prod.outlook.com> <87mwuxseoq.fsf@uwakimon.sk.tsukuba.ac.jp> <87ip5ksf9p.fsf@uwakimon.sk.tsukuba.ac.jp> <87bobbryt0.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Sat, Feb 23, 2013 at 7:37 AM, Stephen J. Turnbull wrote: > Devin Jeanpierre writes: > Nobody's saying we shouldn't bother with security. Any answer needs > to be informed by the recognition that nothing we can design is proof > against the Sufficiently Stupid/Lazy User, that's all I'm trying to > say. Sorry. Fair enough. > But security probably does have a cost in terms of inconvenience and > restriction on capabilities. My question is "given that people can > and will do stupid things with relatively safe libraries like json, > what is the point of providing something intermediate between json and > pickle?" In more detail, what features can we provide that don't > involve the known risks of pickle that would be sufficiently > attractive to users that they don't go to pickle anyway? I believe that the features I'm suggesting meet that criterion (but see below for discussion of risk). Nothing will ever be sufficient to drive away all unwarranted use of pickle, but I feel like these two features are really big ones that would go a long way towards making the secure thing almost as easy in almost every circumstance. As long as I've ever personally wanted, although I can't speak for others. > You mention handling cycles, which adds minimal risk (unprepared code > could infloop on the unpacked data, but that's not the serializer's > fault), but also "new" types which isn't clear to me. If you mean new > built-in types, can't the json module be extended? (That would apply > to cycles as well, since we know it's possible it should be > automatable.) It can. This brings up an interesting point. YAML already extends JSON with cycle support (via aliases) and support for a notation for marking up nonstandard types (via tagging). For example: >>> yaml.load('&mydict {"a": !!python/tuple ["b", *mydict]}') {'a': ('b', {...})} PyYAML is useless security-wise, but if we're going to extend the json module, this would probably be the direction to go. > If you mean user-defined types, we're back where we > started, with merely unpacking data running code whose provenance we > don't know. That actually isn't where we started. We started with a serialization format that includes such data as ""c__builtin__\neval\n(c__builtin__\nraw_input\n(S'py> '\ntRtR." (try running pickle.loads on that in Python 2). What I had in mind from the start was something where only whitelisted constructors are used to reconstitute python values from the serialized code. Then we're moved from trusting the input, to trusting the competence of authors of our objects in modules that we imported. In cerealizer there is a global registry of classes that profess to handle input securely. Obviously, they might be wrong, and maybe a user of a serialization library would want to provide a much smaller whitelist. Maybe even the bigger whitelist should be disabled by default, if we really want to be careful, and there should be a security warning in the docs if you try to use the global registry. So for example, there's the following things: # nominally safe; module authors only register if they believe # their deserialization code is safe even with untrusted input my_unserializer.loads("...", whitelist=my_unserializer.PSEUDOSAFE_GLOBAL_REGISTRY) # nominally safe; if not, then a security bug in python my_unserializer.loads("...", whitelist=set()) -- Devin From stephen at xemacs.org Sat Feb 23 16:57:52 2013 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sun, 24 Feb 2013 00:57:52 +0900 Subject: [Python-ideas] Adding a safe alternative to pickle in the standard library In-Reply-To: References: <20130221121135.5a1aabd6@pitrou.net> <512621FF.6070606@trueblade.com> <20130221150049.2731682e@pitrou.net> <51262B08.3030600@trueblade.com> <1E2DA813-A8AF-46CF-B7D4-5A1C1953B781@yahoo.com> <0f836a6d49664d2cb93c97bf496bab90@BLUPR03MB035.namprd03.prod.outlook.com> <87mwuxseoq.fsf@uwakimon.sk.tsukuba.ac.jp> <87ip5ksf9p.fsf@uwakimon.sk.tsukuba.ac.jp> <87bobbryt0.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <87621jrpj3.fsf@uwakimon.sk.tsukuba.ac.jp> Devin Jeanpierre writes: > I believe that the features I'm suggesting meet that criterion (but > see below for discussion of risk). OK, thanks for the discussion. I need to get back to $DAYJOB (at 1am on a Saturday :-P ), but I'll chew over that for a bit before interjecting again. ;-) From ethan at stoneleaf.us Sat Feb 23 19:00:54 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 23 Feb 2013 10:00:54 -0800 Subject: [Python-ideas] PEP for enum library type? In-Reply-To: References: <20130212163102.50dbf570@pitrou.net> <511A76C0.4070005@stoneleaf.us> <20130212144518.3be491c5@anarchist.wooz.org> <511AC786.80807@pearwood.info> Message-ID: <512903D6.1000804@stoneleaf.us> On 02/12/2013 02:56 PM, Guido van Rossum wrote: > Frankly, enums are not that useful in small programs. For large > programs or libraries, and especially for public APIs, the extra cost > of defining the enum shouldn't count against them. I have to disagree. I'm working on converting data from program A to program B and it involves many small scripts for the different input files. Each script ranges from 10 - 50 lines long, and enums -- which I use to access the various columns from the incoming file -- are incredibly useful. > Let's just import Barry's enums into the stdlib. Weren't you at one point unhappy that Barry's enums were not int based? At any rate, non-int based enums would absolutely *not* work for me. for line in open('somefile.txt'): fields = line.split('\t') name = NameCase(fields[int(NAME)]) business = BsnsCase(fields[int(COMPANY)]) street = AddrCase(fields[int(ADDRESS)]) etc., etc., and stab me now with my keyboard! ;) -- ~Ethan~ From masklinn at masklinn.net Sat Feb 23 19:19:18 2013 From: masklinn at masklinn.net (Masklinn) Date: Sat, 23 Feb 2013 19:19:18 +0100 Subject: [Python-ideas] iterable.__unpack__ method In-Reply-To: References: Message-ID: On 2013-02-23, at 12:58 , Nick Coghlan wrote: > On Sat, Feb 23, 2013 at 8:18 PM, Chris Angelico wrote: >> On Sat, Feb 23, 2013 at 8:14 PM, Terry Reedy wrote: >>> It would be much easier, and have much the same effect, if unpacking simply >>> requested the minumum number of items and stopped raising a ValueError if >>> the iterator has more items. No new protocol is needed. Guido rejected this >>> as silently masking errors. >> >> What if it were done explicitly, though? >> >> Currently: >>>>> a,b,c=range(3) >>>>> a,b,c=range(4) >> Traceback (most recent call last): >> File "", line 1, in >> a,b,c=range(4) >> ValueError: too many values to unpack (expected 3) >>>>> a,b,c,*d=range(6) >>>>> a,b,c,* =range(4) >> SyntaxError: invalid syntax >> >> Suppose the last notation were interpreted as "request values for a, >> b, and c, and then ignore the rest". > > It highly unlikely it will ever be interpreted that way, because it > contradicts the way the unnamed star is used in function headers (that > is, to indicate the start of the keyword only arguments *without* > accepting an arbitrary number of positional arguments). Then again, *args in (Python 3) unpacking is already completely incompatible with *args in functions. That does not strike me as much of an issue. > However, Alex's original idea of an "unpack protocol" (distinct from, > but falling back to, the ordinary iterable protocol) definitely has at > least a few points in its favour. > > 1. Iterating over heterogeneous types doesn't really make sense, but > unpacking them does. A separate protocol lets a type support unpacking > assignment without supporting normal iteration. It would also allow preventing unpacking of iterables for which unpacking makes no sense and is most likely an error, such as sets (well there are limited use cases for unpacking a set, but they are few and far between and wrapping the set in an iter() call would allow them). > 2. The protocol could potentially be designed to allow an *iterable* > to be assigned to the star target rather than requiring it to be > unpacked into a tuple. This could be used not only to make unpacking > assignment safe to use with infinite iterators, but also to make it > cheaper with large sequences (through appropriate use of > itertools.islice in a custom container's __unpack__ implementation) > and with virtual sequences like range() objects. And with immutable sequences through plain sharing. In fact, though that's not supported by either existing __unpack__ proposal, it could even allow for mapping unpacking (and more generally named unpacking) by adding something which tends to be missing from python API: names awareness. Altering your proposal by replacing `target_len` by a sequence of the names on LHS would allow implementing __unpack__ for objects or mappings rather than have to use the (quite unwieldy and backwards-looking) operator.itemgetter and operator.attrgetter. > 3. As Alex noted, if a function that previously returned a 2-tuple > wants to start returning a 3-tuple, that's currently a backwards > incompatible change because it will break unpacking assignment. With > an unpack protocol, such a function can return an object that behaves > like a 3-tuple most of the time, but also transparently supports > unpacking assignment to two targets rather than only supporting three. > > I would suggest a different signature for "__unpack__", though, built > around the fact the star target can be used at most once, but in an > arbitrary location: > > def __unpack__(target_len, star_index=None): > """Unpack values into an iterable of the specified length. > > If star_index is not None, the value at that index will be > an iterable representing the remaining contents that were not > unpacked. > """ > ... > > While I'm somewhere between +0 and +1 on the idea myself, there are > some open questions/potential downsides to the idea: > > - this change increases the complexity of the language, by explicitly > separating the concept of heterogeneous unpacking from homogeneous > iteration. Those *are* two distinct concepts though, so the increased > complexity may be justifiable. I'm +1 on these grounds, while the current unpacking "works" it makes for odd limitations or corner-cases. > - for function parameter unpacking, would we allow "*args" to be an > arbitrary iterable rather than promising that it is always a tuple? We > would probably want to do so for consistency with unpacking assignment > under this scheme, which may lead to some confusing error messages > thrown by existing functions which expect it to always be a tuple. I don't think so, they already have a significant semantic split on the meaning of what follows an *args, and I've got the gut feeling that *args being a tuple is a much more common assumption/expectation for functions than for unpacking (especially since *args in unpacking was only introduced in Python 3) > - we would want to document very clearly that it remains possible to > bypass custom unpacking by explicitly calling iter(obj) rather than > using obj directly. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas From zuo at chopin.edu.pl Sat Feb 23 19:41:36 2013 From: zuo at chopin.edu.pl (Jan Kaliszewski) Date: Sat, 23 Feb 2013 19:41:36 +0100 Subject: [Python-ideas] =?utf-8?q?iterable=2E=5F=5Funpack=5F=5F_method?= In-Reply-To: References: Message-ID: <5862b5c0c1859c5b7f0ccb20d028f05d@chopin.edu.pl> On 23.02.2013 19:19, Masklinn wrote: > On 2013-02-23, at 12:58 , Nick Coghlan wrote: [...] >> - for function parameter unpacking, would we allow "*args" to be an >> arbitrary iterable rather than promising that it is always a tuple? >> We >> would probably want to do so for consistency with unpacking >> assignment >> under this scheme, which may lead to some confusing error messages >> thrown by existing functions which expect it to always be a tuple. > > I don't think so, they already have a significant semantic split on > the meaning of what follows an *args, and I've got the gut feeling > that *args being a tuple is a much more common assumption/expectation > for functions than for unpacking (especially since *args in unpacking > was only introduced in Python 3) Especially that for unpacking it is *a list, not a tuple*. and Cheers. *j PS. And would it be really possible to apply the __unpack__-idea to calls? There had not been and still there is no real consistency between unpacking and call argument binding, e.g. def fun(a, b, c, *args): ... fun(1, 2, with_dunder_unpack) # ^ OK: a = 1; b = 2; c, *args = with_dunder_unpack # but what about: def fun(a, b, c, *args): ... fun(1, 2, 3, 4, 5, with_dunder_unpack) # ^ a = 1; b = 2; c = 3; args = ??? I believe these concepts (unpacking and call argument binding) are simply not really parallel in Python. Yes, `a, b, *seq` in unpacking is *somehow* similar -- on the level of a rough intuition (and that is nice) but IMHO it should not stop us from extending unpacking without worrying much about call binding which was, is and must be (in Python) different in many ways. From ethan at stoneleaf.us Sat Feb 23 20:43:29 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 23 Feb 2013 11:43:29 -0800 Subject: [Python-ideas] Yet another enum proposal :) In-Reply-To: <2ec14b4b-2617-4563-8022-eec921b96fd1@googlegroups.com> References: <2ec14b4b-2617-4563-8022-eec921b96fd1@googlegroups.com> Message-ID: <51291BE1.6040604@stoneleaf.us> On 02/22/2013 11:25 AM, Alex Stewart wrote: > So... thoughts? Having thought about this some more, I really like your idea of allowing types such as float to be enums as well. -- ~Ethan~ From ericsnowcurrently at gmail.com Sat Feb 23 21:10:56 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Sat, 23 Feb 2013 13:10:56 -0700 Subject: [Python-ideas] iterable.__unpack__ method In-Reply-To: References: Message-ID: On Sat, Feb 23, 2013 at 4:58 AM, Nick Coghlan wrote: > However, Alex's original idea of an "unpack protocol" (distinct from, > but falling back to, the ordinary iterable protocol) definitely has at > least a few points in its favour. > > 1. Iterating over heterogeneous types doesn't really make sense, but > unpacking them does. A separate protocol lets a type support unpacking > assignment without supporting normal iteration. This is the big one for me. And as masklinn pointed, it works the other way too (keep an iterable from unpacking). > 2. The protocol could potentially be designed to allow an *iterable* > to be assigned to the star target rather than requiring it to be > unpacked into a tuple. This could be used not only to make unpacking > assignment safe to use with infinite iterators, but also to make it > cheaper with large sequences (through appropriate use of > itertools.islice in a custom container's __unpack__ implementation) > and with virtual sequences like range() objects. Cool, though it would be complicated when the star target isn't the last target. > 3. As Alex noted, if a function that previously returned a 2-tuple > wants to start returning a 3-tuple, that's currently a backwards > incompatible change because it will break unpacking assignment. With > an unpack protocol, such a function can return an object that behaves > like a 3-tuple most of the time, but also transparently supports > unpacking assignment to two targets rather than only supporting three. > > I would suggest a different signature for "__unpack__", though, built > around the fact the star target can be used at most once, but in an > arbitrary location: > > def __unpack__(target_len, star_index=None): > """Unpack values into an iterable of the specified length. > > If star_index is not None, the value at that index will be > an iterable representing the remaining contents that were not > unpacked. > """ > ... +1 > > While I'm somewhere between +0 and +1 on the idea myself, there are > some open questions/potential downsides to the idea: > > - this change increases the complexity of the language, by explicitly > separating the concept of heterogeneous unpacking from homogeneous > iteration. Those *are* two distinct concepts though, so the increased > complexity may be justifiable. Agreed. Also, I expect that __unpack__ would be an opt-in API that falls back to iteration. > - for function parameter unpacking, would we allow "*args" to be an > arbitrary iterable rather than promising that it is always a tuple? We > would probably want to do so for consistency with unpacking assignment > under this scheme, which may lead to some confusing error messages > thrown by existing functions which expect it to always be a tuple. This should be handled/proposed separately, though it is dependent on an unpacking protocol. > - we would want to document very clearly that it remains possible to > bypass custom unpacking by explicitly calling iter(obj) rather than > using obj directly. An unpacking protocol would add power to the language. We really don't have a good way to separate unpacking from iteration. Structured record types like namedtuples aren't exactly sequences, and could make use of the new protocol. (I'm not advocating for the change to namedtuple itself.) -eric From tjreedy at udel.edu Sun Feb 24 01:53:30 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 23 Feb 2013 19:53:30 -0500 Subject: [Python-ideas] iterable.__unpack__ method In-Reply-To: References: Message-ID: On 2/23/2013 6:58 AM, Nick Coghlan wrote: > On Sat, Feb 23, 2013 at 8:18 PM, Chris Angelico wrote: >> On Sat, Feb 23, 2013 at 8:14 PM, Terry Reedy wrote: >>> It would be much easier, and have much the same effect, if unpacking simply >>> requested the minumum number of items and stopped raising a ValueError if >>> the iterator has more items. No new protocol is needed. Guido rejected this >>> as silently masking errors. >> >> What if it were done explicitly, though? >> >> Currently: >>>>> a,b,c=range(3) >>>>> a,b,c=range(4) >> Traceback (most recent call last): >> File "", line 1, in >> a,b,c=range(4) >> ValueError: too many values to unpack (expected 3) >>>>> a,b,c,*d=range(6) >>>>> a,b,c,* =range(4) >> SyntaxError: invalid syntax >> >> Suppose the last notation were interpreted as "request values for a, >> b, and c, and then ignore the rest". I think this is fundamentally the right idea. The 'problem' to be solved, such as it is, is that the multiple assignment unpacker, after requesting the number of values it needs, requests one more that it does not need, does not want, and will not use. If it gets it, it says 'too bad', undoes any work it has done, and raises an error. The only point in doing this is to uncover possible bugs. But some people say they want to knowingly provide extra values and have it not be considered a bug. The solution, if there is to be one, and if that extra behavior is not to be completely undone, is to be able to tell the unpacker to skip the extra check. I strongly feel that the right place to do that is on the target side. This fixes the problem in one place rather than requiring that the solution be duplicated everywhere. > It highly unlikely it will ever be interpreted that way, because it > contradicts the way the unnamed star is used in function headers (that > is, to indicate the start of the keyword only arguments *without* > accepting an arbitrary number of positional arguments). If you want to > ignore excess values in unpacking, a double-underscore is the best > currently available option (even though it has the downside of not > working well with infinite iterators or large sequences): > >>> a,b,c,*__ =range(4) If ,* is not acceptible, how about ,** or ,... or ,None or . I rather like 'a, b, c, ... =' as it clearly implies that we are picking and naming the first three values from 3 or more; '...' clearly cannot be an assignment target. > However, Alex's original idea of an "unpack protocol" (distinct from, > but falling back to, the ordinary iterable protocol) definitely has at > least a few points in its favour. I strongly disagree as I think the two major points are wrong. > 1. Iterating over heterogeneous types doesn't really make sense, but > unpacking them does. A separate protocol lets a type support unpacking > assignment without supporting normal iteration. This, repeated in this > - this change increases the complexity of the language, by explicitly > separating the concept of heterogeneous unpacking from homogeneous > iteration. Those *are* two distinct concepts though, so the increased > complexity may be justifiable. attempt to reintroduce 'heterogeneous type' as a fundamental concept in the language, after being removed with the addition of .count and .index as tuple methods. Since I consider the pairing 'heterogeneous type', to be wrong, especially in Python, I consider this to be a serious regression. Let me repeat some of the previous discussion. In Python 3, every object is an instance of class count. At that level, every collection is homogeneous. At other levels, and for particular purposes, any plural collection might be considered heterogeneous. That is a function of the objects or values in the collection, and not of the collection class itself. So I claim that 'heterogeneous' has not much meaning as an absolute attribute of a 'type'. In any assignment, targets (mostly commonly names) are untyped, or have type 'object' -- take your pick. So iterable (of objects) is the appropriate source. In any case, 'iteration' and 'unpacking' both mean 'accessing the items of a collection one at a time, as individual items rather than as part of a collection'. I do not see any important distinction at all and no justification for complexifying the language again. > 3. As Alex noted, if a function that previously returned a 2-tuple > wants to start returning a 3-tuple, that's currently a backwards > incompatible change because it will break unpacking assignment. With > an unpack protocol, such a function can return an object that behaves > like a 3-tuple most of the time, but also transparently supports > unpacking assignment to two targets rather than only supporting three. This seems to claiming that it is sensible to change the return type of a function to a somewhat incompatible return type. (I am obviously including fixed tuple length in 'return type', as would be explicit in other languages). I believe many/most/all design and interface experts would disagree and would say it would be better to give the new function a new name. The statement "that's currently a backwards incompatible change because it will break unpacking assignment." is a gross understatement that glosses over the fact that returning a 3-tuple instead of a 2-tuple will breaks lots of things. Just two examples: def foo(): return 1,2 def bar(): return tuple('abc') foobar = foo() + bar() oof = reversed(foo()) Changing foo to return 1,2,'x' will mess up both foobar and oof for most uses. -- > 2. The protocol could potentially be designed to allow an *iterable* > to be assigned to the star target rather than requiring it to be > unpacked into a tuple. This could be used not only to make unpacking > assignment safe to use with infinite iterators, but also to make it > cheaper with large sequences (through appropriate use of > itertools.islice in a custom container's __unpack__ implementation) > and with virtual sequences like range() objects. The problem with *x, that it is inefficient, not applicable to infinite iterators, and that assigning the iterable directly to x when *x is in final position is more likely what one wants anyway, is a different issue from the unwanted bug check and exception. Again, I think the solution is an explicit notation on the target side. Perhaps '**x' or 'x*' or something based on _. If you do not like any of those, suggest another. -- Terry Jan Reedy From rosuav at gmail.com Sun Feb 24 02:36:59 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 24 Feb 2013 12:36:59 +1100 Subject: [Python-ideas] iterable.__unpack__ method In-Reply-To: References: Message-ID: On Sun, Feb 24, 2013 at 11:53 AM, Terry Reedy wrote: > If ,* is not acceptible, how about ,** or ,... or ,None or . > I rather like 'a, b, c, ... =' as it clearly implies that we are picking and > naming the first three values from 3 or more; '...' clearly cannot be an > assignment target. I like that. It avoids the confusion with augmented assignment that ,* has (yes, you can avoid the *syntactic* confusion by putting spaces around the equals sign, but it's still visually similar). The only problem might be in documentation, where it might be taken to mean "a, b, c, d, e, and as many more variables as you want", eg indicating that tuple unpacking works with any number of targets. ChrisA From ncoghlan at gmail.com Sun Feb 24 03:12:13 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 24 Feb 2013 12:12:13 +1000 Subject: [Python-ideas] iterable.__unpack__ method In-Reply-To: References: Message-ID: I definitely like Terry's idea of using ", ..." to say "and ignore the rest". Simple, elegant and can be restricted to the last element so it works with infinite iterators and large sequences. It also allows incremental unpacking of ordinary iterators. I still like the idea of an unpacking protocol as well, but the above would cover a lot of use cases without needing a new protocol, and thus should be considered first (since a new protocol is a higher impact change than Terry's syntax suggestion). Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Sun Feb 24 04:01:45 2013 From: python at mrabarnett.plus.com (MRAB) Date: Sun, 24 Feb 2013 03:01:45 +0000 Subject: [Python-ideas] iterable.__unpack__ method In-Reply-To: References: Message-ID: <51298299.9050109@mrabarnett.plus.com> On 2013-02-24 00:53, Terry Reedy wrote: [snip] > If ,* is not acceptible, how about ,** or ,... or ,None or pick>. I rather like 'a, b, c, ... =' as it clearly implies that we are > picking and naming the first three values from 3 or more; '...' clearly > cannot be an assignment target. > [snip] ,** reminds me too much of dict packing/unpacking. -1 ,None looks odd because it looks like you're binding to None. -1 ,... looks good to me. +1 From ncoghlan at gmail.com Sun Feb 24 06:40:49 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 24 Feb 2013 15:40:49 +1000 Subject: [Python-ideas] iterable.__unpack__ method In-Reply-To: References: Message-ID: On Sun, Feb 24, 2013 at 12:12 PM, Nick Coghlan wrote: > I definitely like Terry's idea of using ", ..." to say "and ignore the > rest". Simple, elegant and can be restricted to the last element so it works > with infinite iterators and large sequences. It also allows incremental > unpacking of ordinary iterators. I want to expand on this a bit now I'm back on a real computer, since it wasn't immediately obvious to me how well the ", ..." proposal supports incremental unpacking, but it became clear once I thought of this simple example: iterargs = iter(args) command, ... = iterargs # Grab the first element, leave the rest in the iterator commands[command](*iterargs) # Pass the rest to the command Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From guido at python.org Sun Feb 24 07:24:50 2013 From: guido at python.org (Guido van Rossum) Date: Sat, 23 Feb 2013 22:24:50 -0800 Subject: [Python-ideas] iterable.__unpack__ method In-Reply-To: References: Message-ID: But is that really so much better than command = next(iterargs) # etc. ? And to me the ... Looks too much like something that consumes the rest, rather than leaving it. --Guido (not on a real keyboard) On Saturday, February 23, 2013, Nick Coghlan wrote: > On Sun, Feb 24, 2013 at 12:12 PM, Nick Coghlan > > wrote: > > I definitely like Terry's idea of using ", ..." to say "and ignore the > > rest". Simple, elegant and can be restricted to the last element so it > works > > with infinite iterators and large sequences. It also allows incremental > > unpacking of ordinary iterators. > > I want to expand on this a bit now I'm back on a real computer, since > it wasn't immediately obvious to me how well the ", ..." proposal > supports incremental unpacking, but it became clear once I thought of > this simple example: > > iterargs = iter(args) > command, ... = iterargs # Grab the first element, leave the rest > in the iterator > commands[command](*iterargs) # Pass the rest to the command > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, > Australia > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Feb 24 08:38:39 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 24 Feb 2013 17:38:39 +1000 Subject: [Python-ideas] iterable.__unpack__ method In-Reply-To: References: Message-ID: On Sun, Feb 24, 2013 at 4:24 PM, Guido van Rossum wrote: > But is that really so much better than > > command = next(iterargs) > # etc. > > ? For a single item, using next() instead is OK, the problem is that itertools islice is relatively ugly and unintuitive, and requires you to manually count the number of targets. The next + islice combination also creates a discontinuity between the way you handle getting just the first item, versus getting multiple items. Status quo, 1 item: iterargs = iter(args) command = next(iterargs) # Grab the first, leave the rest commands[command](*iterargs) # Pass the rest to the command Status quo, 2 items (etc): from itertools import islice iterargs = iter(args) command, subcommand = islice(iterargs, 2) # Grab the first two, leave the rest commands[command][subcommand](*iterargs) # Pass the rest to the subcommand Proposal, 1 item: iterargs = iter(args) command, ... = iterargs # Grab the first, leave the rest commands[command](*iterargs) # Pass the rest to the command Proposal, 2 items (etc): iterargs = iter(args) command, subcommand, ... = iterargs # Grab the first two, leave the rest commands[command][subcommand](*iterargs) # Pass the rest to the subcommand > And to me the ... Looks too much like something that consumes the rest, > rather than leaving it. As far as the possible interpretation being "consume and discard the rest" goes, I think it's one of those cases where once you know which it is, it won't be hard to remember. It's certainly something we could continue to live without, but I do think it's a nice way to expand the unpacking syntax to cover additional use cases far more elegantly than the current reliance on a combination of next and itertools.islice. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From bruce at leapyear.org Sun Feb 24 09:04:57 2013 From: bruce at leapyear.org (Bruce Leban) Date: Sun, 24 Feb 2013 00:04:57 -0800 Subject: [Python-ideas] iterable.__unpack__ method In-Reply-To: References: Message-ID: On Sat, Feb 23, 2013 at 10:24 PM, Guido van Rossum wrote: > But is that really so much better than > > command = next(iterargs) > # etc. > > ? > > And to me the ... Looks too much like something that consumes the rest, > rather than leaving it. > > --Guido (not on a real keyboard) > And hypothetically a, b, c = next(iterargs, count=3) or a, b, c = (next * 3)(iterargs) --- Bruce -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sun Feb 24 10:08:57 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 24 Feb 2013 20:08:57 +1100 Subject: [Python-ideas] iterable.__unpack__ method In-Reply-To: References: Message-ID: <5129D8A9.6000000@pearwood.info> On 24/02/13 19:04, Bruce Leban wrote: > And hypothetically [...] > a, b, c = (next * 3)(iterargs) That looks like, well, I'm not sure what it looks like. Haskell maybe? But not Python. -- Steven From larry at hastings.org Sun Feb 24 12:25:56 2013 From: larry at hastings.org (Larry Hastings) Date: Sun, 24 Feb 2013 03:25:56 -0800 Subject: [Python-ideas] iterable.__unpack__ method In-Reply-To: References: Message-ID: <5129F8C4.4080307@hastings.org> On 02/23/2013 11:38 PM, Nick Coghlan wrote: > Status quo, 2 items (etc): > > from itertools import islice > iterargs = iter(args) > command, subcommand = islice(iterargs, 2) # Grab the first two, > leave the rest > commands[command][subcommand](*iterargs) # Pass the rest to the subcommand Or command, subcommand = next(iterargs), next(iterargs) or command = next(iterargs) subcommand = next(iterargs) These don't require the manual counting, nor do they feature the "discontinuity" you mentioned. FWIW I don't think this problem is bad enough, nor this idiom used frequently enough, to warrant new syntax. But I've been wrong before. I was musing on this topic and came up with what is probably a terrible alternate approach. What if there was some way for the RHS to know what the LHS was asking for in a destructuring assignment? Something like def __next_n__(self, n): "Returns a tuple of the next n items from this iterator." and if the object doesn't provide __next_n__ you fall back to the current explicit PyIter_Next() calls and failing if there aren't enough / there are too many. If we had this, we could make the second argument to islice() optional; it could infer how many items you wanted. (I don't think the "a, b, *c, d = rhs" form is relevant here as you'd always consume everything from rhs.) I'm not sure how bad an idea this is. But complicating the iterator protocol for this marginal problem seems like a pretty bad idea. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sun Feb 24 13:59:20 2013 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 24 Feb 2013 23:59:20 +1100 Subject: [Python-ideas] iterable.__unpack__ method In-Reply-To: <5129F8C4.4080307@hastings.org> References: <5129F8C4.4080307@hastings.org> Message-ID: On Sun, Feb 24, 2013 at 10:25 PM, Larry Hastings wrote: > Or > > command, subcommand = next(iterargs), next(iterargs) Err.... is there a language guarantee of the order of evaluation in a tuple, or is this just a "CPython happens to evaluate independent expressions left-to-right"? This is totally broken if the next() calls could be done in either order. ChrisA From steve at pearwood.info Sun Feb 24 15:16:03 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 25 Feb 2013 01:16:03 +1100 Subject: [Python-ideas] iterable.__unpack__ method In-Reply-To: References: <5129F8C4.4080307@hastings.org> Message-ID: <512A20A3.1020006@pearwood.info> On 24/02/13 23:59, Chris Angelico wrote: > On Sun, Feb 24, 2013 at 10:25 PM, Larry Hastings wrote: >> Or >> >> command, subcommand = next(iterargs), next(iterargs) > > Err.... is there a language guarantee of the order of evaluation in a > tuple, or is this just a "CPython happens to evaluate independent > expressions left-to-right"? This is totally broken if the next() calls > could be done in either order. It's a language guarantee. http://docs.python.org/2/reference/expressions.html#evaluation-order -- Steven From rosuav at gmail.com Sun Feb 24 15:52:48 2013 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 25 Feb 2013 01:52:48 +1100 Subject: [Python-ideas] iterable.__unpack__ method In-Reply-To: <512A20A3.1020006@pearwood.info> References: <5129F8C4.4080307@hastings.org> <512A20A3.1020006@pearwood.info> Message-ID: On Mon, Feb 25, 2013 at 1:16 AM, Steven D'Aprano wrote: > On 24/02/13 23:59, Chris Angelico wrote: >> >> On Sun, Feb 24, 2013 at 10:25 PM, Larry Hastings >> wrote: >>> >>> Or >>> >>> command, subcommand = next(iterargs), next(iterargs) >> >> >> Err.... is there a language guarantee of the order of evaluation in a >> tuple, or is this just a "CPython happens to evaluate independent >> expressions left-to-right"? This is totally broken if the next() calls >> could be done in either order. > > It's a language guarantee. > > http://docs.python.org/2/reference/expressions.html#evaluation-order Ah, so it is. My bad, sorry! In that case, sure, this works. It still violates DRY though, naming the iterable twice and relying on the reader noticing that that means "take two off this one". But that's a much weaker concern. ChrisA From vito.detullio at gmail.com Sun Feb 24 16:45:16 2013 From: vito.detullio at gmail.com (Vito De Tullio) Date: Sun, 24 Feb 2013 16:45:16 +0100 Subject: [Python-ideas] iterable.__unpack__ method References: <5129F8C4.4080307@hastings.org> <512A20A3.1020006@pearwood.info> Message-ID: Chris Angelico wrote: >>>> command, subcommand = next(iterargs), next(iterargs) > Ah, so it is. My bad, sorry! In that case, sure, this works. It still > violates DRY though, naming the iterable twice and relying on the > reader noticing that that means "take two off this one". well, about the DRY it's the same, but I found the "old stupid" command = next(iterargs) subcommand = next(iterargs) mostly more explicit in the "take two off this one" maybe it's... too much explicit (and it's not a single expression!!! anathema!) but I found slightly more readable -- ZeD From guido at python.org Sun Feb 24 17:07:19 2013 From: guido at python.org (Guido van Rossum) Date: Sun, 24 Feb 2013 08:07:19 -0800 Subject: [Python-ideas] iterable.__unpack__ method In-Reply-To: References: <5129F8C4.4080307@hastings.org> <512A20A3.1020006@pearwood.info> Message-ID: DRY is not the only principle to consider here. On Sunday, February 24, 2013, Chris Angelico wrote: > On Mon, Feb 25, 2013 at 1:16 AM, Steven D'Aprano > > wrote: > > On 24/02/13 23:59, Chris Angelico wrote: > >> > >> On Sun, Feb 24, 2013 at 10:25 PM, Larry Hastings > > > >> wrote: > >>> > >>> Or > >>> > >>> command, subcommand = next(iterargs), next(iterargs) > >> > >> > >> Err.... is there a language guarantee of the order of evaluation in a > >> tuple, or is this just a "CPython happens to evaluate independent > >> expressions left-to-right"? This is totally broken if the next() calls > >> could be done in either order. > > > > It's a language guarantee. > > > > http://docs.python.org/2/reference/expressions.html#evaluation-order > > Ah, so it is. My bad, sorry! In that case, sure, this works. It still > violates DRY though, naming the iterable twice and relying on the > reader noticing that that means "take two off this one". But that's a > much weaker concern. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From oscar.j.benjamin at gmail.com Sun Feb 24 17:42:20 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Sun, 24 Feb 2013 16:42:20 +0000 Subject: [Python-ideas] iterable.__unpack__ method In-Reply-To: References: Message-ID: On 24 February 2013 06:24, Guido van Rossum wrote: > On Saturday, February 23, 2013, Nick Coghlan wrote: >> On Sun, Feb 24, 2013 at 12:12 PM, Nick Coghlan wrote: >> > I definitely like Terry's idea of using ", ..." to say "and ignore the >> > rest". Simple, elegant and can be restricted to the last element so it >> > works >> > with infinite iterators and large sequences. It also allows incremental >> > unpacking of ordinary iterators. >> >> I want to expand on this a bit now I'm back on a real computer, since >> it wasn't immediately obvious to me how well the ", ..." proposal >> supports incremental unpacking, but it became clear once I thought of >> this simple example: >> >> iterargs = iter(args) >> command, ... = iterargs # Grab the first element, leave the rest >> in the iterator >> commands[command](*iterargs) # Pass the rest to the command > > But is that really so much better than > > command = next(iterargs) > # etc. Calls to next() need to be wrapped in try/except StopIteration (it's a bad idea to leak these exceptions). So then it becomes: try: command = next(iterator) except StopIteration: raise ValueError('need more than 0 values to unpack') Presumably with this proposal command, ... = iterator would raise a ValueError if the iterator is empty. Oscar From zuo at chopin.edu.pl Sun Feb 24 21:14:28 2013 From: zuo at chopin.edu.pl (Jan Kaliszewski) Date: Sun, 24 Feb 2013 21:14:28 +0100 Subject: [Python-ideas] =?utf-8?q?iterable=2E=5F=5Funpack=5F=5F_method?= In-Reply-To: References: Message-ID: <44528db0ff81ea147dfc71d97e472798@chopin.edu.pl> 24.02.2013 17:42, Oscar Benjamin wrote: > Calls to next() need to be wrapped in try/except StopIteration (it's > a > bad idea to leak these exceptions). So then it becomes: > > try: > command = next(iterator) > except StopIteration: > raise ValueError('need more than 0 values to unpack') Not necessarily. You can use: command = next(iterator, None) Cheers. *j From python at mrabarnett.plus.com Sun Feb 24 21:34:47 2013 From: python at mrabarnett.plus.com (MRAB) Date: Sun, 24 Feb 2013 20:34:47 +0000 Subject: [Python-ideas] iterable.__unpack__ method In-Reply-To: <44528db0ff81ea147dfc71d97e472798@chopin.edu.pl> References: <44528db0ff81ea147dfc71d97e472798@chopin.edu.pl> Message-ID: <512A7967.1060709@mrabarnett.plus.com> On 2013-02-24 20:14, Jan Kaliszewski wrote: > 24.02.2013 17:42, Oscar Benjamin wrote: > >> Calls to next() need to be wrapped in try/except StopIteration (it's >> a >> bad idea to leak these exceptions). So then it becomes: >> >> try: >> command = next(iterator) >> except StopIteration: >> raise ValueError('need more than 0 values to unpack') > > Not necessarily. You can use: > > command = next(iterator, None) > That's not the same thing. Normally when unpacking, it raises ValueError if there are too few items. That's what we want to preserve. From foogod at gmail.com Sun Feb 24 23:07:24 2013 From: foogod at gmail.com (Alex Stewart) Date: Sun, 24 Feb 2013 14:07:24 -0800 (PST) Subject: [Python-ideas] iterable.__unpack__ method In-Reply-To: <512A7967.1060709@mrabarnett.plus.com> References: <44528db0ff81ea147dfc71d97e472798@chopin.edu.pl> <512A7967.1060709@mrabarnett.plus.com> Message-ID: It seems to me this thread has actually split into discussing several different, fairly independent issues. ==== Unpacking protocol First, there is the question I originally raised about separating the "unpacking protocol" from the "iterator protocol". I would like to clarify a couple of points on this: I proposed __unpack__ the way I did specifically for a few reasons: 1. It is a fairly small change to the language (it does not require any new syntax, does not change the behavior of any current code, and can be completely ignored by programmers who don't care about the issues it addresses) 2. There are (in my opinion) already pretty good parallels with this sort of change that have already been implemented in Python (for example, separating the concept of "iterable" from the concept of "indexable"). This seemed to be a fairly natural and easy-to-explain extension of the same sort of philosophies that Python has already largely adopted in other contexts. 3. It is consistent with the general idea in Python that an object can choose to emulate whichever behaviors of other objects make sense, and not emulate other ones that do not make sense. In my opinion, "iterating", and "unpacking" are two related-but-not-the-same language concepts, which currently are unnecessarily entangled. 4. It would help significantly with several types of problems that do not currently have great solutions. People have proposed various alternatives that would help with one or two of these, but none of the proposed alternatives deal with all of them (or really even most of them). In summary, some of the other proposals have other useful features, and might be worth considering, but many of them seem far more of a stretch than my original proposal, and I think there are other good reasons why the original proposal is still worth looking at on its own merits. Also, regarding the method signature "(min_count, max_count)" vs "(target_len, star_index)". I specifically discarded the latter in favor of the former when I was putting together the proposal because (a) the "star_index" way does not really gain us anything useful over the "min/max" way, (b) "min/max" seemed a bit more future-proof (if, in the future, for example, somebody came up with an extended-extended unpack syntax which allowed for (for example) optional parameters, or the ability to specify an upper-bound on the *extra stuff, it would not require any change to the __unpack__ protocol to support it), (c) "min/max" is less about the particular implementation and more about conceptually what it means to unpack something, which seemed more appropriate when defining the concept of an "unpack protocol", and (d) this also means that it could conceivably be called in other cases (or directly by Python code that knew what it was doing) to produce more sophisticated results for some objects. Ultimately, I could live with "target_len, star_index", but I still think "min_count, max_count" is a better way to do things, and I'd be curious whether anybody can come up with an argument why "target_len, star_index" would be actually more useful or preferable in any way.. ==== Partial unpacking Second, there is the question of allowing explicit (or implicit) "partial unpacking" (i.e. allowing the programmer to say "I only want to unpack the first n items, and leave any others in the iterator) I agree with the general sentiment that doing this implicitly (i.e. just not checking for "too many values to unpack") is silently masking errors, which is a bad idea, and anyway it would potentially cause problems with all kinds of existing code, and we just shouldn't go down that road. I do think it might be nice to have a way to specify in the syntax to just leave any extra data un-iterated, but to be honest I haven't really felt any particular warm-fuzzies about any of the proposed syntaxes thus far. (The one I like best is Terry's "a, b, c, ... = foo", though.) However, this is really fundamentally a completely different question, which addresses a different use-case, than the unpacking protocol question, and should not be confused as addressing the same issues (because it doesn't). Specifically, lots of folks seem to be ignoring the fact that any "unpacking protocol" (whether it's the implicit one we have now, or a more explicit alternative like __unpack__) has two sides to it (the provider and the consumer). These two sides are not always written by the same person or expecting the same things (for example, when unpacking an object which was passed into a routine or returned by some other code). There are times when the person writing the assignment statement does know "I only want these items and not the rest", but there are also times when the object itself potentially knows better "only these items make sense in this context". I would emphasize that there are currently ways (even if they're less than ideal) for the former to be done (i.e. calling next() or using itertools), but there is currently no way to do the latter at all (because the object is currently not given any way to get any information about the context it's being used in). We can make the former case more convenient if we want to, but it really doesn't fix the other side of the problem at all. ==== Iterator end-testing The third issue that has been brought up is the slightly-related issue of how unpacking currently attempts to consume an extra item from an iterator, just to check for length. This is, in my opinion, a valid flaw which is worth discussing, but it is actually not a flaw with unpacking, or really related in any way to unpacking except by happenstance. The real issue here is not in how unpacking is done but a limitation in the iteration protocol itself. The problem is that there is no standard way to query an iterator to find out if it has more data available without automatically consuming that next data element in the process. In my opinion, if we want to deal with this issue properly, we should not be trying to hack around it in the single case of unpacking but instead fix the larger core problem by adding some way to check for the end of an iterator without side-effects (and then of course change unpacking to use that instead). --Alex -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Mon Feb 25 01:57:51 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 24 Feb 2013 19:57:51 -0500 Subject: [Python-ideas] iterable.__unpack__ method In-Reply-To: References: Message-ID: On 2/24/2013 1:24 AM, Guido van Rossum wrote: > But is that really so much better than > > command = next(iterargs) > # etc. > > ? > > And to me the ... Looks too much like something that consumes the rest, > rather than leaving it. I considered $ for $top, but I was sure you would not want to waste $ in this. In my original suggestion, I also suggested None as in a, b, c, None = source as another possibly to signal "I want source iteration to stop even if there are more items, and I don't want an exception raised even if there are, because I think I know what I am doing and I do not want it to be considered a bug." Any other short spelling/signal would be fine, too. -- Terry Jan Reedy From random832 at fastmail.us Mon Feb 25 04:09:28 2013 From: random832 at fastmail.us (Random832) Date: Sun, 24 Feb 2013 22:09:28 -0500 Subject: [Python-ideas] iterable.__unpack__ method In-Reply-To: References: Message-ID: <512AD5E8.4030204@fastmail.us> On 02/22/2013 02:33 PM, Alex Stewart wrote: > > In the case of simple unpacking ("a, b, c = foo"), min_count and > max_count will be the same value (3). In the case of extended > unpacking ("a, b, c, *d = foo"), min_count would be the smallest > length that will not cause an unpacking exception (3, in this case), > and max_count would be None. > > 1. It would allow for "smart" unpackable objects (such as __ in my > enum example), which would "just work" no matter how many values > they're required to produce. > If you want "smart" unpackable objects, why not allow d in your *d example to be something other than a list? Like, if you have a,*b = range(1,5); b could literally be range(2,5). -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Mon Feb 25 04:41:48 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 24 Feb 2013 22:41:48 -0500 Subject: [Python-ideas] Add lookahead iterator (peeker) to itertools Message-ID: An iterator iter represents the remainder of some collection, concrete or not, finite or not. If the remainder is not empty, its .__next__ method selects one object, mutates iter to represent the reduced remainder (now possibly empty), and returns the one item. At various times, people have asked for the ability to determine whether an iterator is exhausted, whether a next() call will return or raise. If it will return, people have also asked for the ability to peek at what the return would be, without disturbing what the return will be. For instance, on the 'iterable.__unpack__ method' Alex Stewart today wrote: > The problem is that there is no standard way to query an iterator > to find out if it has more data available without automatically > consuming that next data element in the rocess. It turns out that there is a solution that gives the ability to both test emptiness (in the standard way) and peek ahead, without modifying the iterator protocol. It merely require a wrapper iterator, much like the ones in itertools. I have posted one before and give my current version below. Does anyone else this should be added to itertools? It seems to not be completely obvious to everyone, is more complex that some of the existing itertools, and cannot be composed from them either. (Nor can it be written as a generator function.) Any of the names can be changed. Perhaps the class should be 'peek' and the lookahead object something else. The sentinel should be read-only if possible. I considered whether the peek object should be read-only, but someone would say that they *want* be able to replace the next object to be yielded. Peeking into an exhausted iterable could raise instead of returning the sentinel, but I don't know if that would be more useful. ---------------- class lookahead(): "Wrap iterator with lookahead to both peek and test exhausted" _NONE = object() def __init__(self, iterable): self._it = iter(iterable) self._set_peek() def __iter__(self): return self def __next__(self): ret = self.peek self._set_peek() return ret def _set_peek(self): try: self.peek = next(self._it) except StopIteration: self.peek = self._NONE def __bool__(self): return self.peek is not self._NONE def test_lookahead(): it = lookahead('abc') while it: a = it.peek b = next(it) print('next:', b, '; is peek:', a is b ) test_lookahead() -------------------- >>> next: a ; is peek: True next: b ; is peek: True next: c ; is peek: True -- Terry Jan Reedy From tjreedy at udel.edu Mon Feb 25 04:43:58 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 24 Feb 2013 22:43:58 -0500 Subject: [Python-ideas] iterable.__unpack__ method In-Reply-To: References: <44528db0ff81ea147dfc71d97e472798@chopin.edu.pl> <512A7967.1060709@mrabarnett.plus.com> Message-ID: On 2/24/2013 5:07 PM, Alex Stewart wrote: > ==== Unpacking protocol > > First, there is the question I originally raised about separating the > "unpacking protocol" from the "iterator protocol". I would like to > clarify a couple of points on this: > > I proposed __unpack__ the way I did specifically for a few reasons: > > 1. It is a fairly small change to the language To me, taking something simple and elegant and complexifying it, to little benefit I can see other than to solve a problem in the wrong place, in no small matter. Certainly, adding a new special method to some category of objects has not effect unless changes are made to the interpreter elsewhere to make automatic use of that method. Otherwise, you could just define a normal method to be used by user code. > 2. There are (in my opinion) already pretty good parallels with this > sort of change that have already been implemented in Python (for > example, separating the concept of "iterable" from the concept of > "indexable"). The related but distinct concepts of sequential access and random access are basic to computation and were *always* distinct in Python. For loop statements were always the way to sequentially access objects in a collection (by default, all of them). It happens that the original *implementation* of sequential access, use by for loops, was by means of pseudo random access. This worked ok for randomly accessible sequences but was clumsy for anything else. (Sequential access requires memory, random access does not.) The addition of __iter__ along with __next__ added more benefits than __next__ alone would have. > 3. It is consistent with the general idea in Python that an object can > choose to emulate whichever behaviors of other objects make sense, > and not emulate other ones that do not make sense. In my opinion, > "iterating", and "unpacking" are two related-but-not-the-same > language concepts, which currently are unnecessarily entangled. I do not see that at all. As near as I can tell, both mean 'sequential access'. You are that one that seems to me to be entangling 'bind objects to targets one at a time' with 'provide objects one at a time'. The point of both old and new iterator protocols was and is to decouple (disentangle) consumers from providers. At one time, target-list binding *was* entangled -- with tuples -- because it did not use the iterator protocol that for statement did. I admit I do not understand your __unpack__ proposal, since it seemed vague and incomplete to me. But until I see your conceptual distinction, the details do not matter to me. > 4. It would help significantly with several types of problems that do > not currently have great solutions. I obviously do not see the same pile of problems that would justify a new special method. ... > no way to query an iterator to find out if it has more data available See how in my separate post: "Add lookahead iterator (peeker) to itertools" -- Terry Jan Reedy From larry at hastings.org Mon Feb 25 05:06:39 2013 From: larry at hastings.org (Larry Hastings) Date: Sun, 24 Feb 2013 20:06:39 -0800 Subject: [Python-ideas] iterable.__unpack__ method In-Reply-To: References: Message-ID: <512AE34F.9020902@hastings.org> On 02/24/2013 04:57 PM, Terry Reedy wrote: > On 2/24/2013 1:24 AM, Guido van Rossum wrote: >> But is that really so much better than >> >> command = next(iterargs) >> # etc. >> >> ? >> >> And to me the ... Looks too much like something that consumes the rest, >> rather than leaving it. > > I considered $ for $top, but I was sure you would not want to waste $ > in this. If we are seriously considering this language addition, may I counter-propose the syntax a, b *= iterable because star already has something vaguely to do with unpacking, something something. *handwave* //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From wolfgang.maier at biologie.uni-freiburg.de Mon Feb 25 10:51:17 2013 From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier) Date: Mon, 25 Feb 2013 09:51:17 +0000 (UTC) Subject: [Python-ideas] Add lookahead iterator (peeker) to itertools References: Message-ID: Terry Reedy writes: > class lookahead(): > "Wrap iterator with lookahead to both peek and test exhausted" ... +1 That's a nice tool that I'd love to have in itertools. Cheers, Wolfgang From solipsis at pitrou.net Mon Feb 25 11:07:17 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 25 Feb 2013 11:07:17 +0100 Subject: [Python-ideas] Add lookahead iterator (peeker) to itertools References: Message-ID: <20130225110717.232ce082@pitrou.net> Le Sun, 24 Feb 2013 22:41:48 -0500, Terry Reedy a ?crit : > > def test_lookahead(): > it = lookahead('abc') > while it: > a = it.peek > b = next(it) > print('next:', b, '; is peek:', a is b ) def test_lookahead(): it = iter('abc') while True: it, peeking = itertools.tee(it) try: a = next(peeking) except StopIteration: break b = next(it) print('next:', b, '; is peek:', a is b ) Regards Antoine. From p.f.moore at gmail.com Mon Feb 25 11:08:42 2013 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 25 Feb 2013 10:08:42 +0000 Subject: [Python-ideas] Add lookahead iterator (peeker) to itertools In-Reply-To: References: Message-ID: On 25 February 2013 09:51, Wolfgang Maier wrote: > Terry Reedy writes: > >> class lookahead(): >> "Wrap iterator with lookahead to both peek and test exhausted" > ... > > +1 That's a nice tool that I'd love to have in itertools. It's not a bad idea, but I don't like the fact that as written it turns a finite iterator into an infinite one (returning an endless sequence of sentinels after the underlying iterator is exhausted). That seems to me to be very prone to errors if you don't keep it very clear whether you're dealing with iterators or lookahead-wrapped iterators: >>> from lookahead import lookahead >>> it = lookahead(range(3)) >>> list(it) The problem is that once you wrap an iterator with lookahead() you can't use the underlying iterator directly any more without losing data, as you've consumed a value from it. So you have to use the wrapped version, and the differing behaviour means you need to write your code differently. I'd prefer it if lookahead(it) behaved exactly like it, except that it had a new peek() method for getting the lookahead value, and maybe a finished() method to tell if next() will raise StopIteration or not. Bikeshed away over what should happen if peek() is called when finished() is true :-) (Disclaimer: I have no real-world use cases for this feature, so the comments above are largely theoretical objections...) Paul. From wolfgang.maier at biologie.uni-freiburg.de Mon Feb 25 11:32:34 2013 From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier) Date: Mon, 25 Feb 2013 10:32:34 +0000 (UTC) Subject: [Python-ideas] =?utf-8?q?iterable=2E=5F=5Funpack=5F=5F_method?= References: Message-ID: Nick Coghlan writes: > Status quo, 2 items (etc): > > from itertools import islice > iterargs = iter(args) > command, subcommand = islice(iterargs, 2) # Grab the first two, > leave the rest > commands[command][subcommand](*iterargs) # Pass the rest to the subcommand When it comes to getting multiple items from an iterator, I prefer wrapping things in my own generator function: def x_iter(iterator,n): """Return n items from iterator.""" i=[iterator]*n while True: try: result=[next(i[0])] except StopIteration: # iterator exhausted immediately, end the generator break for e in i[1:]: try: result.append(next(e)) except StopIteration: # iterator exhausted after returning at least one item, but before returning n raise ValueError("only %d value(s) left in iterator, expected %d" % (len(result),n)) yield result Compared to islice, this has the advantage of working properly in for loops: >>> it=iter(range(1,11)) >>> for c,d in x_iter(it,2): print(c,d) 1 2 3 4 5 6 7 8 9 10 Maybe one could improve itertools.islice accordingly?? > Proposal, 2 items (etc): > > iterargs = iter(args) > command, subcommand, ... = iterargs # Grab the first two, leave the rest > commands[command][subcommand](*iterargs) # Pass the rest to the subcommand > +1 for this. I think it's very readable. I think it should raise differently though depending on whether iterargs is exhausted right away (StopIteration) or during unpacking (ValueError). Best, Wolfgang From mark.hackett at metoffice.gov.uk Mon Feb 25 12:29:02 2013 From: mark.hackett at metoffice.gov.uk (Mark Hackett) Date: Mon, 25 Feb 2013 11:29:02 +0000 Subject: [Python-ideas] Adding a safe alternative to pickle in the standard library In-Reply-To: References: Message-ID: <201302251129.02804.mark.hackett@metoffice.gov.uk> On Friday 22 Feb 2013, Devin Jeanpierre wrote: > On Fri, Feb 22, 2013 at 12:41 PM, Andrew Barnert wrote: > > The difference is that json.loads is just deserialize(foo), which > > pickle.loads inherently has some eval mixed in. > > > > That's why I think for most use cases, the answer is making json easier > > to extend, not making pickle easier to secure. > > My original suggestion was to add a third thing, such as cerealizer, Maybe we can ask Wil WHEATon... Apologies... From storchaka at gmail.com Mon Feb 25 12:58:59 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 25 Feb 2013 13:58:59 +0200 Subject: [Python-ideas] Add lookahead iterator (peeker) to itertools In-Reply-To: <20130225110717.232ce082@pitrou.net> References: <20130225110717.232ce082@pitrou.net> Message-ID: On 25.02.13 12:07, Antoine Pitrou wrote: > Le Sun, 24 Feb 2013 22:41:48 -0500, > Terry Reedy a ?crit : >> >> def test_lookahead(): >> it = lookahead('abc') >> while it: >> a = it.peek >> b = next(it) >> print('next:', b, '; is peek:', a is b ) > > def test_lookahead(): > it = iter('abc') > while True: > it, peeking = itertools.tee(it) This should be outside a loop. > try: > a = next(peeking) > except StopIteration: > break > b = next(it) > print('next:', b, '; is peek:', a is b ) From masklinn at masklinn.net Mon Feb 25 13:02:30 2013 From: masklinn at masklinn.net (Masklinn) Date: Mon, 25 Feb 2013 13:02:30 +0100 Subject: [Python-ideas] Add lookahead iterator (peeker) to itertools In-Reply-To: References: Message-ID: On 2013-02-25, at 11:08 , Paul Moore wrote: > >>>> from lookahead import lookahead >>>> it = lookahead(range(3)) >>>> list(it) > > > The problem is that once you wrap an iterator with lookahead() you > can't use the underlying iterator directly any more without losing > data, as you've consumed a value from it. So you have to use the > wrapped version That's a drawback to most itertools wrappers though, and one which makes sense since Python iterators and generators are not replayable/restartable. From tjreedy at udel.edu Mon Feb 25 13:03:12 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 25 Feb 2013 07:03:12 -0500 Subject: [Python-ideas] Add lookahead iterator (peeker) to itertools In-Reply-To: References: Message-ID: On 2/25/2013 5:08 AM, Paul Moore wrote: > On 25 February 2013 09:51, Wolfgang Maier > wrote: >> Terry Reedy writes: >> >>> class lookahead(): >>> "Wrap iterator with lookahead to both peek and test exhausted" >> ... >> >> +1 That's a nice tool that I'd love to have in itertools. > > It's not a bad idea, but I don't like the fact that as written it > turns a finite iterator into an infinite one (returning an endless > sequence of sentinels after the underlying iterator is exhausted). This is a bug in this re-write. The corrected .__next__ def __next__(self): if self: ret = self.peek self._set_peek() return ret else: raise StopIteration() passes the test with this addition try: next(it) assert False, "Next should have raised StopIteration" except StopIteration: pass >>> list(lookahead('abc')) == list('abc') True -- Terry Jan Reedy From jsbueno at python.org.br Mon Feb 25 13:37:11 2013 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Mon, 25 Feb 2013 09:37:11 -0300 Subject: [Python-ideas] Add lookahead iterator (peeker) to itertools In-Reply-To: References: Message-ID: Hi all! What is the problem with iterttols.tee even? It is almots the samething - although much more flexible, than what is proposed on this thread. >>> from itertools import tee >>> my_iter, peek = tee(iter(range(3)), 2) >>> next(peek) 0 >>> next(my_iter) 0 >>> js -><- On 25 February 2013 09:03, Terry Reedy wrote: > On 2/25/2013 5:08 AM, Paul Moore wrote: >> >> On 25 February 2013 09:51, Wolfgang Maier >> wrote: >>> >>> Terry Reedy writes: >>> >>>> class lookahead(): >>>> "Wrap iterator with lookahead to both peek and test exhausted" >>> >>> ... >>> >>> +1 That's a nice tool that I'd love to have in itertools. >> >> >> It's not a bad idea, but I don't like the fact that as written it >> turns a finite iterator into an infinite one (returning an endless >> sequence of sentinels after the underlying iterator is exhausted). > > > This is a bug in this re-write. The corrected .__next__ > > def __next__(self): > if self: > > ret = self.peek > self._set_peek() > return ret > else: > raise StopIteration() > > passes the test with this addition > > try: > next(it) > assert False, "Next should have raised StopIteration" > except StopIteration: > pass > >>>> list(lookahead('abc')) == list('abc') > > True > > -- > Terry Jan Reedy > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas From zuo at chopin.edu.pl Mon Feb 25 13:45:50 2013 From: zuo at chopin.edu.pl (Jan Kaliszewski) Date: Mon, 25 Feb 2013 13:45:50 +0100 Subject: [Python-ideas] =?utf-8?q?iterable=2E=5F=5Funpack=5F=5F_method?= In-Reply-To: References: Message-ID: <66901ffd88e8837eb58590472ad06fee@chopin.edu.pl> 24.02.2013 07:24, Guido van Rossum wrote: [...] > And to me the ... Looks too much like something that consumes the > rest, rather than leaving it. To me either. And we already have the syntax for consuming the rest: a, b, *seq = iterable But maybe it could be extended to include the following variant: a, b, *() = iterable -- expressing the "leave the rest untouched" behaviour? Doesn't such an empty-tuple-literal-like syntax suggest strongly enough that no items are consumed? Cheers. *j From masklinn at masklinn.net Mon Feb 25 13:56:12 2013 From: masklinn at masklinn.net (Masklinn) Date: Mon, 25 Feb 2013 13:56:12 +0100 Subject: [Python-ideas] Add lookahead iterator (peeker) to itertools In-Reply-To: References: Message-ID: <300C301B-D25A-4BA0-8F9C-01390330C629@masklinn.net> On 2013-02-25, at 13:37 , Joao S. O. Bueno wrote: > Hi all! > > What is the problem with iterttols.tee even? > It is almots the samething - although much more flexible, than what is > proposed on this thread. > > >>>> from itertools import tee >>>> my_iter, peek = tee(iter(range(3)), 2) >>>> next(peek) > 0 >>>> next(my_iter) > 0 >>>> Verbose way to do the same thing: you have to tee() + next() once per "peek()". And splitting to subroutines means these subroutines will have to return an iterator alongside any potential value as they'll have to tee() themselves. From ron3200 at gmail.com Mon Feb 25 16:29:52 2013 From: ron3200 at gmail.com (Ron Adam) Date: Mon, 25 Feb 2013 09:29:52 -0600 Subject: [Python-ideas] Add lookahead iterator (peeker) to itertools In-Reply-To: References: Message-ID: <512B8370.6040807@gmail.com> On 02/24/2013 09:41 PM, Terry Reedy wrote: > Does anyone else this should be added to itertools? It seems to not be > completely obvious to everyone, is more complex that some of the existing > itertools, and cannot be composed from them either. (Nor can it be written > as a generator function.) > > Any of the names can be changed. Perhaps the class should be 'peek' and the > lookahead object something else. The sentinel should be read-only if > possible. I considered whether the peek object should be read-only, but > someone would say that they *want* be able to replace the next object to be > yielded. Peeking into an exhausted iterable could raise instead of > returning the sentinel, but I don't know if that would be more useful. > > ---------------- > class lookahead(): > "Wrap iterator with lookahead to both peek and test exhausted" > > _NONE = object() > def __init__(self, iterable): > self._it = iter(iterable) > self._set_peek() > def __iter__(self): > return self > def __next__(self): > ret = self.peek > self._set_peek() > return ret > def _set_peek(self): > try: > self.peek = next(self._it) > except StopIteration: > self.peek = self._NONE > def __bool__(self): > return self.peek is not self._NONE > > def test_lookahead(): > it = lookahead('abc') > while it: > a = it.peek > b = next(it) > print('next:', b, '; is peek:', a is b ) > > test_lookahead() I think with a few small changes I would find it useful. The key feature here is that the result is pre calculated and held until it's needed, rather than calculated when it's asked for. You should catch any exception and hold that as well. On the next .next() call, it should raise the exception if there was one, or emit the value. I'm not sure if using the __bool__ attribute is the best choice. I would prefer a .error flag, along with a .next_value attribute. It would make the code using it easier to follow. it.error <-- True if next(it) will raise an exception. it.next_value <-- The next value, or the exception to raise. Note that iterating a list of exceptions will still work. About it.error. If it was a concurrent version, then it.error could have three values. it.error == True # Will raise an exception it.error == False # Will not raise an exception it.error == None # Still calculating I wonder how this type of generator will behave with "yield from". And if there would be any advantages for writing concurrent (or concurrent acting) code. Of course you really need to think about weather or not this fits the problem being solved. Cheers, Ron From yoavglazner at gmail.com Mon Feb 25 16:45:17 2013 From: yoavglazner at gmail.com (yoav glazner) Date: Mon, 25 Feb 2013 17:45:17 +0200 Subject: [Python-ideas] Add lookahead iterator (peeker) to itertools In-Reply-To: <512B8370.6040807@gmail.com> References: <512B8370.6040807@gmail.com> Message-ID: On Mon, Feb 25, 2013 at 5:29 PM, Ron Adam wrote: > > > On 02/24/2013 09:41 PM, Terry Reedy wrote: > >> Does anyone else this should be added to itertools? It seems to not be >> completely obvious to everyone, is more complex that some of the existing >> itertools, and cannot be composed from them either. (Nor can it be written >> as a generator function.) >> >> Any of the names can be changed. Perhaps the class should be 'peek' and >> the >> lookahead object something else. The sentinel should be read-only if >> possible. I considered whether the peek object should be read-only, but >> someone would say that they *want* be able to replace the next object to >> be >> yielded. Peeking into an exhausted iterable could raise instead of >> returning the sentinel, but I don't know if that would be more useful. >> >> ---------------- >> class lookahead(): >> "Wrap iterator with lookahead to both peek and test exhausted" >> >> _NONE = object() >> def __init__(self, iterable): >> self._it = iter(iterable) >> self._set_peek() >> def __iter__(self): >> return self >> def __next__(self): >> ret = self.peek >> self._set_peek() >> return ret >> def _set_peek(self): >> try: >> self.peek = next(self._it) >> except StopIteration: >> self.peek = self._NONE >> def __bool__(self): >> return self.peek is not self._NONE >> > > >> def test_lookahead(): >> it = lookahead('abc') >> while it: >> a = it.peek >> b = next(it) >> print('next:', b, '; is peek:', a is b ) >> >> test_lookahead() >> > > > I think with a few small changes I would find it useful. > > The key feature here is that the result is pre calculated and held until > it's needed, rather than calculated when it's asked for. > It seems like a buggy feature, think a DB cursor, next has side effects(like row locks and such) -------------- next part -------------- An HTML attachment was scrubbed... URL: From oscar.j.benjamin at gmail.com Mon Feb 25 17:20:41 2013 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Mon, 25 Feb 2013 16:20:41 +0000 Subject: [Python-ideas] Add lookahead iterator (peeker) to itertools In-Reply-To: References: Message-ID: On 25 February 2013 03:41, Terry Reedy wrote: > An iterator iter represents the remainder of some collection, concrete or > not, finite or not. If the remainder is not empty, its .__next__ method > selects one object, mutates iter to represent the reduced remainder (now > possibly empty), and returns the one item. > > At various times, people have asked for the ability to determine whether an > iterator is exhausted, whether a next() call will return or raise. If it > will return, people have also asked for the ability to peek at what the > return would be, without disturbing what the return will be. For instance, > on the 'iterable.__unpack__ method' Alex Stewart today wrote: >> The problem is that there is no standard way to query an iterator >> to find out if it has more data available without automatically >> consuming that next data element in the rocess. [SNIP] At times I thought I wanted the ability to query an iterator without necessarily popping an element. I've generally found that I ended up solving the problem in a different way. My own solution is to have a pushable iterator so that after inspecting a value I can push it back onto the iterator ready for a subsequent next() call. Example code for this looks like: def _pushable(iterable): '''Helper for pushable''' iterator = iter(iterable) stack = [] yield lambda x: stack.append(x) while True: while stack: yield stack.pop() yield next(iterator) def pushable(iterable): '''Make an iterable pushable. >>> iterable, push = pushable(range(9)) >>> next(iterable) 0 >>> next(iterable) 1 >>> push(1) # Push the 1 back on to the iterable >>> list(iterable) [1, 2, 3, 4, 5, 6, 7, 8] ''' gen = _pushable(iterable) push = next(gen) return gen, push Oscar From jeanpierreda at gmail.com Mon Feb 25 17:20:30 2013 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Mon, 25 Feb 2013 11:20:30 -0500 Subject: [Python-ideas] iterable.__unpack__ method In-Reply-To: <66901ffd88e8837eb58590472ad06fee@chopin.edu.pl> References: <66901ffd88e8837eb58590472ad06fee@chopin.edu.pl> Message-ID: On Mon, Feb 25, 2013 at 7:45 AM, Jan Kaliszewski wrote: > But maybe it could be extended to include the following variant: > > a, b, *() = iterable > > -- expressing the "leave the rest untouched" behaviour? Doesn't such > an empty-tuple-literal-like syntax suggest strongly enough that > no items are consumed? I would've interpreted it as unpacking the rest of the iterable into () -- i.e., I'd assume it has the current behavior of failing if the rest of the iterable has anything at all. Of course, you can't unpack anything into (), because Python never had that syntax, but you get the idea. -- Devin From jbvsmo at gmail.com Mon Feb 25 17:29:07 2013 From: jbvsmo at gmail.com (=?ISO-8859-1?Q?Jo=E3o_Bernardo?=) Date: Mon, 25 Feb 2013 13:29:07 -0300 Subject: [Python-ideas] iterable.__unpack__ method In-Reply-To: <66901ffd88e8837eb58590472ad06fee@chopin.edu.pl> References: <66901ffd88e8837eb58590472ad06fee@chopin.edu.pl> Message-ID: > But maybe it could be extended to include the following variant: > > a, b, *() = iterable > > Python already supports this odd syntax a, b, *[] = iterable because it interprets the [] not as an empty list, but as an empty "list of identifiers". Maybe it could be used for something useful. BTW, the del syntax has the same "problem" del a, b, (c,), [d], [] -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Mon Feb 25 19:53:02 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 25 Feb 2013 19:53:02 +0100 Subject: [Python-ideas] Add lookahead iterator (peeker) to itertools References: <20130225110717.232ce082@pitrou.net> Message-ID: <20130225195302.45704b36@pitrou.net> On Mon, 25 Feb 2013 13:58:59 +0200 Serhiy Storchaka wrote: > On 25.02.13 12:07, Antoine Pitrou wrote: > > Le Sun, 24 Feb 2013 22:41:48 -0500, > > Terry Reedy a ?crit : > >> > >> def test_lookahead(): > >> it = lookahead('abc') > >> while it: > >> a = it.peek > >> b = next(it) > >> print('next:', b, '; is peek:', a is b ) > > > > def test_lookahead(): > > it = iter('abc') > > while True: > > it, peeking = itertools.tee(it) > > This should be outside a loop. Only if you restrict yourself to access peeking each time you access it. (which, I suppose, is not the general use case for the lookahead proposal) Regards Antoine. From storchaka at gmail.com Mon Feb 25 21:21:03 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 25 Feb 2013 22:21:03 +0200 Subject: [Python-ideas] Add lookahead iterator (peeker) to itertools In-Reply-To: <20130225195302.45704b36@pitrou.net> References: <20130225110717.232ce082@pitrou.net> <20130225195302.45704b36@pitrou.net> Message-ID: On 25.02.13 20:53, Antoine Pitrou wrote: >>> def test_lookahead(): >>> it = iter('abc') >>> while True: >>> it, peeking = itertools.tee(it) >> >> This should be outside a loop. > > Only if you restrict yourself to access peeking each time you access it. > (which, I suppose, is not the general use case for the lookahead > proposal) Only if your do not want to consume O(N) memory and spend O(N**2) time. From solipsis at pitrou.net Mon Feb 25 21:27:22 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 25 Feb 2013 21:27:22 +0100 Subject: [Python-ideas] Add lookahead iterator (peeker) to itertools References: <20130225110717.232ce082@pitrou.net> <20130225195302.45704b36@pitrou.net> Message-ID: <20130225212722.0d6de65b@pitrou.net> On Mon, 25 Feb 2013 22:21:03 +0200 Serhiy Storchaka wrote: > On 25.02.13 20:53, Antoine Pitrou wrote: > >>> def test_lookahead(): > >>> it = iter('abc') > >>> while True: > >>> it, peeking = itertools.tee(it) > >> > >> This should be outside a loop. > > > > Only if you restrict yourself to access peeking each time you access it. > > (which, I suppose, is not the general use case for the lookahead > > proposal) > > Only if your do not want to consume O(N) memory and spend O(N**2) time. No, that's beside the point. If you don't consume "peeking" in lock-step with "it", then "peeking" and "it" become desynchronized and therefore the semantics are wrong w.r.t to the original feature request (where "peeking" is supposed to be some proxy to "it", not an independently-running iterator). Regards Antoine. From barry at python.org Mon Feb 25 22:08:11 2013 From: barry at python.org (Barry Warsaw) Date: Mon, 25 Feb 2013 16:08:11 -0500 Subject: [Python-ideas] Add lookahead iterator (peeker) to itertools References: Message-ID: <20130225160811.680ecc42@anarchist.wooz.org> On Feb 25, 2013, at 04:20 PM, Oscar Benjamin wrote: >At times I thought I wanted the ability to query an iterator without >necessarily popping an element. I've generally found that I ended up >solving the problem in a different way. My own solution is to have a >pushable iterator so that after inspecting a value I can push it back >onto the iterator ready for a subsequent next() call. The email package has one of these in its feedparser module, called BufferedSubFile. It needs to be able to push lines of text back onto the stack that the normal iteration pops off. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From foogod at gmail.com Mon Feb 25 22:52:32 2013 From: foogod at gmail.com (Alex Stewart) Date: Mon, 25 Feb 2013 13:52:32 -0800 (PST) Subject: [Python-ideas] iterable.__unpack__ method In-Reply-To: <512AD5E8.4030204@fastmail.us> References: <512AD5E8.4030204@fastmail.us> Message-ID: <68bec4db-8e8a-4113-ad90-247193ad8a86@googlegroups.com> On Sunday, February 24, 2013 7:09:28 PM UTC-8, Random832 wrote: > If you want "smart" unpackable objects, why not allow d in your *d > example to be something other than a list? > > Like, if you have a,*b = range(1,5); b could literally be range(2,5). > Well, primarily for two reasons: 1. It does not actually do anything to support smart unpackables (i.e. objects that want to be able to produce different values depending on how many items they're asked to produce). This might be useful for the "infinite iterator" case (and I think somebody proposed it for that earlier), but that's a completely different issue than smart-unpacking.. 2. More importantly, it would almost certainly break some existing code that relies on the fact that the contents of the extended-unpacking-term will always be a list, which would be bad. --Alex -------------- next part -------------- An HTML attachment was scrubbed... URL: From foogod at gmail.com Mon Feb 25 23:25:03 2013 From: foogod at gmail.com (Alex Stewart) Date: Mon, 25 Feb 2013 14:25:03 -0800 (PST) Subject: [Python-ideas] iterable.__unpack__ method In-Reply-To: References: <44528db0ff81ea147dfc71d97e472798@chopin.edu.pl> <512A7967.1060709@mrabarnett.plus.com> Message-ID: On Sunday, February 24, 2013 7:43:58 PM UTC-8, Terry Reedy wrote: > On 2/24/2013 5:07 PM, Alex Stewart wrote: > > 1. It is a fairly small change to the language > > To me, taking something simple and elegant and complexifying it, to > little benefit I can see other than to solve a problem in the wrong > place, in no small matter. I did not say it was a small matter. I said it was a small change to the language. From a purely practical perspective of what changes would be required to the language itself in order to support this proposal, in my opinion, it would not require that much to implement (it would probably only require a couple of lines of new code and some doc changes), and is very unlikely to affect any existing code or tools. (Syntax changes are, in my opinion, far more disruptive changes to the language: They require a change to the parser as well as the interpreter, and potentially changes to all manner of other utilities (debuggers/code-checkers/AST-manipulators/etc) out there in the wild. They make it much harder to write Python code that supports older Python releases (with method changes, you can check for the existence of a feature in the interpreter and selectively use or not use it, but you typically can't use new syntax constructs in anything that might be run in an older Python release, because the parser will bomb out instead). They also require programmers to learn new syntax (if nothing else, to understand what it means when they come across it in somebody else's code). What's more, every syntax change potentially limits what can be done with the syntax in the future (for example, by using up special characters or constructs so they're not available for other things), and so on. Personally, I think there should be a much higher bar for syntax changes than for adding a new special method, not the other way around.) You also seem to be defining "in the wrong place" as "in a different way than I personally have a use for".. As I pointed out (which you seem to have completely skipped over in your reply), there are use cases where the consumer side is the right (or only) place, and there are other different use cases where the producer side is the right (or only) place. The two are not mutually-exclusive, and improving one does not really address the other issue at all. I want to emphasize that I am not saying that we shouldn't extend unpacking notation to support partial-unpacking; in fact I think it might well be a good idea. What I am saying, however, is that if we do, I do think it's a much bigger change to the language than what I proposed, and it still doesn't actually address most of the issues that __unpack__ was intended to deal with, so it's really not an either-or sort of thing anyway. Certainly, adding a new special method to > some category of objects has not effect unless changes are made to the > interpreter elsewhere to make automatic use of that method. Otherwise, > you could just define a normal method to be used by user code. My proposal does not include adding a new special method to any existing objects. It adds the _ability_ for people to add a special method to objects *in the future* and have the interpreter take advantage of it. It therefore should have no effect for any existing code. > 2. There are (in my opinion) already pretty good parallels with this > > sort of change that have already been implemented in Python (for > > example, separating the concept of "iterable" from the concept of > > "indexable"). > > The related but distinct concepts of sequential access and random access > are basic to computation and were *always* distinct in Python. Oh really? Then how, prior to the development of the iterator protocol, did one define an object which was accessible sequentially but not randomly in the Python language? If you couldn't do that, you can't claim that the concepts were really distinct in Python, in my opinion. The truth is that the concept of an iterable object vs. an indexable object may have been distinct *in some people's minds*, but it was *not* originally distinct in the Python language itself. People decided that that should change, so the language was extended to make that happen. It should also be noted that the two concepts were not distinct in *everyone's* minds (I remember straightening out confusion on this point in numerous people's understanding (and still need to occasionally), particularly those who did not come from a classical computer-science background). However, just because there were some people who didn't see them as distinct did not mean that it was an invalid way for other people to view them, or that they shouldn't ultimately be distinct in the language.. With all respect (and I do actually mean that), your primary argument against it seems to be one of intellectual complacency: "I've never considered that they might be different, so they shouldn't ever be separated, because then I don't have to change the way I think about things". The addition of __iter__ along with __next__ > added more benefits than __next__ alone would have. > You seem to be making my point for me. In essence, __unpack__ is really just the equivalent for the unpacking operation to what __iter__ is for "for" loops. Is it strictly required in order to support the basic behavior? No (you could do for loops over simple iterables with just a variant of __next__, if you really wanted to), but it allows some more sophisticated objects to provide more useful behaviors by giving them more context about how they are being invoked (in the case of __iter__, by telling them ahead of time that they are going to be used in an iteration context, and allowing them to create or alter the initial object that the for loop interacts with). This provides "more benefits than __next__ alone would have", so it was added to the language. Logically, by the same argument, and in almost exactly the same way, __unpack__ provides more benefits than __iter__ alone would have (by telling them that they are going to be used in a particular unpacking context, and allowing them to alter the object used for that type of iteration), and therefore should arguably be part of the language as well. > 3. It is consistent with the general idea in Python that an object can > > choose to emulate whichever behaviors of other objects make sense, > > and not emulate other ones that do not make sense. In my opinion, > > "iterating", and "unpacking" are two related-but-not-the-same > > language concepts, which currently are unnecessarily entangled. > > I do not see that at all. As near as I can tell, both mean 'sequential > access'. Not true. Iteration means "sequential access". Unpacking means "sequential access with inherent length constraints", which is the bit you seem to be ignoring. The most notable consequence of this (but definitely not the only one) is that a for loop (for example) will never raise ValueError, but the unpacking operation will, if it gets what it considers "wrong" results back from the unpacked object. In this way the two operations do not interpret the data that the object in question provides in the same way, and it does not really have the same meaning. Yes, from a purely mechanical perspective, they perform similar operations, but programming interfaces are not solely defined by mechanics. The difference here is one of implied contracts. In simple iteration, there is explicitly no minimum or maximum number of values that the iterable is required to produce, and in fact the entire protocol is based on the assumed perspective that returning a value is always the "success" condition and not being able to return one is the "failure" case, and in that context it makes sense to design iterators so that they should always return more data if they're able to do so. In unpacking, however, when it gets to the end of the variable list, the conditions are reversed, and actually returning a value becomes a failure condition. This change of meaning, however, is not communicated in any way to the iterator (this is just one example: there are similar contract issues inherent in extended unpacking as well). This is, in my opinion, a flaw in the interface, because the language is misrepresenting (bounded) unpacking as (unbounded) iteration when calling the iterable object. I admit I do not understand your __unpack__ proposal, since it seemed > vague and incomplete to me. But until I see your conceptual distinction, > the details do not matter to me. > I'm not exactly sure why you consider it vague or incomplete. I provided a specific description of exactly what changes to the language I was proposing, with method signatures, a description of the new interpreter behavior and examples of data values in typical scenarios, and accompanied it with a list of several different use-cases which I believed it would provide significant utility. I'm not really sure how I could have gotten more specific without providing a diff against the CPython sources (which seems a bit much for an initial proposal).. > 4. It would help significantly with several types of problems that do > > not currently have great solutions. > > I obviously do not see the same pile of problems that would justify a > new special method. Obviously, although I'm not really sure why not because I did explicitly state a bunch of them in my original post. You seem to have just ignored all of them except for the issue of partial unpacking, which frankly was not even the most important or interesting use case I presented, in my opinion. I would recommend you might want to go back and read the last part of my initial post where I presented several potential use cases for all this. I do get the distinct impression, however, that you've already made up your mind and, right or wrong, there's nothing I could possibly say which would ever change it, so at this point I'm not sure whether it really matters (sigh)... > no way to query an iterator to find out if it has more data available See how in my separate post: > "Add lookahead iterator (peeker) to itertools" I'm not necessarily opposed to that proposal, but I should point out that it does not actually address the problem I mentioned in this thread at all, so it's really kinda irrelevant to that discussion. (Even if that is added to itertools, unpacking still won't use it, nor does it provide any way to safely query iterables which produce side-effects, even if there might be a way for them to provide that info without the side-effects, nor does it provide a way to test for end conditions without changing the underlying iterable (which might be used elsewhere in scopes outside the lookahead-wrapper), etc..) --Alex -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Tue Feb 26 00:30:03 2013 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 26 Feb 2013 12:30:03 +1300 Subject: [Python-ideas] iterable.__unpack__ method In-Reply-To: References: <66901ffd88e8837eb58590472ad06fee@chopin.edu.pl> Message-ID: <512BF3FB.9090803@canterbury.ac.nz> Devin Jeanpierre wrote: > On Mon, Feb 25, 2013 at 7:45 AM, Jan Kaliszewski wrote: >> a, b, *() = iterable > > Of course, you can't unpack anything into (), because Python never had > that syntax, but you get the idea. -1, this is just as arbitrary as ... or a lone *. I prefer ... myself, because visually it has a low profile and doesn't draw undue attention to something that you're saying you don't care about. Maybe ... could be blessed as an official "don't care" assignment target, like _ in Haskell. That would make this usage less of a special case (although it would still be partially special, since it suppresses unpacking of the rest of the iterator). -- Greg From greg.ewing at canterbury.ac.nz Tue Feb 26 00:37:48 2013 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 26 Feb 2013 12:37:48 +1300 Subject: [Python-ideas] iterable.__unpack__ method In-Reply-To: References: <66901ffd88e8837eb58590472ad06fee@chopin.edu.pl> Message-ID: <512BF5CC.4050602@canterbury.ac.nz> Jo?o Bernardo wrote: > Python already supports this odd syntax > > a, b, *[] = iterable > > because it interprets the [] not as an empty list, but as an empty "list > of identifiers". Maybe it could be used for something useful. No, because it already has a meaning: there must be no more values left in the sequence. > BTW, the del syntax has the same "problem" > > del a, b, (c,), [d], [] Or just [] = iterable The surprising thing is that a special case seems to be made for (): >>> () = [] File "", line 1 SyntaxError: can't assign to () It's surprising because () and [] are otherwise completely interchangeable for unpacking purposes. -- Greg From steve at pearwood.info Tue Feb 26 01:07:25 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 26 Feb 2013 11:07:25 +1100 Subject: [Python-ideas] iterable.__unpack__ method In-Reply-To: <512BF3FB.9090803@canterbury.ac.nz> References: <66901ffd88e8837eb58590472ad06fee@chopin.edu.pl> <512BF3FB.9090803@canterbury.ac.nz> Message-ID: <512BFCBD.40702@pearwood.info> On 26/02/13 10:30, Greg Ewing wrote: > Devin Jeanpierre wrote: >> On Mon, Feb 25, 2013 at 7:45 AM, Jan Kaliszewski wrote: >>> a, b, *() = iterable >> >> Of course, you can't unpack anything into (), because Python never had >> that syntax, but you get the idea. > > -1, this is just as arbitrary as ... or a lone *. > > I prefer ... myself, because visually it has a low profile > and doesn't draw undue attention to something that you're > saying you don't care about. > > Maybe ... could be blessed as an official "don't care" > assignment target, like _ in Haskell. That would make this > usage less of a special case (although it would still be > partially special, since it suppresses unpacking of the > rest of the iterator). Please no. _ is already overloaded too much. In i18n contexts, _ is conventionally used as the function for getting display text. In the interactive interpreter, _ is used for the last result. And in source code, _ is often used by convention as a "don't care" variable. Turning _ into a magic "stop unpacking" symbol would break code that uses it as a "don't care" target when unpacking: spam, _, _, ham, eggs = stuff and would be ambiguous when there is an underscore as the right-most target. Would it mean, unpack but I don't care about it, or don't unpack? I'm still not seeing why this is important enough to add magic syntax. If you want to unpack the first five values from an iterator, without exhausting it, we already have some good solutions: spam, ham, eggs, cheese, tomato = (next(it) for i in range(5)) or spam, ham, eggs, cheese, tomato = itertools.islice(it, 5) Neither of these are so confusing to read or onerous to use as to justify new magic syntax: spam, ham, eggs, cheese, tomato, $$$$ = it # for some value of $$$ -- Steven From abarnert at yahoo.com Tue Feb 26 01:12:04 2013 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 25 Feb 2013 16:12:04 -0800 Subject: [Python-ideas] My wishlist for Tulip In-Reply-To: References: Message-ID: <0660447D-BC71-4B49-A2F6-9CE00643A958@yahoo.com> Sorry for the top post, but I'm on a phone. I'd love to see a trivial chat server in the demos. There are many designs for async servers that work great with echo and even http but fall down with chat (because of interactions between clients), and it would be very useful to see how tulip makes it easy to do it right. Sent from a random iPhone. On Feb 22, 2013, at 13:17, Guido van Rossum wrote: > Those are all good ideas! Now I just have to find time to implement them... > > On Fri, Feb 22, 2013 at 9:15 AM, Giampaolo Rodol? wrote: >> I understand it's still beta but anyways, here's my little wish list for Tulip. >> >> * provide a 'demo' / 'example' directory containing very simple >> scripts showing the most basic usages such as: >> >> - echo_tcp_client.py >> - echo_tcp_server.py >> - echo_tcp_server_w_timeout.py (same as echo_server.py but also >> disconnects the client after a certain time of inactivity) >> - echo_tcp_ssl_client.py >> - echo_tcp_ssl_server.py >> - echo_udp_client.py >> - echo_udp_server.py >> >> * move all *test*.py scripts in a separate 'test' directory >> >> * if it's not part of the API intended to be public move >> tulip/http_client.py elsewhere ('examples'/'demo' or a brand new >> 'scripts'/'tools' directory) >> >> * (minor) same for check.py, crawl.py, curl.py, sslsrv.py which looks >> like they belong elsewhere >> >> * write a simple benchmark framework testing (at least) sending, >> receiving and the internal scheduler (I'd like to help with this one) >> >> >> --- Giampaolo >> http://code.google.com/p/pyftpdlib/ >> http://code.google.com/p/psutil/ >> http://code.google.com/p/pysendfile/ >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> http://mail.python.org/mailman/listinfo/python-ideas > > > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas From zuo at chopin.edu.pl Tue Feb 26 01:13:30 2013 From: zuo at chopin.edu.pl (Jan Kaliszewski) Date: Tue, 26 Feb 2013 01:13:30 +0100 Subject: [Python-ideas] =?utf-8?q?iterable=2E=5F=5Funpack=5F=5F_method?= In-Reply-To: <512BF5CC.4050602@canterbury.ac.nz> References: <66901ffd88e8837eb58590472ad06fee@chopin.edu.pl> <512BF5CC.4050602@canterbury.ac.nz> Message-ID: <366654f2dcc7518781582d9e7eacca91@chopin.edu.pl> 26.02.2013 00:37, Greg Ewing wrote: > Jo?o Bernardo wrote: >> Python already supports this odd syntax >> a, b, *[] = iterable Indeed, I didn't know that. :-| >> because it interprets the [] not as an empty list, but as an empty >> "list of identifiers". Maybe it could be used for something useful. > > No, because it already has a meaning: there must be no more > values left in the sequence. > >> BTW, the del syntax has the same "problem" >> del a, b, (c,), [d], [] > > Or just > > [] = iterable > > The surprising thing is that a special case seems to be > made for (): > >>>> () = [] > File "", line 1 > SyntaxError: can't assign to () > > It's surprising because () and [] are otherwise completely > interchangeable for unpacking purposes. Not entirely... >>> a, b, [c, d] = 1, 2, (3, 4) >>> a, b, (c, d) = 1, 2, (3, 4) OK. >>> a, b, *[c, d] = 1, 2, 3, 4 >>> a, b, *(c, d) = 1, 2, 3, 4 OK as well -- *but*: >>> a, b, [] = 1, 2, [] # it's ok >>> a, b, () = 1, 2, () # but it's not File "", line 1 SyntaxError: can't assign to () ...and: >>> a, b, *[] = 1, 2 # it's ok >>> a, b, *() = 1, 2 # but it's not File "", line 1 SyntaxError: can't assign to () Strange... (inb4: http://www.youtube.com/watch?v=5IgB8XKR23c#t=118s ). *j From python at mrabarnett.plus.com Tue Feb 26 01:19:54 2013 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 26 Feb 2013 00:19:54 +0000 Subject: [Python-ideas] iterable.__unpack__ method In-Reply-To: <512BFCBD.40702@pearwood.info> References: <66901ffd88e8837eb58590472ad06fee@chopin.edu.pl> <512BF3FB.9090803@canterbury.ac.nz> <512BFCBD.40702@pearwood.info> Message-ID: <512BFFAA.2020706@mrabarnett.plus.com> On 2013-02-26 00:07, Steven D'Aprano wrote: > On 26/02/13 10:30, Greg Ewing wrote: >> Devin Jeanpierre wrote: >>> On Mon, Feb 25, 2013 at 7:45 AM, Jan Kaliszewski wrote: >>>> a, b, *() = iterable >>> >>> Of course, you can't unpack anything into (), because Python never had >>> that syntax, but you get the idea. >> >> -1, this is just as arbitrary as ... or a lone *. >> >> I prefer ... myself, because visually it has a low profile >> and doesn't draw undue attention to something that you're >> saying you don't care about. >> >> Maybe ... could be blessed as an official "don't care" >> assignment target, like _ in Haskell. That would make this >> usage less of a special case (although it would still be >> partially special, since it suppresses unpacking of the >> rest of the iterator). > > Please no. _ is already overloaded too much. In i18n contexts, _ is > conventionally used as the function for getting display text. In the > interactive interpreter, _ is used for the last result. And in source > code, _ is often used by convention as a "don't care" variable. > [snip] He didn't say that we should use "_", he said that "..." would be like "_" in Haskell. From fafhrd91 at gmail.com Tue Feb 26 01:41:33 2013 From: fafhrd91 at gmail.com (Nikolay Kim) Date: Mon, 25 Feb 2013 16:41:33 -0800 Subject: [Python-ideas] My wishlist for Tulip In-Reply-To: <0660447D-BC71-4B49-A2F6-9CE00643A958@yahoo.com> References: <0660447D-BC71-4B49-A2F6-9CE00643A958@yahoo.com> Message-ID: <18F634D4-9C75-4FCC-9C33-917DCAE72CC8@gmail.com> You can check my experimental httpclient code. it has web socket example, server + cmd client. https://github.com/fafhrd91/httpclient On Feb 25, 2013, at 4:12 PM, Andrew Barnert wrote: > Sorry for the top post, but I'm on a phone. > > I'd love to see a trivial chat server in the demos. There are many designs for async servers that work great with echo and even http but fall down with chat (because of interactions between clients), and it would be very useful to see how tulip makes it easy to do it right. > > Sent from a random iPhone. > > On Feb 22, 2013, at 13:17, Guido van Rossum wrote: > >> Those are all good ideas! Now I just have to find time to implement them... >> >> On Fri, Feb 22, 2013 at 9:15 AM, Giampaolo Rodol? wrote: >>> I understand it's still beta but anyways, here's my little wish list for Tulip. >>> >>> * provide a 'demo' / 'example' directory containing very simple >>> scripts showing the most basic usages such as: >>> >>> - echo_tcp_client.py >>> - echo_tcp_server.py >>> - echo_tcp_server_w_timeout.py (same as echo_server.py but also >>> disconnects the client after a certain time of inactivity) >>> - echo_tcp_ssl_client.py >>> - echo_tcp_ssl_server.py >>> - echo_udp_client.py >>> - echo_udp_server.py >>> >>> * move all *test*.py scripts in a separate 'test' directory >>> >>> * if it's not part of the API intended to be public move >>> tulip/http_client.py elsewhere ('examples'/'demo' or a brand new >>> 'scripts'/'tools' directory) >>> >>> * (minor) same for check.py, crawl.py, curl.py, sslsrv.py which looks >>> like they belong elsewhere >>> >>> * write a simple benchmark framework testing (at least) sending, >>> receiving and the internal scheduler (I'd like to help with this one) >>> >>> >>> --- Giampaolo >>> http://code.google.com/p/pyftpdlib/ >>> http://code.google.com/p/psutil/ >>> http://code.google.com/p/pysendfile/ >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> http://mail.python.org/mailman/listinfo/python-ideas >> >> >> >> -- >> --Guido van Rossum (python.org/~guido) >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> http://mail.python.org/mailman/listinfo/python-ideas > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas From jbvsmo at gmail.com Tue Feb 26 02:20:40 2013 From: jbvsmo at gmail.com (=?ISO-8859-1?Q?Jo=E3o_Bernardo?=) Date: Mon, 25 Feb 2013 22:20:40 -0300 Subject: [Python-ideas] iterable.__unpack__ method In-Reply-To: <512BF5CC.4050602@canterbury.ac.nz> References: <66901ffd88e8837eb58590472ad06fee@chopin.edu.pl> <512BF5CC.4050602@canterbury.ac.nz> Message-ID: 2013/2/25 Greg Ewing > Jo?o Bernardo wrote: > >> Python already supports this odd syntax >> >> a, b, *[] = iterable >> >> because it interprets the [] not as an empty list, but as an empty "list >> of identifiers". Maybe it could be used for something useful. >> > > No, because it already has a meaning: there must be no more > values left in the sequence. > > Why have two things with the same meaning? a, b = iterable a, b, *[] = iterable Both are the same... The *[] thing is about 100% useless right now. And the empty "list" syntax is so not used it was possible to segfault pypy by putting that in a for loop. > > BTW, the del syntax has the same "problem" >> >> del a, b, (c,), [d], [] >> > > Or just > > [] = iterable > > The surprising thing is that a special case seems to be > made for (): > > >>> () = [] > File "", line 1 > SyntaxError: can't assign to () > > It's surprising because () and [] are otherwise completely > interchangeable for unpacking purposes That's what I'm saying... [] and () during unpaking aren't the same thing when they normally should... The same happens with the "del" syntax! One can think about this to see if an iterable is empty and raise an error if it is not: [] = iterable But *[], = iterable Has the exact same meaning and is very cryptic. BTW: This is the code golf winner to make a list of chars: *s, = 'string' -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Tue Feb 26 02:29:23 2013 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 26 Feb 2013 01:29:23 +0000 (UTC) Subject: [Python-ideas] Add lookahead iterator (peeker) to itertools References: <20130225160811.680ecc42@anarchist.wooz.org> Message-ID: Barry Warsaw writes: > > On Feb 25, 2013, at 04:20 PM, Oscar Benjamin wrote: > > >At times I thought I wanted the ability to query an iterator without > >necessarily popping an element. I've generally found that I ended up > >solving the problem in a different way. My own solution is to have a > >pushable iterator so that after inspecting a value I can push it back > >onto the iterator ready for a subsequent next() call. > > The email package has one of these in its feedparser module, called > BufferedSubFile. It needs to be able to push lines of text back onto the > stack that the normal iteration pops off. It's also a common pattern in lexers/parsers/compilers where you need to push back characters/tokens while doing lookahead for e.g. disambiguation. Regards, Vinay Sajip From abarnert at yahoo.com Tue Feb 26 03:56:24 2013 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 25 Feb 2013 18:56:24 -0800 Subject: [Python-ideas] iterable.__unpack__ method In-Reply-To: <5129D8A9.6000000@pearwood.info> References: <5129D8A9.6000000@pearwood.info> Message-ID: On Feb 24, 2013, at 1:08, Steven D'Aprano wrote: > On 24/02/13 19:04, Bruce Leban wrote: > >> And hypothetically > [...] >> a, b, c = (next * 3)(iterargs) > > That looks like, well, I'm not sure what it looks like. Haskell maybe? But not Python. Not Haskell either. If you told me this was an expression in a language that's like Haskell but with python (or C) syntax, I'd expect it to mean something like compose_n(next, 3)(iterargs), aka next(next(next(iterargs))), not map(next, iterargs * 3). Maybe next(3, iterargs) or next(3)(iterargs), but that's just islice. From abarnert at yahoo.com Tue Feb 26 04:21:06 2013 From: abarnert at yahoo.com (Andrew Barnert) Date: Mon, 25 Feb 2013 19:21:06 -0800 Subject: [Python-ideas] My wishlist for Tulip In-Reply-To: <18F634D4-9C75-4FCC-9C33-917DCAE72CC8@gmail.com> References: <0660447D-BC71-4B49-A2F6-9CE00643A958@yahoo.com> <18F634D4-9C75-4FCC-9C33-917DCAE72CC8@gmail.com> Message-ID: On Feb 25, 2013, at 16:41, Nikolay Kim wrote: > You can check my experimental httpclient code. it has web socket example, server + cmd client. > > https://github.com/fafhrd91/httpclient How does that answer my request? The whole point was to show how passing information between socket handlers (or, more generally, coro chains) is easier with tulip than with traditional async solutions. An http client doesn't do that. (And it's probably much more complex, too.) > On Feb 25, 2013, at 4:12 PM, Andrew Barnert wrote: > >> Sorry for the top post, but I'm on a phone. >> >> I'd love to see a trivial chat server in the demos. There are many designs for async servers that work great with echo and even http but fall down with chat (because of interactions between clients), and it would be very useful to see how tulip makes it easy to do it right. >> >> Sent from a random iPhone. >> >> On Feb 22, 2013, at 13:17, Guido van Rossum wrote: >> >>> Those are all good ideas! Now I just have to find time to implement them... >>> >>> On Fri, Feb 22, 2013 at 9:15 AM, Giampaolo Rodol? wrote: >>>> I understand it's still beta but anyways, here's my little wish list for Tulip. >>>> >>>> * provide a 'demo' / 'example' directory containing very simple >>>> scripts showing the most basic usages such as: >>>> >>>> - echo_tcp_client.py >>>> - echo_tcp_server.py >>>> - echo_tcp_server_w_timeout.py (same as echo_server.py but also >>>> disconnects the client after a certain time of inactivity) >>>> - echo_tcp_ssl_client.py >>>> - echo_tcp_ssl_server.py >>>> - echo_udp_client.py >>>> - echo_udp_server.py >>>> >>>> * move all *test*.py scripts in a separate 'test' directory >>>> >>>> * if it's not part of the API intended to be public move >>>> tulip/http_client.py elsewhere ('examples'/'demo' or a brand new >>>> 'scripts'/'tools' directory) >>>> >>>> * (minor) same for check.py, crawl.py, curl.py, sslsrv.py which looks >>>> like they belong elsewhere >>>> >>>> * write a simple benchmark framework testing (at least) sending, >>>> receiving and the internal scheduler (I'd like to help with this one) >>>> >>>> >>>> --- Giampaolo >>>> http://code.google.com/p/pyftpdlib/ >>>> http://code.google.com/p/psutil/ >>>> http://code.google.com/p/pysendfile/ >>>> _______________________________________________ >>>> Python-ideas mailing list >>>> Python-ideas at python.org >>>> http://mail.python.org/mailman/listinfo/python-ideas >>> >>> >>> >>> -- >>> --Guido van Rossum (python.org/~guido) >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> http://mail.python.org/mailman/listinfo/python-ideas >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> http://mail.python.org/mailman/listinfo/python-ideas > From ncoghlan at gmail.com Tue Feb 26 04:50:18 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 26 Feb 2013 13:50:18 +1000 Subject: [Python-ideas] iterable.__unpack__ method In-Reply-To: References: <66901ffd88e8837eb58590472ad06fee@chopin.edu.pl> <512BF5CC.4050602@canterbury.ac.nz> Message-ID: On Tue, Feb 26, 2013 at 11:20 AM, Jo?o Bernardo wrote: > > 2013/2/25 Greg Ewing >> >> Jo?o Bernardo wrote: >>> >>> Python already supports this odd syntax >>> >>> a, b, *[] = iterable >>> >>> because it interprets the [] not as an empty list, but as an empty "list >>> of identifiers". Maybe it could be used for something useful. >> >> >> No, because it already has a meaning: there must be no more >> values left in the sequence. >> > > Why have two things with the same meaning? > > a, b = iterable > > a, b, *[] = iterable > > Both are the same... The *[] thing is about 100% useless right now. And almost certainly an unintentional quirk of the way the Grammar is constructed. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From steve at pearwood.info Tue Feb 26 05:03:22 2013 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 26 Feb 2013 15:03:22 +1100 Subject: [Python-ideas] iterable.__unpack__ method In-Reply-To: <512BFFAA.2020706@mrabarnett.plus.com> References: <66901ffd88e8837eb58590472ad06fee@chopin.edu.pl> <512BF3FB.9090803@canterbury.ac.nz> <512BFCBD.40702@pearwood.info> <512BFFAA.2020706@mrabarnett.plus.com> Message-ID: <512C340A.304@pearwood.info> On 26/02/13 11:19, MRAB wrote: > On 2013-02-26 00:07, Steven D'Aprano wrote: >> On 26/02/13 10:30, Greg Ewing wrote: >>> Devin Jeanpierre wrote: >>>> On Mon, Feb 25, 2013 at 7:45 AM, Jan Kaliszewski wrote: >>>>> a, b, *() = iterable >>>> >>>> Of course, you can't unpack anything into (), because Python never had >>>> that syntax, but you get the idea. >>> >>> -1, this is just as arbitrary as ... or a lone *. >>> >>> I prefer ... myself, because visually it has a low profile >>> and doesn't draw undue attention to something that you're >>> saying you don't care about. >>> >>> Maybe ... could be blessed as an official "don't care" >>> assignment target, like _ in Haskell. That would make this >>> usage less of a special case (although it would still be >>> partially special, since it suppresses unpacking of the >>> rest of the iterator). >> >> Please no. _ is already overloaded too much. In i18n contexts, _ is >> conventionally used as the function for getting display text. In the >> interactive interpreter, _ is used for the last result. And in source >> code, _ is often used by convention as a "don't care" variable. >> > [snip] > He didn't say that we should use "_", he said that "..." would be like > "_" in Haskell. /facepalm And so he did. Sorry about that Greg! But still, I think this does help demonstrate that symbols are not always the easiest to read at a glance. It's too easy for the eyes to slide right off them, unless they are as familiar as basic arithmetic. -- Steven From pyideas at rebertia.com Tue Feb 26 05:51:24 2013 From: pyideas at rebertia.com (Chris Rebert) Date: Mon, 25 Feb 2013 20:51:24 -0800 Subject: [Python-ideas] Adding a safe alternative to pickle in the standard library In-Reply-To: References: <20130221121135.5a1aabd6@pitrou.net> <512621FF.6070606@trueblade.com> <20130221150049.2731682e@pitrou.net> <51262B08.3030600@trueblade.com> <1E2DA813-A8AF-46CF-B7D4-5A1C1953B781@yahoo.com> <0f836a6d49664d2cb93c97bf496bab90@BLUPR03MB035.namprd03.prod.outlook.com> <87mwuxseoq.fsf@uwakimon.sk.tsukuba.ac.jp> <87ip5ksf9p.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Fri, Feb 22, 2013 at 11:08 AM, Bruce Leban wrote: > To take this back to the ideas stage, one idea might be to integrate hmac > into pickle. At a minimum, provide some sample code showing how to wrap an > hmac around a pickled object. This sounds very much like the `itsdangerous` library (which uses JSON by default, but the serializer backend is pluggable): http://pythonhosted.org/itsdangerous/ Cheers, Chris From storchaka at gmail.com Tue Feb 26 10:19:00 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 26 Feb 2013 11:19:00 +0200 Subject: [Python-ideas] Add lookahead iterator (peeker) to itertools In-Reply-To: <20130225212722.0d6de65b@pitrou.net> References: <20130225110717.232ce082@pitrou.net> <20130225195302.45704b36@pitrou.net> <20130225212722.0d6de65b@pitrou.net> Message-ID: On 25.02.13 22:27, Antoine Pitrou wrote: > On Mon, 25 Feb 2013 22:21:03 +0200 > Serhiy Storchaka > wrote: >> On 25.02.13 20:53, Antoine Pitrou wrote: >>>>> def test_lookahead(): >>>>> it = iter('abc') >>>>> while True: >>>>> it, peeking = itertools.tee(it) >>>> >>>> This should be outside a loop. >>> >>> Only if you restrict yourself to access peeking each time you access it. >>> (which, I suppose, is not the general use case for the lookahead >>> proposal) >> >> Only if your do not want to consume O(N) memory and spend O(N**2) time. > > No, that's beside the point. If you don't consume "peeking" in > lock-step with "it", then "peeking" and "it" become desynchronized and > therefore the semantics are wrong w.r.t to the original feature > request (where "peeking" is supposed to be some proxy to "it", not an > independently-running iterator). Yes, of course, you should consume "peeking" in lock-step with "it". My note is that if you create tee every iteration, this will lead to an linear increase in memory consumption and degradation of speed. From masklinn at masklinn.net Tue Feb 26 10:23:03 2013 From: masklinn at masklinn.net (Masklinn) Date: Tue, 26 Feb 2013 10:23:03 +0100 Subject: [Python-ideas] Add lookahead iterator (peeker) to itertools In-Reply-To: References: <20130225110717.232ce082@pitrou.net> <20130225195302.45704b36@pitrou.net> <20130225212722.0d6de65b@pitrou.net> Message-ID: On 2013-02-26, at 10:19 , Serhiy Storchaka wrote: > On 25.02.13 22:27, Antoine Pitrou wrote: >> On Mon, 25 Feb 2013 22:21:03 +0200 >> Serhiy Storchaka >> wrote: >>> On 25.02.13 20:53, Antoine Pitrou wrote: >>>>>> def test_lookahead(): >>>>>> it = iter('abc') >>>>>> while True: >>>>>> it, peeking = itertools.tee(it) >>>>> >>>>> This should be outside a loop. >>>> >>>> Only if you restrict yourself to access peeking each time you access it. >>>> (which, I suppose, is not the general use case for the lookahead >>>> proposal) >>> >>> Only if your do not want to consume O(N) memory and spend O(N**2) time. >> >> No, that's beside the point. If you don't consume "peeking" in >> lock-step with "it", then "peeking" and "it" become desynchronized and >> therefore the semantics are wrong w.r.t to the original feature >> request (where "peeking" is supposed to be some proxy to "it", not an >> independently-running iterator). > > Yes, of course, you should consume "peeking" in lock-step with "it". > > My note is that if you create tee every iteration, this will lead to an linear increase in memory consumption and degradation of speed. I believe I also saw situations where I blew through the recursion limit by stacking too many tees, but I can't reproduce it right now. From storchaka at gmail.com Tue Feb 26 10:44:36 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 26 Feb 2013 11:44:36 +0200 Subject: [Python-ideas] Add lookahead iterator (peeker) to itertools In-Reply-To: References: <20130225110717.232ce082@pitrou.net> <20130225195302.45704b36@pitrou.net> <20130225212722.0d6de65b@pitrou.net> Message-ID: On 26.02.13 11:23, Masklinn wrote: > On 2013-02-26, at 10:19 , Serhiy Storchaka wrote: >> My note is that if you create tee every iteration, this will lead to an linear increase in memory consumption and degradation of speed. > > I believe I also saw situations where I blew through the recursion limit > by stacking too many tees, but I can't reproduce it right now. http://bugs.python.org/issue13454 ? From masklinn at masklinn.net Tue Feb 26 11:06:45 2013 From: masklinn at masklinn.net (Masklinn) Date: Tue, 26 Feb 2013 11:06:45 +0100 Subject: [Python-ideas] Add lookahead iterator (peeker) to itertools In-Reply-To: References: <20130225110717.232ce082@pitrou.net> <20130225195302.45704b36@pitrou.net> <20130225212722.0d6de65b@pitrou.net> Message-ID: <5C12F640-6C24-49E7-A6CD-56A0DFC9199E@masklinn.net> On 2013-02-26, at 10:44 , Serhiy Storchaka wrote: > On 26.02.13 11:23, Masklinn wrote: >> On 2013-02-26, at 10:19 , Serhiy Storchaka wrote: >>> My note is that if you create tee every iteration, this will lead to an linear increase in memory consumption and degradation of speed. >> >> I believe I also saw situations where I blew through the recursion limit >> by stacking too many tees, but I can't reproduce it right now. > > http://bugs.python.org/issue13454 ? No, it was a recursion limit, I was trying to use iterators in a situation where I needed multiline lookaheads and skipping so I stacked at least 1 tee + 1 islice per item and it blew up at one point. I can't reproduce it anymore and I can't find the original code (replacing it with straight lists ended up being simpler) Although I can trivially get a segfault with islice: from itertools import islice, count it = count() while True: it = islice(it, 0) Run this, C-c at some point, every CPython version on my machine segfaults (pypy does not) From storchaka at gmail.com Tue Feb 26 11:37:25 2013 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 26 Feb 2013 12:37:25 +0200 Subject: [Python-ideas] Add lookahead iterator (peeker) to itertools In-Reply-To: <5C12F640-6C24-49E7-A6CD-56A0DFC9199E@masklinn.net> References: <20130225110717.232ce082@pitrou.net> <20130225195302.45704b36@pitrou.net> <20130225212722.0d6de65b@pitrou.net> <5C12F640-6C24-49E7-A6CD-56A0DFC9199E@masklinn.net> Message-ID: On 26.02.13 12:06, Masklinn wrote: > Although I can trivially get a segfault with islice: > > from itertools import islice, count > > it = count() > while True: > it = islice(it, 0) > > Run this, C-c at some point, every CPython version on my machine > segfaults (pypy does not) Thank you. This looks as pretty same issue as with tee(). I knew that there must be more such catches, but could not find them. http://bugs.python.org/issue17300 From masklinn at masklinn.net Tue Feb 26 11:57:45 2013 From: masklinn at masklinn.net (Masklinn) Date: Tue, 26 Feb 2013 11:57:45 +0100 Subject: [Python-ideas] Add lookahead iterator (peeker) to itertools In-Reply-To: References: <20130225110717.232ce082@pitrou.net> <20130225195302.45704b36@pitrou.net> <20130225212722.0d6de65b@pitrou.net> <5C12F640-6C24-49E7-A6CD-56A0DFC9199E@masklinn.net> Message-ID: <361FF751-43F2-48D1-B4CB-3C4D76E70608@masklinn.net> On 2013-02-26, at 11:37 , Serhiy Storchaka wrote: > On 26.02.13 12:06, Masklinn wrote: >> Although I can trivially get a segfault with islice: >> >> from itertools import islice, count >> >> it = count() >> while True: >> it = islice(it, 0) >> >> Run this, C-c at some point, every CPython version on my machine >> segfaults (pypy does not) > > Thank you. This looks as pretty same issue as with tee(). I knew that there must be more such catches, but could not find them. > > http://bugs.python.org/issue17300 > Cool. And I looked back in the VCS, turns out the code hasn't been lost but the issue was not in the stdlib, it was a custom iterator (used as a wrapper for a bunch of operations) which needed to be reapplied very often (code basically went Iterator -> mix of tee, chain and dropwhile -> Iterator -> same mix), essentially doing the following: import itertools _placeholder = object() class It(object): def __init__(self, stream): self.stream = iter(stream) self.stopped = False def __iter__(self): return self def __next__(self): if self.stopped: raise StopIteration() val = next(self.stream, _placeholder) if val is _placeholder: self.stopped = True raise StopIteration() return val it = itertools.count() while True: it = It(it) next(it) I'm not sure if there's any way to implement such a wrapping iterator in a way which does not ultimately blow the stack (save in C taking after itertools implementations I guess as they don't seem to have the issue) From solipsis at pitrou.net Tue Feb 26 12:15:22 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 26 Feb 2013 12:15:22 +0100 Subject: [Python-ideas] Add lookahead iterator (peeker) to itertools References: <20130225110717.232ce082@pitrou.net> <20130225195302.45704b36@pitrou.net> <20130225212722.0d6de65b@pitrou.net> Message-ID: <20130226121522.55d3fcae@pitrou.net> Le Tue, 26 Feb 2013 11:19:00 +0200, Serhiy Storchaka a ?crit : > On 25.02.13 22:27, Antoine Pitrou wrote: > > On Mon, 25 Feb 2013 22:21:03 +0200 > > Serhiy Storchaka > > wrote: > >> On 25.02.13 20:53, Antoine Pitrou wrote: > >>>>> def test_lookahead(): > >>>>> it = iter('abc') > >>>>> while True: > >>>>> it, peeking = itertools.tee(it) > >>>> > >>>> This should be outside a loop. > >>> > >>> Only if you restrict yourself to access peeking each time you > >>> access it. (which, I suppose, is not the general use case for the > >>> lookahead proposal) > >> > >> Only if your do not want to consume O(N) memory and spend O(N**2) > >> time. > > > > No, that's beside the point. If you don't consume "peeking" in > > lock-step with "it", then "peeking" and "it" become desynchronized > > and therefore the semantics are wrong w.r.t to the original feature > > request (where "peeking" is supposed to be some proxy to "it", not > > an independently-running iterator). > > Yes, of course, you should consume "peeking" in lock-step with "it". > > My note is that if you create tee every iteration, this will lead to > an linear increase in memory consumption and degradation of speed. Apparently, itertools.tee() is optimized for tee'ing a tee: >>> it = iter("abc") >>> id(it) 31927760 >>> it, peeking = itertools.tee(it) >>> id(it) 31977128 >>> it, peeking = itertools.tee(it) >>> id(it) 31977128 Regards Antoine. From demianbrecht at gmail.com Tue Feb 26 22:18:21 2013 From: demianbrecht at gmail.com (Demian Brecht) Date: Tue, 26 Feb 2013 13:18:21 -0800 Subject: [Python-ideas] FileCookieJar (and concrete implementations). Message-ID: Context: http://bugs.python.org/issue16942 (my patch, changing FileCookieJar to be an abc, defining the interfaces for *FileCookieJar). This pertains to Terry's question about whether or not it makes sense that an abstract base class extends a concrete class. After putting in a little thought, he's right. It doesn't make sense. After further thought, I'm relatively confident that the hierarchy as it stands should be changed. Currently what's implemented in the stdlib looks like this: CookieJar | FileCookieJar | | | MozillaCookieJar LWPCookieJar What I'm proposing is that the structure is broken to be the following: FileCookieJarProcessor CookieJar | | | MozillaCookieJarProcessor LWPCookieJarProcessor The intention here is to have processors that operate /on/ a cookiejar object via composition rather than inheritance. This aligns with how urllib.request.HTTPCookieProcessor works (which has the added bonus of cross-module consistency). The only attributes that concrete FileCookieJarProcessor classes touch (at least, in the stdlib) are _cookies and _cookies_lock. I have mixed feelings about whether these should continue to be noted as "non-public" with the _ prefix or not as keeping the _ would break convention of operating on non-public fields, but am unsure of the ramifications of changing them to public. Making this change then allows for FileCookieJar(Processor) to be an abstract base class without inheriting from CookieJar which doesn't make a whole lot of sense from an architecture standpoint. I have yet to see what impact these changes have to the cookiejar extensions at http://wwwsearch.sf.net but plan on doing so if this approach seems sound. This will obviously break backwards compatibility, so I'm not entirely sure what best practice is around that: leave well enough alone even though it might not make sense, keep the old implementations around and deprecate them to be eventually replaced by the processors, or other ideas? -- Demian Brecht http://demianbrecht.github.com From tjreedy at udel.edu Tue Feb 26 22:57:36 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 26 Feb 2013 16:57:36 -0500 Subject: [Python-ideas] Add lookahead iterator (peeker) to itertools In-Reply-To: <512B8370.6040807@gmail.com> References: <512B8370.6040807@gmail.com> Message-ID: On 2/25/2013 10:29 AM, Ron Adam wrote: > > > On 02/24/2013 09:41 PM, Terry Reedy wrote: >> class lookahead(): >> "Wrap iterator with lookahead to both peek and test exhausted" >> >> _NONE = object() >> def __init__(self, iterable): >> self._it = iter(iterable) >> self._set_peek() >> def __iter__(self): >> return self >> def __next__(self): >> ret = self.peek >> self._set_peek() >> return ret >> def _set_peek(self): >> try: >> self.peek = next(self._it) >> except StopIteration: >> self.peek = self._NONE >> def __bool__(self): >> return self.peek is not self._NONE > >> >> def test_lookahead(): >> it = lookahead('abc') >> while it: >> a = it.peek >> b = next(it) >> print('next:', b, '; is peek:', a is b ) >> >> test_lookahead() Since revised. > I think with a few small changes I would find it useful. > > The key feature here is that the result is pre calculated and held until > it's needed, rather than calculated when it's asked for. > > You should catch any exception and hold that as well. On the next > .next() call, it should raise the exception if there was one, or emit > the value. My revised version has def _set_peek(self): try: self.peek = next(self._it) except StopIteration: self.peek = self._NONE That could easily be changed to e except Exception as e: self.peek = self._NONE self._error = e __next would then raise self._error instead of explicitly StopIteration. I do not especially like the redundancy of two 'exhausted' indicators and thought of storing e as self.peek, but an iterator can legitimately yield exception classes and instances. I think some argument can be made that if the iterator is broken, the exception should be raised immediately even if it means not returning the last item. No user should be expecting anything other than StopIteration. > I'm not sure if using the __bool__ attribute is the best choice. I am ;-). It hides the test for exhaustion, which could change, without complication. > I would prefer a .error flag, along with a .next_value attribute. It > would make the code using it easier to follow. Not clear to me, but a minor detail. > it.error <-- True if next(it) will raise an exception. > it.next_value <-- The next value, or the exception to raise. > About it.error. If it was a concurrent version, then it.error could > have three values. > > it.error == True # Will raise an exception > it.error == False # Will not raise an exception > it.error == None # Still calculating > > I wonder how this type of generator will behave with "yield from". lookaheads are iterators, but not generators and 'yield from' requires a generator. A generator function could recompute items to yield, but it cannot add methods or attributes to the generator instances it will produce. -- Terry Jan Reedy From tjreedy at udel.edu Wed Feb 27 01:44:26 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 26 Feb 2013 19:44:26 -0500 Subject: [Python-ideas] iterable.__unpack__ method In-Reply-To: References: <44528db0ff81ea147dfc71d97e472798@chopin.edu.pl> <512A7967.1060709@mrabarnett.plus.com> Message-ID: On 2/25/2013 5:25 PM, Alex Stewart wrote: Negative putdowns are off-topic and not persuasive. I starting with 1.3 in March 1997 and first posted a month later. While cautious about changes and additions, I have mostly been in favor of those that have been released. I started using 3.0 in beta and starting using 3.3.0 with the first alpha to get the new stuff. > On Sunday, February 24, 2013 7:43:58 PM UTC-8, Terry Reedy wrote: > The related but distinct concepts of sequential access and random > access > are basic to computation and were *always* distinct in Python. > > Oh really? Yes really! > Then how, prior to the development of the iterator protocol, > did one define an object which was accessible sequentially but not > randomly in the Python language? As I said, by using the original fake-getitem iterator protocol, which still works, instead of the newer iter-next iterator protocol. Take any python-coded iterator class, such as my lookahead class. Remove or comment out the 'def __iter__(self): return self' statement. Change the header line 'def __next__(self):' to 'def __getitem__(self, n):'. Instances of the revised class will *today* work with for statements. Doing this with lookahead (3.3): >>> for item in lookahead('abc'): print(item) a b c >>> dir(lookahead) ['_NONE', '__bool__', '__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__gt__', '__hash__', '__init__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__qualname__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_set_peek'] See, no __iter__, no __next__, but a __getitem__ that ignores the index passed. Here is where it gets a bit crazy. >>> s = lookahead('abc') >>> s[0] 'a' >>> s[0], s[300] ('b', 'c') >>> s[0] Traceback... StopIteration Still, I think enabling generators was more of a motivation than discouraging nonsense like the above (which, obviously, is still possible! but also which people did not intentionally do except as a demonstration). A practical difference between sequential and random access is that sequential access usually requires history ('state'), which random access does not. The __iter__ method and iter() function allows the separation of iterators, with iteration state, from an underlying concrete collections (if there is one), which usually does not need the iteration state. (Files, which *are* iterators and which do not have a separate iterator class, are unusual among builtins.) This enables multiple stateful iterators and nested for loops like this: >>> s = 'abc' >>> for c in lookahead(s): for d in lookahead(s): print((c,d)) ('a', 'a') ('a', 'b') ('a', 'c') ('b', 'a') ('b', 'b') ('b', 'c') ('c', 'a') ('c', 'b') ('c', 'c') > If you couldn't do that, you can't > claim that the concepts were really distinct in Python, in my opinion. But I and anyone could and still can, so I can and will make the claim and reject arguments based on the false counter-claim. --- Back to unpack and informing the source as to the number n of items expected to be requested. Case 1. The desired response of the source to n is generic. Example a: we want the source to simply refuse to produce more than the number specified at the start. The current generic solutions are to either make the exact number of explicit next() calls needed or to wrap the iterator in islice(iterator, n) (which in turn will make the number of next() calls needed). Example b : if the source does yield n items, we want it to then yield the residual iterator. Again, a simple generic wrapper does the job. import itertools def slice_it(iterable, n): "Yield up to n items from iterable and if successful, the residual iterator:" it = iter(iterable) for i in range(n): yield next(it) else: yield it a, b, c, rest = slice_it(itertools.count(), 3) print(a, b, c, rest) d, e = itertools.islice(rest, 2) print(d, e) >>> 0 1 2 count(3) 3 4 Conclusion: generic iterator behavior modification should be done by generic wrappers. This is the philosophy and practice of comprehensions, built-in wrappers (enumerate, filter, map, reversed, and zip), and itertools. Case 2. The desired response is specific to a class or even each instance. Example: object has attributes a, b, c, d (of lesser importance), and can calculate e. The pieces and structures it should yield depend on n as in the following table. 1 ((a,b),c) 2 (a,b), c 3 a, b, c 4 a, b, c, d 5 a, b, c, d, e Solution: write a method, call it .unpack(n), that returns an iterator that will produce the objects specified in the table. This can be done today with no change to Python. It can be done whether or not there is a .__iter__ method to produce a generic default iterator for the object. And, of course, xxx.unpack can have whatever signature is appropriate to xxx. It seems to me that this procedure can handle any special collection or structure breakup need. Comment 1: if people can pass explicit counts to islice (or slice_it), they can pass explicit counts to .unpack. Comment 2: we are not disagreeing that people might want to do custom count-dependent disassembly or that they should be able to do so. It can already be done. Areas of disagreement: 1. consumer-source interdependency: you seem to think there is something special about the consumer assigning items to multiple targets in one statement, as opposed to doing anything else, including doing the multiple assignments in multiple statements. I do not. Moreover, I so far consider introducing such dependency in the core to be a regression. 2. general usefulness: you want .unpack to be standardized and made a special method. I think it is inherently variable enough and and the need rare enough to not justify that. I retrieved your original post and plan to look at it sometime to see if I missed anything not covered by the above. But enough for today. -- Terry Jan Reedy From ron3200 at gmail.com Wed Feb 27 02:50:17 2013 From: ron3200 at gmail.com (Ron Adam) Date: Tue, 26 Feb 2013 19:50:17 -0600 Subject: [Python-ideas] Add lookahead iterator (peeker) to itertools In-Reply-To: References: <512B8370.6040807@gmail.com> Message-ID: <512D6659.1080802@gmail.com> On 02/26/2013 03:57 PM, Terry Reedy wrote: >> I would prefer a .error flag, along with a .next_value attribute. It >> would make the code using it easier to follow. > > Not clear to me, but a minor detail. > >> it.error <-- True if next(it) will raise an exception. >> it.next_value <-- The next value, or the exception to raise. > > >> About it.error. If it was a concurrent version, then it.error could >> have three values. >> >> it.error == True # Will raise an exception >> it.error == False # Will not raise an exception >> it.error == None # Still calculating >> >> I wonder how this type of generator will behave with "yield from". > > lookaheads are iterators, but not generators and 'yield from' requires a > generator. A generator function could recompute items to yield, but it > cannot add methods or attributes to the generator instances it will produce. Yep, I missed that point. I was thinking it may be more useful in the case of generators along with yield from, but it isn't as straight forward as the iterator case as you correctly pointed out. (More or less just thinking out loud at this point.) My intuition/feelings was that in both the cases of a look-ahead iterator and a calc-ahead generator, (if we could manage that), should act the same in as many ways as possible. And a calc-ahead generator should act like a concurrent instance. The reason for thinking that is, it would allow the calc-ahead generator instance to be replaced with a concurrent instance without changing anything. So if (in theory) a concurrent generator instance wouldn't hold exceptions, then your look-ahead iterator shouldn't also. But if everything happens on the next and send calls, then it makes handling exceptions a bit easier as you don't have to special case the instance creation part. But iterators don't necessarily need to match all generator behaviours, It's just may preference that they do as much as possible. :-) Of course Guido may have something in what he's doing now that would be very much like this. A kind of "futures" generator. I've been away from these boards for a while and haven't caught up with everything yet. Cheers, Ron From raymond.hettinger at gmail.com Wed Feb 27 05:46:29 2013 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Tue, 26 Feb 2013 20:46:29 -0800 Subject: [Python-ideas] range In-Reply-To: References: <20130222150557.36884c3f@pitrou.net> <20130222194310.75129c1f@pitrou.net> Message-ID: On Feb 23, 2013, at 3:11 AM, Nick Coghlan wrote: > I suspect for many cases involving more advanced > range variants, containment would be a better option than inheritance, > particular if you want to implement a type that can't be reliably > described through a (start, stop, step) triple. I concur. Subclassing range() is just asking for trouble. Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From tkluck at infty.nl Wed Feb 27 12:27:14 2013 From: tkluck at infty.nl (Timo Kluck) Date: Wed, 27 Feb 2013 12:27:14 +0100 Subject: [Python-ideas] easier subprocess piping and redirection Message-ID: Dear python enthousiasts, While replacing some of my bash scripts by python scripts, I found the following useful. I wrote a small utility module that allows piping input and output in a very similar way to how you do it in bash. The major advantage over just using shell=True is that this does not expose us to the latter's security risks and quoting hazards. The major advantage of using a pipe instead of just manually feeding the output of one process to the other is that with a pipe, it doesn't all have to fit into memory at once. However, the Python way of doing that is rather cumbersome: >>> from subprocess import Popen, PIPE >>> p1 = Popen(['echo', 'a\nb'], stdout=PIPE) >>> p2 = Popen([head], '-n1'], stdin=p1.stdout, stdout=PIPE) >>> p2.communicate() ('a\n', None) I've been able to replace this by: >>> from pipeutils import call >>> (call('echo', 'a\nb') | call('head', '-n1')).output() 'a\n' And similarly, I can do direct file redirects like this: >>> (call('echo', 'Hello world!') > 'test.txt').do() 0 >>> (call('cat') < 'test.txt').output() 'Hello world!\n' I think that this is a lot more concise and readable. As far as I'm concerned, this was the only reason I would ever use bash scripts instead of python scripts. I would love to hear opinions on this. How would people like this as a library? The source is online at https://bitbucket.org/tkluck/pipeutils Best, Timo From stefan at drees.name Wed Feb 27 12:57:41 2013 From: stefan at drees.name (Stefan Drees) Date: Wed, 27 Feb 2013 12:57:41 +0100 Subject: [Python-ideas] easier subprocess piping and redirection In-Reply-To: References: Message-ID: <512DF4B5.5040707@drees.name> Hi Timo, On 27.02.13 12:27, Timo Kluck wrote: > Dear python enthousiasts, > > While replacing some of my bash scripts by python scripts, I found the > following useful. I wrote a small utility module that allows piping > input and output in a very similar way to how you do it in bash. The > major advantage over just using shell=True is that this does not > expose us to the latter's security risks and quoting hazards. > > The major advantage of using a pipe instead of just manually feeding > the output of one process to the other is that with a pipe, it doesn't > all have to fit into memory at once. However, the Python way of doing > that is rather cumbersome: > > >>> from subprocess import Popen, PIPE > >>> p1 = Popen(['echo', 'a\nb'], stdout=PIPE) > >>> p2 = Popen([head], '-n1'], stdin=p1.stdout, stdout=PIPE) > >>> p2.communicate() > ('a\n', None) > > I've been able to replace this by: > > >>> from pipeutils import call > >>> (call('echo', 'a\nb') | call('head', '-n1')).output() > 'a\n' > thanks a lot for sharing this. How does this relate to plumbum :-? >>> from plumbum.cmd import echo, head >>> chain = echo['a\nb'] | head['-n1'] >>> chain() u'a\n' ($> pip install plumbum) ... http://plumbum.readthedocs.org/en/latest/ > And similarly, I can do direct file redirects like this: > > >>> (call('echo', 'Hello world!') > 'test.txt').do() > 0 > >>> (call('cat') < 'test.txt').output() > 'Hello world!\n' > > I think that this is a lot more concise and readable. As far as I'm > concerned, this was the only reason I would ever use bash scripts > instead of python scripts. > > I would love to hear opinions on this. How would people like this as a library? > > The source is online at https://bitbucket.org/tkluck/pipeutils > ... All the best, Stefan. From tkluck at infty.nl Wed Feb 27 13:20:17 2013 From: tkluck at infty.nl (Timo Kluck) Date: Wed, 27 Feb 2013 13:20:17 +0100 Subject: [Python-ideas] easier subprocess piping and redirection In-Reply-To: <512DF4B5.5040707@drees.name> References: <512DF4B5.5040707@drees.name> Message-ID: 2013/2/27 Stefan Drees : > > thanks a lot for sharing this. How does this relate to plumbum :-? > > >>> from plumbum.cmd import echo, head > >>> chain = echo['a\nb'] | head['-n1'] > >>> chain() > u'a\n' Fantastic! I kind of guessed that it would have existed already, but didn't know where to find it. Thanks for pointing this out! Timo From phd at phdru.name Wed Feb 27 14:23:29 2013 From: phd at phdru.name (Oleg Broytman) Date: Wed, 27 Feb 2013 17:23:29 +0400 Subject: [Python-ideas] easier subprocess piping and redirection In-Reply-To: References: <512DF4B5.5040707@drees.name> Message-ID: <20130227132329.GA18897@iskra.aviel.ru> On Wed, Feb 27, 2013 at 01:20:17PM +0100, Timo Kluck wrote: > 2013/2/27 Stefan Drees : > > > > thanks a lot for sharing this. How does this relate to plumbum :-? > > > > >>> from plumbum.cmd import echo, head > > >>> chain = echo['a\nb'] | head['-n1'] > > >>> chain() > > u'a\n' > > Fantastic! I kind of guessed that it would have existed already, but > didn't know where to find it. Thanks for pointing this out! Very easy -- search for packages at Python Package Index, PyPI: https://pypi.python.org/pypi https://pypi.python.org/pypi?%3Aaction=search&term=pipe&submit=search Add your package to the Index so other people can find it. Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From tjreedy at udel.edu Wed Feb 27 21:43:39 2013 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 27 Feb 2013 15:43:39 -0500 Subject: [Python-ideas] range In-Reply-To: References: <20130222150557.36884c3f@pitrou.net> <20130222194310.75129c1f@pitrou.net> Message-ID: On 2/26/2013 11:46 PM, Raymond Hettinger wrote: > > On Feb 23, 2013, at 3:11 AM, Nick Coghlan > > wrote: > >> I suspect for many cases involving more advanced >> range variants, containment would be a better option than inheritance, >> particular if you want to implement a type that can't be reliably >> described through a (start, stop, step) triple. > > I concur. Subclassing range() is just asking for trouble. The four builtins that cannot be subclassed are bool, memoryview, range, slice. This is documented for bool; I propose to do so for the others in http://bugs.python.org/issue17279. Would it be correct to say (now) that all 4 are intentional omissions? and not merely oversights? -- Terry Jan Reedy From foogod at gmail.com Wed Feb 27 21:58:47 2013 From: foogod at gmail.com (Alex Stewart) Date: Wed, 27 Feb 2013 12:58:47 -0800 Subject: [Python-ideas] iterable.__unpack__ method In-Reply-To: References: <44528db0ff81ea147dfc71d97e472798@chopin.edu.pl> <512A7967.1060709@mrabarnett.plus.com> Message-ID: On Tue, Feb 26, 2013 at 4:44 PM, Terry Reedy wrote: > On 2/25/2013 5:25 PM, Alex Stewart wrote: > > Negative putdowns are off-topic and not persuasive. None of my statements were intended to be put-downs, and if it came across that way I apologize. It is possible that my frustration showed through on occasion and my language was harsher than it should have been.. > I starting with 1.3 in March 1997 and first posted a month later. For what it's worth (not that I think it's terribly important, really), while I have not generally participated much in online discussions over the years, I actually started using Python about the same time you did, and have watched it evolve over a similar timespan.. > Then how, prior to the development of the iterator protocol, > > did one define an object which was accessible sequentially but not >> randomly in the Python language? >> > > As I said, by using the original fake-getitem iterator protocol, which > still works, instead of the newer iter-next iterator protocol. > [large snip] Ok, yes, it was technically possible with ugly hacks to do it, but I don't consider that to be really the same as "supported by the language". The only way you could do it was by making an object "fake" a random-access interface that actually likely did the wrong thing if accessed randomly. That is not a separation of concepts, that is a twisting of one concept in an incompatible way into the semblance of another as a workaround to the fact that the two concepts are not actually distinct in the language design. Once again, you're confusing "how it's thought of in the particular programmer's mind" with "how the language is actually designed", which are not the same thing. Back to unpack and informing the source as to the number n of items > expected to be requested. > > Case 1. The desired response of the source to n is generic. > If I'm reading you right, I believe this is basically equivalent to "the consumer decides what behavior is most useful". In this case, yes, the correct place to do this is on the consumer side, which is not what __unpack__ is intended to be for anyway (fairly obviously, since it is not a consumer-side change). (For what it's worth, you seem to do a decent job here of arguing against your own propositions to change the unpack syntax, though.. Not sure if that was intentional..) > Case 2. The desired response is specific to a class or even each instance. > [...] Solution: write a method, call it .unpack(n), that returns an iterator that > will produce the objects specified in the table. This can be done today > with no change to Python. It can be done whether or not there is a > .__iter__ method to produce a generic default iterator for the object. And, > of course, xxx.unpack can have whatever signature is appropriate to xxx. It > seems to me that this procedure can handle any special collection or > structure breakup need. > This solution works fine, in the very restrictive case where the programmer knows exactly what type of object they've been given to work with, is aware that it provides this interface, and knows that it's important in that particular case to use it for that particular type of object. If the consumer wants to be able to use both this type of object and other ones that don't provide that interface, then their code suddenly becomes a lot more complicated (having to check the type, or methods of the object, and conditionally do different things). Even if they decide to do that, since there is no established standard (or even convention) for this sort of thing, they then run the risk of being given an object which has an "unpack" method with a completely different signature, or worse yet an object which defines "unpack" that does some completely different operation, unrelated to this ad-hoc protocol. Any time somebody produces a new "smart unpackable" object that doesn't work quite the same as the others (either deliberately, or just because the programmer didn't know that other people were already doing the same thing in a different way), it is quite likely that all of the existing consumer code everywhere will have to be rewritten to support it, or (much more likely) it will be supported haphazardly some places and not others, leading to inconsistent behavior or outright bugs. Even ignoring all of this, it still isn't possible to write an object which has advanced unpacking behavior that works with any existing unpacking consumers, such as libraries or other code not under the object-designer's control. In short, yes, it solves things, in a very limited, incompatible, painful, and largely useless way. Comment 2: we are not disagreeing that people might want to do custom > count-dependent disassembly or that they should be able to do so. It can > already be done. > I disagree that it can already be done in the manner I'm describing, as I've explained. There is, frankly, no existing mechanism which allows an unpacking-producer to do this sort of thing in a standard, consistent, and interchangeable way, and there is also no way at all to do it in a way that is compatible with existing consumer code. > Areas of disagreement: > > 1. consumer-source interdependency: you seem to think there is something > special about the consumer assigning items to multiple targets in one > statement, as opposed to doing anything else, including doing the multiple > assignments in multiple statements. This is not a matter of opinion. It is a *fact* that there is a difference in this case. The difference is, quite simply, that through the use of the unpacking construct, the programmer has given the Python interpreter additional information (which the interpreter is not providing to the producing object). The disagreement appears to be that you believe for some reason it's a good thing to silently discard this information so that nobody can make use of it, whereas I believe it would be beneficial to make it available for those who want to use it. Again, I feel compelled to point out that as far as I can tell, your entire objection on this count boils down to "unpacking must always be the same thing as iteration because that's the way I've always thought about it". Unpacking *currently* uses iteration under the covers because it is a convenient interface that already exists in the language, but there is absolutely no reason why unpacking *must* inherently be defined as the same thing as iteration. You talk about it as if this is a foregone conclusion, but as far as I can tell it's only foregone because you've already arbitrarily decided it to be one way and just won't listen to anybody suggesting anything else. Alternately, if you just can't manage to get past this "unpacking must mean iteration" thing, then don't look at this as a change to unpacking. Look at it instead as an extension to the iterator protocol, which allows an iteration consumer to tell the iteration producer more information about the number of items they are wanting to obtain from it. Heck, I actually wouldn't be opposed to making this a general feature of iter(), if it could be done in a backwards-compatible way.. 2. general usefulness: you want .unpack to be standardized and made a > special method. I think it is inherently variable enough and and the need > rare enough to not justify that. > I think I already spoke to this above.. Put simply, if it is not standardized and utilized by the corresponding language constructs, it is essentially useless as a general-purpose solution and only works in very limited cases. Your alternative solutions just aren't solutions to the more general problems. --Alex -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericsnowcurrently at gmail.com Thu Feb 28 11:15:14 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Thu, 28 Feb 2013 03:15:14 -0700 Subject: [Python-ideas] OrderedDict for kwargs and class statement namespace Message-ID: Just wanted to put out some feelers for the feasibility of these two features: * have the **kwargs param be an OrderedDict rather than a dict * have class definitions happen relative to an OrderedDict by default rather than a dict, and still overridable by a metaclass's __prepare__(). Both of these will need OrderedDict in C, which is getting close (issue #16991). -eric -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Thu Feb 28 11:27:42 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 28 Feb 2013 11:27:42 +0100 Subject: [Python-ideas] OrderedDict for kwargs and class statement namespace References: Message-ID: <20130228112742.09f5500b@pitrou.net> Le Thu, 28 Feb 2013 03:15:14 -0700, Eric Snow a ?crit : > Just wanted to put out some feelers for the feasibility of these two > features: > > * have the **kwargs param be an OrderedDict rather than a dict Rather than just feasibility, I would like performance not to regress here. > Both of these will need OrderedDict in C, which is getting close > (issue #16991). Really? Last time I looked, it wasn't getting really close. Regards Antoine. From ericsnowcurrently at gmail.com Thu Feb 28 15:55:03 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Thu, 28 Feb 2013 07:55:03 -0700 Subject: [Python-ideas] OrderedDict for kwargs and class statement namespace In-Reply-To: <20130228112742.09f5500b@pitrou.net> References: <20130228112742.09f5500b@pitrou.net> Message-ID: On Feb 28, 2013 3:28 AM, "Antoine Pitrou" wrote: > > Le Thu, 28 Feb 2013 03:15:14 -0700, > Eric Snow a > ?crit : > > Just wanted to put out some feelers for the feasibility of these two > > features: > > > > * have the **kwargs param be an OrderedDict rather than a dict > > Rather than just feasibility, I would like performance not to regress > here. > > > Both of these will need OrderedDict in C, which is getting close > > (issue #16991). > > Really? Last time I looked, it wasn't getting really close. Everything's there and there are just a few lingering memory-related issues to iron out. So 50% done then... -eric -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Thu Feb 28 16:14:37 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 28 Feb 2013 16:14:37 +0100 Subject: [Python-ideas] OrderedDict for kwargs and class statement namespace References: <20130228112742.09f5500b@pitrou.net> Message-ID: <20130228161437.57f15090@pitrou.net> Le Thu, 28 Feb 2013 07:55:03 -0700, Eric Snow a ?crit : > On Feb 28, 2013 3:28 AM, "Antoine Pitrou" > wrote: > > > > Le Thu, 28 Feb 2013 03:15:14 -0700, > > Eric Snow > > a > > ?crit : > > > Just wanted to put out some feelers for the feasibility of these > > > two features: > > > > > > * have the **kwargs param be an OrderedDict rather than a dict > > > > Rather than just feasibility, I would like performance not to > > regress here. > > > > > Both of these will need OrderedDict in C, which is getting close > > > (issue #16991). > > > > Really? Last time I looked, it wasn't getting really close. > > Everything's there and there are just a few lingering memory-related > issues to iron out. So 50% done then... And it also has to be reviewed in deep. To quote you: "The memory-related issues are pushing well past my experience". Regards Antoine. From ncoghlan at gmail.com Thu Feb 28 16:20:02 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 1 Mar 2013 01:20:02 +1000 Subject: [Python-ideas] range In-Reply-To: References: <20130222150557.36884c3f@pitrou.net> <20130222194310.75129c1f@pitrou.net> Message-ID: On Thu, Feb 28, 2013 at 6:43 AM, Terry Reedy wrote: > On 2/26/2013 11:46 PM, Raymond Hettinger wrote: >> >> >> On Feb 23, 2013, at 3:11 AM, Nick Coghlan >> > > wrote: >> >>> I suspect for many cases involving more advanced >>> range variants, containment would be a better option than inheritance, >>> particular if you want to implement a type that can't be reliably >>> described through a (start, stop, step) triple. >> >> >> I concur. Subclassing range() is just asking for trouble. > > > The four builtins that cannot be subclassed are bool, memoryview, range, > slice. This is documented for bool; I propose to do so for the others in > http://bugs.python.org/issue17279. Would it be correct to say (now) that all > 4 are intentional omissions? and not merely oversights? Yes, I think so. People will have to be *real* convincing to explain a case where composition isn't a more appropriate solution Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From donspauldingii at gmail.com Thu Feb 28 16:30:50 2013 From: donspauldingii at gmail.com (Don Spaulding) Date: Thu, 28 Feb 2013 09:30:50 -0600 Subject: [Python-ideas] OrderedDict for kwargs and class statement namespace In-Reply-To: <20130228112742.09f5500b@pitrou.net> References: <20130228112742.09f5500b@pitrou.net> Message-ID: On Thu, Feb 28, 2013 at 4:27 AM, Antoine Pitrou wrote: > Le Thu, 28 Feb 2013 03:15:14 -0700, > Eric Snow a > ?crit : > > Just wanted to put out some feelers for the feasibility of these two > > features: > > > > * have the **kwargs param be an OrderedDict rather than a dict > > Rather than just feasibility, I would like performance not to regress > here. > > I can appreciate not wanting to see "Python 3.4 killed our performance" articles all over news.ycombinator.com. But isn't it worth it to have a (small, but acceptable) performance hit for the sake of opening up use cases which are not currently possible in Python without ugly hacks? For an example of the "recommended" way to get the ordering of your class attributes: http://stackoverflow.com/questions/3288107/how-can-i-get-fields-in-an-original-order It seems to me that the "right thing" for python to do when given an ordered list of key=value pairs in a function call or class definition, is to retain the order. So what's an acceptable level of performance regression for the sake of doing things the "right way" here? -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Thu Feb 28 16:48:10 2013 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 28 Feb 2013 16:48:10 +0100 Subject: [Python-ideas] OrderedDict for kwargs and class statement namespace References: <20130228112742.09f5500b@pitrou.net> Message-ID: <20130228164810.74f2d367@pitrou.net> Le Thu, 28 Feb 2013 09:30:50 -0600, Don Spaulding a ?crit : > > For an example of the "recommended" way to get the ordering of your > class attributes: > http://stackoverflow.com/questions/3288107/how-can-i-get-fields-in-an-original-order This is already possible with the __prepare__ magic method. http://docs.python.org/3.4/reference/datamodel.html#preparing-the-class-namespace > It seems to me that the "right thing" for python to do when given an > ordered list of key=value pairs in a function call or class > definition, is to retain the order. So what's an acceptable level of > performance regression for the sake of doing things the "right way" > here? Or, rather, what is the benefit of doing things "the right way"? There are incredibly few cases for relying on the order of key=value pairs in function calls. Regards Antoine. From donspauldingii at gmail.com Thu Feb 28 17:14:48 2013 From: donspauldingii at gmail.com (Don Spaulding) Date: Thu, 28 Feb 2013 10:14:48 -0600 Subject: [Python-ideas] OrderedDict for kwargs and class statement namespace In-Reply-To: <20130228164810.74f2d367@pitrou.net> References: <20130228112742.09f5500b@pitrou.net> <20130228164810.74f2d367@pitrou.net> Message-ID: On Thu, Feb 28, 2013 at 9:48 AM, Antoine Pitrou wrote: > Le Thu, 28 Feb 2013 09:30:50 -0600, > Don Spaulding > a ?crit : > > > > For an example of the "recommended" way to get the ordering of your > > class attributes: > > > http://stackoverflow.com/questions/3288107/how-can-i-get-fields-in-an-original-order > > This is already possible with the __prepare__ magic method. > > http://docs.python.org/3.4/reference/datamodel.html#preparing-the-class-namespace > > Sure. Case in point: Django has been working around it since at least python 2.4. > > It seems to me that the "right thing" for python to do when given an > > ordered list of key=value pairs in a function call or class > > definition, is to retain the order. So what's an acceptable level of > > performance regression for the sake of doing things the "right way" > > here? > > Or, rather, what is the benefit of doing things "the right way"? There > are incredibly few cases for relying on the order of key=value pairs in > function calls. > "If you build it, they will come..." When I originally encountered the need for python to retain the order of kwargs that my caller specified, it surprised me that there wasn't more clamoring for kwargs being an OrderedDict. However, since my development timeline didn't allow for holding the project up while a patch was pushed through python-dev and out into a real python release, I sucked it up, forced my calling code to send in hand-crafted OrderedDicts and called it a day. I think most developers don't even stop to think that the language *could* be different, they just put in the workaround and move on. I think if python stopped dropping the order of kwargs on the floor today, you'd see people start to rely on the order of kwargs tomorrow. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Thu Feb 28 17:41:17 2013 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 28 Feb 2013 08:41:17 -0800 Subject: [Python-ideas] OrderedDict for kwargs and class statement namespace In-Reply-To: References: <20130228112742.09f5500b@pitrou.net> <20130228164810.74f2d367@pitrou.net> Message-ID: <512F88AD.30306@stoneleaf.us> On 02/28/2013 08:14 AM, Don Spaulding wrote: > > > > On Thu, Feb 28, 2013 at 9:48 AM, Antoine Pitrou > wrote: > > Le Thu, 28 Feb 2013 09:30:50 -0600, > Don Spaulding > > a ?crit : > > > > For an example of the "recommended" way to get the ordering of your > > class attributes: > >http://stackoverflow.com/questions/3288107/how-can-i-get-fields-in-an-original-order > > This is already possible with the __prepare__ magic method. > http://docs.python.org/3.4/reference/datamodel.html#preparing-the-class-namespace > > > Sure. Case in point: Django has been working around it since at least python 2.4. > > > It seems to me that the "right thing" for python to do when given an > > ordered list of key=value pairs in a function call or class > > definition, is to retain the order. So what's an acceptable level of > > performance regression for the sake of doing things the "right way" > > here? > > Or, rather, what is the benefit of doing things "the right way"? There > are incredibly few cases for relying on the order of key=value pairs in > function calls. > > > "If you build it, they will come..." > > When I originally encountered the need for python to retain the order of kwargs that my caller specified, it surprised > me that there wasn't more clamoring for kwargs being an OrderedDict. However, since my development timeline didn't > allow for holding the project up while a patch was pushed through python-dev and out into a real python release, I > sucked it up, forced my calling code to send in hand-crafted OrderedDicts and called it a day. I think most developers > don't even stop to think that the language *could* be different, they just put in the workaround and move on. > > I think if python stopped dropping the order of kwargs on the floor today, you'd see people start to rely on the order > of kwargs tomorrow. +1 I'd already be relying on it if it were there. -- ~Ethan~ From guido at python.org Thu Feb 28 18:32:57 2013 From: guido at python.org (Guido van Rossum) Date: Thu, 28 Feb 2013 09:32:57 -0800 Subject: [Python-ideas] OrderedDict for kwargs and class statement namespace In-Reply-To: References: Message-ID: Hm. I write code regularly that takes a **kwargs dict and stores it into some other longer-lasting datastructure. Having that other datastructure suddenly receive OrderedDicts instead of plain dicts might definitely cause some confusion and possible performance issues. And I don't recall ever having wanted to know the order of the kwargs in the call. But even apart from that backwards incompatibility I think this feature is too rarely useful to make it the default behavior -- if anything I want **kwargs to become faster! On Thu, Feb 28, 2013 at 2:15 AM, Eric Snow wrote: > Just wanted to put out some feelers for the feasibility of these two > features: > > * have the **kwargs param be an OrderedDict rather than a dict > * have class definitions happen relative to an OrderedDict by default rather > than a dict, and still overridable by a metaclass's __prepare__(). > > Both of these will need OrderedDict in C, which is getting close (issue > #16991). > > -eric > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > -- --Guido van Rossum (python.org/~guido) From ncoghlan at gmail.com Thu Feb 28 19:02:54 2013 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 1 Mar 2013 04:02:54 +1000 Subject: [Python-ideas] OrderedDict for kwargs and class statement namespace In-Reply-To: References: Message-ID: On 1 Mar 2013 03:34, "Guido van Rossum" wrote: > > Hm. I write code regularly that takes a **kwargs dict and stores it > into some other longer-lasting datastructure. Having that other > datastructure suddenly receive OrderedDicts instead of plain dicts > might definitely cause some confusion and possible performance issues. > And I don't recall ever having wanted to know the order of the kwargs > in the call. > > But even apart from that backwards incompatibility I think this > feature is too rarely useful to make it the default behavior -- if > anything I want **kwargs to become faster And PEP 422 is designed to make it easier to share a common __prepare__ method with different post processing. Cheers, Nick. > > On Thu, Feb 28, 2013 at 2:15 AM, Eric Snow wrote: > > Just wanted to put out some feelers for the feasibility of these two > > features: > > > > * have the **kwargs param be an OrderedDict rather than a dict > > * have class definitions happen relative to an OrderedDict by default rather > > than a dict, and still overridable by a metaclass's __prepare__(). > > > > Both of these will need OrderedDict in C, which is getting close (issue > > #16991). > > > > -eric > > > > > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > http://mail.python.org/mailman/listinfo/python-ideas > > > > > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Feb 28 19:56:51 2013 From: guido at python.org (Guido van Rossum) Date: Thu, 28 Feb 2013 10:56:51 -0800 Subject: [Python-ideas] OrderedDict for kwargs and class statement namespace In-Reply-To: References: Message-ID: But __prepare__ only works for classes, not for methods, right? On Thu, Feb 28, 2013 at 10:02 AM, Nick Coghlan wrote: > > On 1 Mar 2013 03:34, "Guido van Rossum" wrote: >> >> Hm. I write code regularly that takes a **kwargs dict and stores it >> into some other longer-lasting datastructure. Having that other >> datastructure suddenly receive OrderedDicts instead of plain dicts >> might definitely cause some confusion and possible performance issues. >> And I don't recall ever having wanted to know the order of the kwargs >> in the call. >> >> But even apart from that backwards incompatibility I think this >> feature is too rarely useful to make it the default behavior -- if >> anything I want **kwargs to become faster > > And PEP 422 is designed to make it easier to share a common __prepare__ > method with different post processing. > > Cheers, > Nick. > >> >> On Thu, Feb 28, 2013 at 2:15 AM, Eric Snow >> wrote: >> > Just wanted to put out some feelers for the feasibility of these two >> > features: >> > >> > * have the **kwargs param be an OrderedDict rather than a dict >> > * have class definitions happen relative to an OrderedDict by default >> > rather >> > than a dict, and still overridable by a metaclass's __prepare__(). >> > >> > Both of these will need OrderedDict in C, which is getting close (issue >> > #16991). >> > >> > -eric >> > >> > >> > _______________________________________________ >> > Python-ideas mailing list >> > Python-ideas at python.org >> > http://mail.python.org/mailman/listinfo/python-ideas >> > >> >> >> >> -- >> --Guido van Rossum (python.org/~guido) >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> http://mail.python.org/mailman/listinfo/python-ideas -- --Guido van Rossum (python.org/~guido) From christian at python.org Thu Feb 28 20:30:33 2013 From: christian at python.org (Christian Heimes) Date: Thu, 28 Feb 2013 20:30:33 +0100 Subject: [Python-ideas] OrderedDict for kwargs and class statement namespace In-Reply-To: References: Message-ID: Am 28.02.2013 11:15, schrieb Eric Snow: > Just wanted to put out some feelers for the feasibility of these two > features: > > * have the **kwargs param be an OrderedDict rather than a dict > * have class definitions happen relative to an OrderedDict by default > rather than a dict, and still overridable by a metaclass's __prepare__(). > > Both of these will need OrderedDict in C, which is getting close (issue > #16991). Raymond was/is working on a modification of the dict's internal data structure that reduces its memory consumption. IIRC the modification adds partial ordered as a side effect. The keys are ordered in insertion order until keys are removed. http://mail.python.org/pipermail/python-dev/2012-December/123028.html Christian From ericsnowcurrently at gmail.com Thu Feb 28 22:16:02 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Thu, 28 Feb 2013 14:16:02 -0700 Subject: [Python-ideas] OrderedDict for kwargs and class statement namespace In-Reply-To: References: Message-ID: On Thu, Feb 28, 2013 at 10:32 AM, Guido van Rossum wrote: > Hm. I write code regularly that takes a **kwargs dict and stores it > into some other longer-lasting datastructure. Having that other > datastructure suddenly receive OrderedDicts instead of plain dicts > might definitely cause some confusion and possible performance issues. Could you elaborate on what confusion it might cause? As to performance relative to dict, this has definitely been my primary concern. I agree that the impact has to be insignificant for the **kwargs proposal to go anywhere. Obviously OrderedDict in C had better be an improvement over the pure Python version or there isn't much point. However, it goes beyond that in the cases where we might replace current uses of dict with OrderedDict. My plan has been to do a bunch of performance comparison once the implementation is complete and tune it as much as possible with an eye toward the main potential internal use cases. From my point of view, those are **kwargs and class namespace. This is in part why I've brought those two up. For instance, one guidepost I've used is that typically **kwargs is going to be small. However, even for large kwargs I don't want any performance difference to be a problem. > And I don't recall ever having wanted to know the order of the kwargs > in the call. Though it may sound a little odd, the first use case I bumped into was with OrderedDict itself: OrderedDict(a=1, b=2, c=3) There were a few other reasonable use cases mentioned in other threads a while back. I'll dig them up if that would help. > > But even apart from that backwards incompatibility I think this > feature is too rarely useful to make it the default behavior -- if > anything I want **kwargs to become faster! You mean something like possibly not unpacking the **kwargs in a call into another dict? def f(**kwargs): return kwargs d = {'a': 1, 'b': 2} assert d is f(**d) Certainly faster, though definitely a semantic difference. -eric From guido at python.org Thu Feb 28 22:28:15 2013 From: guido at python.org (Guido van Rossum) Date: Thu, 28 Feb 2013 13:28:15 -0800 Subject: [Python-ideas] OrderedDict for kwargs and class statement namespace In-Reply-To: References: Message-ID: On Thu, Feb 28, 2013 at 1:16 PM, Eric Snow wrote: > On Thu, Feb 28, 2013 at 10:32 AM, Guido van Rossum wrote: >> Hm. I write code regularly that takes a **kwargs dict and stores it >> into some other longer-lasting datastructure. Having that other >> datastructure suddenly receive OrderedDicts instead of plain dicts >> might definitely cause some confusion and possible performance issues. > > Could you elaborate on what confusion it might cause? Well, x.__class__ is different, repr(x) is different, ... > As to performance relative to dict, this has definitely been my > primary concern. I agree that the impact has to be insignificant for > the **kwargs proposal to go anywhere. Obviously OrderedDict in C had > better be an improvement over the pure Python version or there isn't > much point. However, it goes beyond that in the cases where we might > replace current uses of dict with OrderedDict. What happens to the performance if I insert many thousands (or millions) of items to an OrderedDict? What happens to the space it uses? The thing is, what started out as OrderedDict stays one, but its lifetime may be long and the assumptions around dict performance are rather extreme (or we wouldn't be redoing the implementation regularly). > My plan has been to do a bunch of performance comparison once the > implementation is complete and tune it as much as possible with an eye > toward the main potential internal use cases. From my point of view, > those are **kwargs and class namespace. This is in part why I've > brought those two up. I'm fine with doing this by default for a class namespace; the type of cls.__dict__ is already a non-dict (it's a proxy) and it's unlikely to have 100,000 entries. For **kwds I'm pretty concerned; the use cases seem flimsy. > For instance, one guidepost I've used is that typically **kwargs is > going to be small. However, even for large kwargs I don't want any > performance difference to be a problem. So, in my use case, the kwargs is small, but the object may live a long and productive life after the function call is only a faint memory, and it might grow dramatically. IOW I have very high standards for backwards compatibility here. >> And I don't recall ever having wanted to know the order of the kwargs >> in the call. > > Though it may sound a little odd, the first use case I bumped into was > with OrderedDict itself: > > OrderedDict(a=1, b=2, c=3) But because of the self-referentiality, this doesn't prove anything. :-) > There were a few other reasonable use cases mentioned in other threads > a while back. I'll dig them up if that would help. It would. >> But even apart from that backwards incompatibility I think this >> feature is too rarely useful to make it the default behavior -- if >> anything I want **kwargs to become faster! > > You mean something like possibly not unpacking the **kwargs in a call > into another dict? > > def f(**kwargs): > return kwargs > d = {'a': 1, 'b': 2} > assert d is f(**d) > > Certainly faster, though definitely a semantic difference. No, that would introduce nasty aliasing problems in some cases. I've actually written code that depends on the copy being made here. (Yesterday, actually. :-) -- --Guido van Rossum (python.org/~guido) From foogod at gmail.com Thu Feb 28 22:32:01 2013 From: foogod at gmail.com (Alex Stewart) Date: Thu, 28 Feb 2013 13:32:01 -0800 Subject: [Python-ideas] OrderedDict for kwargs and class statement namespace In-Reply-To: References: Message-ID: On Thu, Feb 28, 2013 at 11:30 AM, Christian Heimes wrote: > Raymond was/is working on a modification of the dict's internal data > structure that reduces its memory consumption. IIRC the modification > adds partial ordered as a side effect. The keys are ordered in insertion > order until keys are removed. > While this is potentially convenient, in order for people to really be able to use it for these purposes it would have to be reliable behavior going forward. The question is therefore: do we actually want to explicitly make this behavior part of the dict API such that all future implementations of dict (and all implementations in any Python, not just CPython) must be guaranteed to also behave this way? I suspect the answer should be "no". On Thu, Feb 28, 2013 at 7:30 AM, Don Spaulding wrote: > So what's an acceptable level of performance regression for the sake of > doing things the "right way" here? > For class declarations, I think a little performance hit might not be too big a deal, but those are really not the most important use case here, since there are already ways to work around this for classes if one wants to. However, for function calls, which I actually believe is the more important question, I think we would have to think very hard about anything that introduced any real performance overhead, because functions are typically called a _lot_. Adding even a small extra delay to function calls could impact the overall performance of some types of Python programs substantially.. I also agree with Guido that there may be some unexpected consequences of suddenly changing the type of the kwargs parameter passed to functions which were coded assuming it would always be a plain-ol' dict.. I think it might be doable, but we'd have to be very sure that OrderedDict is exactly compatible in every conceivable way.. Perhaps instead we could compromise by keeping the default case as it is, but providing some mechanism for the function declaration to specify that it wants ordering information passed to it? I'm not sure what a good syntax for this would be, though ("(*args, ***okwargs)"? "(*args, **kwargs, *kworder)"? Not sure I'm really big on either of those, actually, but you get the idea..) --Alex -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericsnowcurrently at gmail.com Thu Feb 28 22:37:45 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Thu, 28 Feb 2013 14:37:45 -0700 Subject: [Python-ideas] OrderedDict for kwargs and class statement namespace In-Reply-To: References: Message-ID: On Thu, Feb 28, 2013 at 11:02 AM, Nick Coghlan wrote: > And PEP 422 is designed to make it easier to share a common __prepare__ > method with different post processing. A major use case for __prepare__() is to have the class definition use an OrderedDict. It's even a point of discussion in PEP 3115. While I agree with the the conclusion of the PEP that using only OrderedDict is inferior to __prepare__(), I also think defaulting to OrderedDict is viable and useful. Using __prepare__() necessitates the use of a metaclass, which most people consider black magic. Even when you can inherit from a class that has a custom metaclass (like collections.abc.ABC), it still necessitates inheritance and the chance for metaclass conflicts. While I'm on board with PEP 422, I'm not clear on how it helps here. If class namespaces were ordered by default then class decorators, which are typically much easier to comprehend, would steal yet another use case from metaclasses. The recent discussions on enums got me thinking about this. In some cases you want you class customization logic to be inherited. That's where you've got to use a metaclass. In other cases you don't. Decorators work great for that. If your class decorator needs to have ordering information, then you also have to use a metaclass. Having OrderedDict be the default class namespace would address that. -eric From ned at nedbatchelder.com Thu Feb 28 22:51:42 2013 From: ned at nedbatchelder.com (Ned Batchelder) Date: Thu, 28 Feb 2013 16:51:42 -0500 Subject: [Python-ideas] OrderedDict for kwargs and class statement namespace In-Reply-To: <512F88AD.30306@stoneleaf.us> References: <20130228112742.09f5500b@pitrou.net> <20130228164810.74f2d367@pitrou.net> <512F88AD.30306@stoneleaf.us> Message-ID: <512FD16E.30503@nedbatchelder.com> On 2/28/2013 11:41 AM, Ethan Furman wrote: > On 02/28/2013 08:14 AM, Don Spaulding wrote: >> >> >> >> On Thu, Feb 28, 2013 at 9:48 AM, Antoine Pitrou > > wrote: >> >> Le Thu, 28 Feb 2013 09:30:50 -0600, >> Don Spaulding > > >> a ?crit : >> > >> > For an example of the "recommended" way to get the ordering of >> your >> > class attributes: >> >http://stackoverflow.com/questions/3288107/how-can-i-get-fields-in-an-original-order >> >> This is already possible with the __prepare__ magic method. >> http://docs.python.org/3.4/reference/datamodel.html#preparing-the-class-namespace >> >> >> Sure. Case in point: Django has been working around it since at >> least python 2.4. >> >> > It seems to me that the "right thing" for python to do when >> given an >> > ordered list of key=value pairs in a function call or class >> > definition, is to retain the order. So what's an acceptable >> level of >> > performance regression for the sake of doing things the "right >> way" >> > here? >> >> Or, rather, what is the benefit of doing things "the right way"? >> There >> are incredibly few cases for relying on the order of key=value >> pairs in >> function calls. >> >> >> "If you build it, they will come..." >> >> When I originally encountered the need for python to retain the order >> of kwargs that my caller specified, it surprised >> me that there wasn't more clamoring for kwargs being an OrderedDict. >> However, since my development timeline didn't >> allow for holding the project up while a patch was pushed through >> python-dev and out into a real python release, I >> sucked it up, forced my calling code to send in hand-crafted >> OrderedDicts and called it a day. I think most developers >> don't even stop to think that the language *could* be different, they >> just put in the workaround and move on. >> >> I think if python stopped dropping the order of kwargs on the floor >> today, you'd see people start to rely on the order >> of kwargs tomorrow. > > +1 > > I'd already be relying on it if it were there. > Could you advance the discussion by elaborating your use case? I've never had need for ordered kwargs, so I'm having a hard time seeing how they would be useful. --Ned. > -- > ~Ethan~ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > http://mail.python.org/mailman/listinfo/python-ideas > From ericsnowcurrently at gmail.com Thu Feb 28 22:53:49 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Thu, 28 Feb 2013 14:53:49 -0700 Subject: [Python-ideas] OrderedDict for kwargs and class statement namespace In-Reply-To: References: Message-ID: On Thu, Feb 28, 2013 at 2:28 PM, Guido van Rossum wrote: > What happens to the performance if I insert many thousands (or > millions) of items to an OrderedDict? What happens to the space it > uses? The thing is, what started out as OrderedDict stays one, but its > lifetime may be long and the assumptions around dict performance are > rather extreme (or we wouldn't be redoing the implementation > regularly). > So, in my use case, the kwargs is small, but the object may live a > long and productive life after the function call is only a faint > memory, and it might grow dramatically. Good point. My own use of **kwargs rarely sees the object leave the function or get very big, and this aspect of it just hadn't come up to broaden my point of view. I'm glad we're having this discussion. My intuition is that such a use case would be pretty rare, but even then... > > IOW I have very high standards for backwards compatibility here. ...Python has an exceptional standard in this regard. FWIW, I agree. I would want to be confident about the ramifications before we made any change like this. >> There were a few other reasonable use cases mentioned in other threads >> a while back. I'll dig them up if that would help. > > It would. Okay. > >>> But even apart from that backwards incompatibility I think this >>> feature is too rarely useful to make it the default behavior -- if >>> anything I want **kwargs to become faster! >> >> You mean something like possibly not unpacking the **kwargs in a call >> into another dict? >> >> def f(**kwargs): >> return kwargs >> d = {'a': 1, 'b': 2} >> assert d is f(**d) >> >> Certainly faster, though definitely a semantic difference. > > No, that would introduce nasty aliasing problems in some cases. I've > actually written code that depends on the copy being made here. > (Yesterday, actually. :-) You can't blame me for trying to make **kwargs-as-OrderedDict seem like a great idea in comparison. -eric From ericsnowcurrently at gmail.com Thu Feb 28 23:18:08 2013 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Thu, 28 Feb 2013 15:18:08 -0700 Subject: [Python-ideas] OrderedDict for kwargs and class statement namespace In-Reply-To: References: Message-ID: On Thu, Feb 28, 2013 at 2:32 PM, Alex Stewart wrote: > such that all future implementations of > dict (and all implementations in any Python, not just CPython) must be > guaranteed to also behave this way? This is also a good point applied to both **kwargs and class namespace. Before we did either we'd need to be clear on the impact on other Python implementors. > However, for function calls, which I actually believe is the more important > question, I think we would have to think very hard about anything that > introduced any real performance overhead, because functions are typically > called a _lot_. Adding even a small extra delay to function calls could > impact the overall performance of some types of Python programs > substantially.. **kwargs-as-OrderedDict impacts performance in two ways: the performance of packing the "unbound" keyword arguments into the OrderedDict and the performance of OrderedDict in normal operation after its handed off to the function. Otherwise you don't get a performance impact on functions. > > I also agree with Guido that there may be some unexpected consequences of > suddenly changing the type of the kwargs parameter passed to functions which > were coded assuming it would always be a plain-ol' dict.. I think it might > be doable, but we'd have to be very sure that OrderedDict is exactly > compatible in every conceivable way.. For Python backward compatibility is a cornerstone. I'm surprised there isn't something in the Zen about it. For **kwargs the bar for compatibility is especially high and I agree with that. > > Perhaps instead we could compromise by keeping the default case as it is, > but providing some mechanism for the function declaration to specify that it > wants ordering information passed to it? I'm not sure what a good syntax > for this would be, though ("(*args, ***okwargs)"? "(*args, **kwargs, > *kworder)"? Not sure I'm really big on either of those, actually, but you > get the idea..) **kwargs packing happens in the interpreter rather than in relationship to functionality provided by the function, so whatever the mechanism it would have be something the interpreter consumes, either a new API/syntax for building functions or a new syntax like you've mentioned. This has come up before. Classes have metaclasses (and __prepare__). Modules have loaders. Poor, poor functions. Because of the same concerns you've already expressed regarding the criticality of function performance, they miss out on all sorts of fun--inside their highly optimized box looking out at the other types showing off their cool new features all the time. It just isn't fair. :) -eric From zuo at chopin.edu.pl Thu Feb 28 23:51:21 2013 From: zuo at chopin.edu.pl (Jan Kaliszewski) Date: Thu, 28 Feb 2013 23:51:21 +0100 Subject: [Python-ideas] OrderedDict for kwargs and class statement namespace In-Reply-To: References: Message-ID: <32fd3c22d72c258ae8e49c7c3fa10f98@chopin.edu.pl> 28.02.2013 11:15, Eric Snow wrote: > * have the **kwargs param be an OrderedDict rather than a dict > * have class definitions happen relative to an OrderedDict by > default rather than a dict, and still overridable by a metaclass's > __prepare__(). While having class namespace ordered sounds very nice, ordered **kwargs sometimes may not be a desirable feature -- especially when you want to keep a cooperatively used interface as simple as possible (because of the risk that keyword argument order could then be taken into consideration by *some* actors while others would still operate with the assumption that argument order cannot be known...). > Both of these will need OrderedDict in C, which is getting close > (issue #16991). Ad issue #16991: will the present semantics of inheriting from OrderedDict be kept untouched? (I mean: which methods call other methods => which methods you need to override etc.) If not we'll have a backward compatibility issue (maybe not very serious but still...). Cheers. *j