Re: [Python-ideas] anonymous object support
[adding python-ideas back to CC] On Mon, Jul 25, 2011 at 5:40 PM, Paul Moore <p.f.moore@gmail.com> wrote:
You could do that if it seems reasonable or useful in that particular case, but more often than not I think you'd do something like: variable = Dummy(foo=lambda x: x) expression_involving(variable) Having one line of code that accomplishes a simple task (creating an object and saving it in a variable) instead of having to write N lines for each attribute makes for more readability not less IMO. You can still break lines inside the parens and add each attribute on its own line, but the indentation would at least clue the reader to the fact that it's really a single operation. Having N lines, especially if N is more than 2 or 3, is begging for the whole thing to be moved off into a helper function. And in tests this kind of operation happens all the time.
https://github.com/has207/flexmock/blob/master/tests/flexmock_test.py You can take a look at the tests starting at line 56 -- the flexmock() function does exactly what I'm proposing when given keyword args.
It wasn't my message that took the list off the CC :) Cheers, -Herman
On 25 July 2011 10:46, Herman Sheremetyev <herman@swebpage.com> wrote:
Thanks. That code does look nice and clean, although it is once again using artificially simple attributes (although that's not surprising in test code). But I'm still puzzled as to what you're asking for. Clearly (I assume) you're not asking for flexmock to be added to the stdlib. So are you asking for something like flexmock - in which case, what are the use cases for it (that aren't use cases for flexmock itself)? Sorry if I'm being dense here. If others understand the request feel free to drop this conversation as I'm not sure I'm adding much that I haven't already said.
From my point of view, I see 3 separate aspects:
1. Specialised functions like flexmock, which fill a particular niche, and which quite reasonably hide the complexity of creating "anonymous" objects behind a simple API. No problem here, these are clearly useful, but they don't need to go into the stdlib unless their particular focus warrants it. 2. A generic "create object with dynamically assigned attributes" function. I haven't seen any use cases for this which aren't rare enough that a multi-line answer, or Nick's type() workaround, aren't sufficient. 3. Some functionality to make writing flexmock-like functions a bit easier. Looking at the implementation of flexmock (specifically the Mock class) I don't see any obvious place it would make sense to use such a thing. You couldn't use type() to simplify Mock.__init__ as far as I can see, and clearly type() is far too lightweight to replace Mock completely. So again, I see no clear use case here. But if this boils down to "I wouldn't use the proposed function myself", then so be it. I'm happy to leave the people who *would* use it to thrash out the details and the benefits. Paul.
On Mon, Jul 25, 2011 at 9:18 PM, Paul Moore <p.f.moore@gmail.com> wrote:
No, I'm not asking for flexmock to be added to stdlib :) I'm writing a different library based on my experiences with porting flexmock that has a much cleaner and more Pythonic API that I *might* propose to be added at some point but it's not really at the stage where it's ready for discussion. To answer your question, the use cases are exactly the same as *some* of the uses for flexmock. This is precisely why I've proposed it as I really don't think we need things like flexmock to do this. Flexmock's primary function is creating partial mocks, checking expectations, and ensuring it cleans up after itself. The fact that it allows for creation of objects on the fly is not really what it's for, though it provides that functionality, like most of the other mocking libraries, simply because it's missing and is such a common pattern in tests that a testing library would not really be complete unless it provided it.
Test code == code :) It's more code than "actual" code in many cases, so I don't know what defines these use cases as rare. Tests are a pretty obvious place where this is useful but I don't think it's the only place and being able to create objects in this manner would improve readability in a number of places where small, throw-away objects are used. A multi-line solution first requires a user-defined class. And I feel like we really shouldn't have to define a class just to create an object that we throw away two lines later. Using type()'s current totally unwieldy API to do such an extremely simple and common thing is again not really a good answer IMO. I proposed some changes that shouldn't break type()'s current API while making it usable for this particular pattern in my other message. It's not a big change and I'd be happy to implement it and send in a patch if nobody has any strong objections. Cheers, -Herman
On Tue, Jul 26, 2011 at 12:13 AM, Herman Sheremetyev <herman@swebpage.com> wrote:
Defining classes isn't a big deal. It's just fancy syntax for calling the metaclass (as the equivalent code based on a direct call to type shows). If it really bothers you, hide it in a helper function: def anon_object(**class_attrs): class Anon(object): pass Anon.__dict__.update(class_attrs) return Anon() def anon_object(**class_attrs): return type('Anon', (), class_attrs)() Or you could have a bit of mercy on the people that are going to have to maintain this code and give the class a meaningful name that will be displayed in tracebacks. Maybe go crazy and do something radical like give it a docstring. The person lost in the maze of twisty anonymous objects, strangely alike, may be you in 6 months time.
What you propose is not common. You have presented no use cases other than mock objects, and you have stated that your own proposal is inadequate for that use case. Anonymous objects trade a little bit of typing and thought for the code author for the loss of meaningful information that could benefit future maintainers. There is absolutely zero reason to add additional complexity to the language core or the standard library for such a niche (and questionable) use case when a simple wrapper function around type can do the job. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, Jul 26, 2011 at 9:18 AM, Nick Coghlan <ncoghlan@gmail.com> wrote: <snip>
As you yourself pointed out, the complexity is already there in the core language hiding in a single line of API description to the type() function. It's already there to be used, but provides very little documentation and an API that is sure to baffle anyone that comes across code that uses it. To reiterate, I am proposing improving the type() API to provide some reasonable defaults while retaining backwards compatibility. In other words, making an *existing* difficult-to-use API into an intuitive one. FWIW, I think the type() function is not really a great choice for making classes on the fly. But if that's what we have to use then let's at least make it a little more user-friendly. -Herman
On Mon, Jul 25, 2011 at 7:30 PM, Herman Sheremetyev <herman@swebpage.com> wrote:
Can you summarize the proposal for those who didn't follow the thread blow-by-blow? Your first message proposed keyword arguments to object(), which can't fly because object() creates objects without a __dict__. Now you suddenly seem to have switched to adding more complexity to type(). But what exactly?
So you are proposing to make an undesirable API more user-friendly. Isn't that creating an attractive nuisance? -- --Guido van Rossum (python.org/~guido)
On Tue, Jul 26, 2011 at 11:42 AM, Guido van Rossum <guido@python.org> wrote:
So the initial solution with type() posted above was: obj = type('Foo', (), dict(foo=(lambda x: x)))() Adding the necessary "self" to the lambda, using an empty name, and replacing dict() with {} makes it slightly easier to parse: obj = type('', (), {'foo': lambda self, x: x})() [here are the API changes I propose] Going an extra step and making it possible for the dictionary to be passed in as keyword arguments would make it even nicer: obj = type('', (), foo=lambda self, x: x)() And going one final step to give those first two positional arguments default values (empty string and empty tuple? or "Anonymous" and empty tuple?) would make it even better: obj = type(foo=lambda self, x: x)()
Well, I think it's not very intuitive that type() can be used as a class constructor. But it is what it is at this stage, so if I come across it in code or write it myself I'd rather it provided some reasonable defaults. -Herman
On Mon, Jul 25, 2011 at 7:58 PM, Herman Sheremetyev <herman@swebpage.com> wrote:
Oh, that is bad. A single class object costs many thousands of bytes in overhead and contains lots of cycles so GC needs to come along to clean it up. And all that to create a single tiny object?
Same problem.
I recommend that you create a library module that implements this more efficiently and then see how often you *really* use it.
There's a way of looking at that which makes it very logical, and intuition is learned. (This is counter-intuitive, but true nevertheless. :-)
My expectation is that you'll rarely come across a use case where the keyword arguments are actually useful. Plus, creating new dynamic class objects willy-nilly is hugely inefficient. -- --Guido van Rossum (python.org/~guido)
On Tue, Jul 26, 2011 at 12:58 PM, Herman Sheremetyev <herman@swebpage.com> wrote:
Well, I think it's not very intuitive that type() can be used as a class constructor.
Then you don't really understand metaclasses yet (now, metaclasses themselves certainly don't qualify as intuitive, but understanding them is essential to understanding many otherwise confusing elements of Python's type system). type() works the way it does because it is designed to construct a new class object from a name, a tuple of base classes and a namespace dictionary, just like any other metaclass. The slightly hacky (but very convenient) part is actually the ability to use the single argument form of type() to query the metaclass of an existing object. To get back to the ideas discussed in this thread: 1. Use cases where creating a lot of one-shot classes on the fly is a reasonable idea are rare and highly specialised 2. If you want to do so, the obvious way is to use a class statement and adjust __name__ and __dict__ afterwards 3. Alternatively, if you want this functionality as an expression, either the type builtin or a custom function can do the job 4. It is significantly more efficient to abstract out a common class definition and create multiple instances of that class 5. If you just want a namespace to store some data with consistent attribute names, then collections.namedtuple can be used to also make the instance storage efficient. If you store a function in a namedtuple instance then you can still call it, it just won't be passed 'self' as an implicit first parameter (as the descriptor machinery will not be invoked). 6. If you just want an arbitrary attribute holder, than instances of a trivial empty user-defined class can serve that purpose. As with named tuples, functions stored on the instance can still be called, but retrieving them won't invoke the descriptor machinery. 7. Mock objects do *not* qualify as a common use case. Despite the proliferation of libraries that choose to provide slightly different APIs for them, such objects may be used widely in test suites but are still *implemented* rarely - they fall into the "highly specialised" category I mentioned above. The tools are there so that people that know what they're doing can do what needs to be done to achieve their ends. Simplifying the API for dynamic type creation would be an attractive nuisance, as it will likely lead to people writing ill-advised code without giving the matter sufficient thought. Obviously, that's going to happen no matter what we do, but there's no reason for us to deliberately *encourage* bad ideas by making them too easy to write. Points 5 and 6 go back to the original question about namespace objects, which is independent of the dynamic type creation issue. Various attempts have been made to address that over the years, and the current answers are the original approach (just define a namespace type and use instances of it to store arbitrary data, with poor debugging support due to the relative anonymity and arbitrary content of each instance) and the named tuple approach, which provides excellent debugging information and memory efficiency at the cost of being explicit up front regarding the attributes possessed by each instance. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, Jul 26, 2011 at 2:36 PM, Matt Joiner <anacrolix@gmail.com> wrote:
Just as a slight aside, is the special case of type() purely for convenience? Aren't the "new-style" ways to do this __class__ and isinstance()?
Not really - all three do different things: 1. type(x) always gives you the true metaclass of x 2. x.__class__ tells you what x claims to be (which may differ from type(x) for things like proxy objects) 3. isinstance(x, cls) doesn't tell you what x actually *is*, just whether or not it passes the isinstance check (it may be an instance of a subclass, proxying for another object, or just registered if cls refers to an ABC) There has to be *some* way to spell number 1 and Guido chose to use type() for it. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 25 July 2011 10:46, Herman Sheremetyev <herman@swebpage.com> wrote:
Thanks. That code does look nice and clean, although it is once again using artificially simple attributes (although that's not surprising in test code). But I'm still puzzled as to what you're asking for. Clearly (I assume) you're not asking for flexmock to be added to the stdlib. So are you asking for something like flexmock - in which case, what are the use cases for it (that aren't use cases for flexmock itself)? Sorry if I'm being dense here. If others understand the request feel free to drop this conversation as I'm not sure I'm adding much that I haven't already said.
From my point of view, I see 3 separate aspects:
1. Specialised functions like flexmock, which fill a particular niche, and which quite reasonably hide the complexity of creating "anonymous" objects behind a simple API. No problem here, these are clearly useful, but they don't need to go into the stdlib unless their particular focus warrants it. 2. A generic "create object with dynamically assigned attributes" function. I haven't seen any use cases for this which aren't rare enough that a multi-line answer, or Nick's type() workaround, aren't sufficient. 3. Some functionality to make writing flexmock-like functions a bit easier. Looking at the implementation of flexmock (specifically the Mock class) I don't see any obvious place it would make sense to use such a thing. You couldn't use type() to simplify Mock.__init__ as far as I can see, and clearly type() is far too lightweight to replace Mock completely. So again, I see no clear use case here. But if this boils down to "I wouldn't use the proposed function myself", then so be it. I'm happy to leave the people who *would* use it to thrash out the details and the benefits. Paul.
On Mon, Jul 25, 2011 at 9:18 PM, Paul Moore <p.f.moore@gmail.com> wrote:
No, I'm not asking for flexmock to be added to stdlib :) I'm writing a different library based on my experiences with porting flexmock that has a much cleaner and more Pythonic API that I *might* propose to be added at some point but it's not really at the stage where it's ready for discussion. To answer your question, the use cases are exactly the same as *some* of the uses for flexmock. This is precisely why I've proposed it as I really don't think we need things like flexmock to do this. Flexmock's primary function is creating partial mocks, checking expectations, and ensuring it cleans up after itself. The fact that it allows for creation of objects on the fly is not really what it's for, though it provides that functionality, like most of the other mocking libraries, simply because it's missing and is such a common pattern in tests that a testing library would not really be complete unless it provided it.
Test code == code :) It's more code than "actual" code in many cases, so I don't know what defines these use cases as rare. Tests are a pretty obvious place where this is useful but I don't think it's the only place and being able to create objects in this manner would improve readability in a number of places where small, throw-away objects are used. A multi-line solution first requires a user-defined class. And I feel like we really shouldn't have to define a class just to create an object that we throw away two lines later. Using type()'s current totally unwieldy API to do such an extremely simple and common thing is again not really a good answer IMO. I proposed some changes that shouldn't break type()'s current API while making it usable for this particular pattern in my other message. It's not a big change and I'd be happy to implement it and send in a patch if nobody has any strong objections. Cheers, -Herman
On Tue, Jul 26, 2011 at 12:13 AM, Herman Sheremetyev <herman@swebpage.com> wrote:
Defining classes isn't a big deal. It's just fancy syntax for calling the metaclass (as the equivalent code based on a direct call to type shows). If it really bothers you, hide it in a helper function: def anon_object(**class_attrs): class Anon(object): pass Anon.__dict__.update(class_attrs) return Anon() def anon_object(**class_attrs): return type('Anon', (), class_attrs)() Or you could have a bit of mercy on the people that are going to have to maintain this code and give the class a meaningful name that will be displayed in tracebacks. Maybe go crazy and do something radical like give it a docstring. The person lost in the maze of twisty anonymous objects, strangely alike, may be you in 6 months time.
What you propose is not common. You have presented no use cases other than mock objects, and you have stated that your own proposal is inadequate for that use case. Anonymous objects trade a little bit of typing and thought for the code author for the loss of meaningful information that could benefit future maintainers. There is absolutely zero reason to add additional complexity to the language core or the standard library for such a niche (and questionable) use case when a simple wrapper function around type can do the job. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, Jul 26, 2011 at 9:18 AM, Nick Coghlan <ncoghlan@gmail.com> wrote: <snip>
As you yourself pointed out, the complexity is already there in the core language hiding in a single line of API description to the type() function. It's already there to be used, but provides very little documentation and an API that is sure to baffle anyone that comes across code that uses it. To reiterate, I am proposing improving the type() API to provide some reasonable defaults while retaining backwards compatibility. In other words, making an *existing* difficult-to-use API into an intuitive one. FWIW, I think the type() function is not really a great choice for making classes on the fly. But if that's what we have to use then let's at least make it a little more user-friendly. -Herman
On Mon, Jul 25, 2011 at 7:30 PM, Herman Sheremetyev <herman@swebpage.com> wrote:
Can you summarize the proposal for those who didn't follow the thread blow-by-blow? Your first message proposed keyword arguments to object(), which can't fly because object() creates objects without a __dict__. Now you suddenly seem to have switched to adding more complexity to type(). But what exactly?
So you are proposing to make an undesirable API more user-friendly. Isn't that creating an attractive nuisance? -- --Guido van Rossum (python.org/~guido)
On Tue, Jul 26, 2011 at 11:42 AM, Guido van Rossum <guido@python.org> wrote:
So the initial solution with type() posted above was: obj = type('Foo', (), dict(foo=(lambda x: x)))() Adding the necessary "self" to the lambda, using an empty name, and replacing dict() with {} makes it slightly easier to parse: obj = type('', (), {'foo': lambda self, x: x})() [here are the API changes I propose] Going an extra step and making it possible for the dictionary to be passed in as keyword arguments would make it even nicer: obj = type('', (), foo=lambda self, x: x)() And going one final step to give those first two positional arguments default values (empty string and empty tuple? or "Anonymous" and empty tuple?) would make it even better: obj = type(foo=lambda self, x: x)()
Well, I think it's not very intuitive that type() can be used as a class constructor. But it is what it is at this stage, so if I come across it in code or write it myself I'd rather it provided some reasonable defaults. -Herman
On Mon, Jul 25, 2011 at 7:58 PM, Herman Sheremetyev <herman@swebpage.com> wrote:
Oh, that is bad. A single class object costs many thousands of bytes in overhead and contains lots of cycles so GC needs to come along to clean it up. And all that to create a single tiny object?
Same problem.
I recommend that you create a library module that implements this more efficiently and then see how often you *really* use it.
There's a way of looking at that which makes it very logical, and intuition is learned. (This is counter-intuitive, but true nevertheless. :-)
My expectation is that you'll rarely come across a use case where the keyword arguments are actually useful. Plus, creating new dynamic class objects willy-nilly is hugely inefficient. -- --Guido van Rossum (python.org/~guido)
On Tue, Jul 26, 2011 at 12:58 PM, Herman Sheremetyev <herman@swebpage.com> wrote:
Well, I think it's not very intuitive that type() can be used as a class constructor.
Then you don't really understand metaclasses yet (now, metaclasses themselves certainly don't qualify as intuitive, but understanding them is essential to understanding many otherwise confusing elements of Python's type system). type() works the way it does because it is designed to construct a new class object from a name, a tuple of base classes and a namespace dictionary, just like any other metaclass. The slightly hacky (but very convenient) part is actually the ability to use the single argument form of type() to query the metaclass of an existing object. To get back to the ideas discussed in this thread: 1. Use cases where creating a lot of one-shot classes on the fly is a reasonable idea are rare and highly specialised 2. If you want to do so, the obvious way is to use a class statement and adjust __name__ and __dict__ afterwards 3. Alternatively, if you want this functionality as an expression, either the type builtin or a custom function can do the job 4. It is significantly more efficient to abstract out a common class definition and create multiple instances of that class 5. If you just want a namespace to store some data with consistent attribute names, then collections.namedtuple can be used to also make the instance storage efficient. If you store a function in a namedtuple instance then you can still call it, it just won't be passed 'self' as an implicit first parameter (as the descriptor machinery will not be invoked). 6. If you just want an arbitrary attribute holder, than instances of a trivial empty user-defined class can serve that purpose. As with named tuples, functions stored on the instance can still be called, but retrieving them won't invoke the descriptor machinery. 7. Mock objects do *not* qualify as a common use case. Despite the proliferation of libraries that choose to provide slightly different APIs for them, such objects may be used widely in test suites but are still *implemented* rarely - they fall into the "highly specialised" category I mentioned above. The tools are there so that people that know what they're doing can do what needs to be done to achieve their ends. Simplifying the API for dynamic type creation would be an attractive nuisance, as it will likely lead to people writing ill-advised code without giving the matter sufficient thought. Obviously, that's going to happen no matter what we do, but there's no reason for us to deliberately *encourage* bad ideas by making them too easy to write. Points 5 and 6 go back to the original question about namespace objects, which is independent of the dynamic type creation issue. Various attempts have been made to address that over the years, and the current answers are the original approach (just define a namespace type and use instances of it to store arbitrary data, with poor debugging support due to the relative anonymity and arbitrary content of each instance) and the named tuple approach, which provides excellent debugging information and memory efficiency at the cost of being explicit up front regarding the attributes possessed by each instance. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, Jul 26, 2011 at 2:36 PM, Matt Joiner <anacrolix@gmail.com> wrote:
Just as a slight aside, is the special case of type() purely for convenience? Aren't the "new-style" ways to do this __class__ and isinstance()?
Not really - all three do different things: 1. type(x) always gives you the true metaclass of x 2. x.__class__ tells you what x claims to be (which may differ from type(x) for things like proxy objects) 3. isinstance(x, cls) doesn't tell you what x actually *is*, just whether or not it passes the isinstance check (it may be an instance of a subclass, proxying for another object, or just registered if cls refers to an ABC) There has to be *some* way to spell number 1 and Guido chose to use type() for it. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (6)
-
Guido van Rossum -
Herman Sheremetyev -
Masklinn -
Matt Joiner -
Nick Coghlan -
Paul Moore