Extend module objects to support properties

Properties are a wonderful facility. But they only work on conventional objects. Specifically, they *don't* work on module objects. It would be nice to extend module objects so properties worked there too. For example, Victor Stinner's currently proposed PEP 433 adds two new methods to the sys module: sys.getdefaultcloexc() and sys.setdefaultcloexc(). What are we, Java? Surely this would be much nicer as a property, sys.defaultcloexc. //arry/

On 30.01.2013 02:06, Larry Hastings wrote:
Would be nice, but I'm not sure how you'd implement this, since module contents are accessed directly via the module dictionary, so the attribute lookup hook to add the property magic is missing. Overall, it would be great to have modules behave more like classes. This idea has been floating around for years, but hasn't gone far due to the above direct content dict access approach taken by the Python code. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 30 2013)
::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

On Wed, Jan 30, 2013 at 11:06 AM, Larry Hastings <larry@hastings.org> wrote:
As MAL notes, the issues with such an approach are: - code executed at module scope - code in inner scopes that uses "global" - code that uses globals() - code that directly modifies a module's __dict__ There is too much code that expects to be able to modify a module's namespace directly without going through the attribute access machinery. However, a slightly more practical suggestion might be: 1. Officially bless the practice of placing class instances in sys.modules (currently this is tolerated, since it's the only way to manage things like lazy module loading, but not officially recommended as the way to achieve "module properties") 2. Change sys from a module object to an ordinary class instance Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Nick Coghlan, 30.01.2013 10:54:
The Cython project has been wanting this feature for years. We even considered writing our own Module (sub-)type for this, but didn't get ourselves convinced that all of the involved hassle was really worth it. The main drive behind it is full control over setters to allow for safe and fast internal C level access to module globals (which usually don't change from the outside but may...). Currently, users help themselves by explicitly declaring globals as static internal names that are invisible to external Python code. Allowing regular objects in sys.modules would be one way to do it, but these things are a lot more involved at the C level than at the Python level due to the C level module setup procedure. I wouldn't mind letting such a feature appear at the C level first, even though the Python syntax would be pretty obvious anyway. It's not like people would commonly mess around with sys.__dict__. (Although, many C modules have a Python module wrapper these days, not sure if and how this should get passed through.) Stefan

On 01/30/2013 01:54 AM, Nick Coghlan wrote:
Of those four issues, the latter two are wontfix. Code that futzes with an object's __dict__ bypasses the property machinery but this is already viewed as acceptable. Obviously the point of the proposal is to change the behavior of the first two. Whether this is manageable additional complexity, or fast enough, remains to be seen--which is why this is in ideas not dev. Also, I'm not sure there are any existing globals that we'd want to convert into properties. Assuming this is only used for new globals, this change hopefully wouldn't break existing code. (Fingers crossed.) //arry/

On 01/30/2013 04:45 PM, Steven D'Aprano wrote:
Well, hmm. The thing is, properties--at least the existing implementation with classes--doesn't mesh well with direct access via the dict. So, right now,
math.__dict__['pi'] 3.141592653589793
If we change math.pi to be a property it wouldn't be in the dict anymore. So that has the possibility of breaking code. We could ameliorate it with
math.__dict__['pi'] = math.pi
But if the user assigns a different value to math.__dict__['pi'], math.pi will diverge, which again could break code. (Who might try to assign a different value to pi? The 1897 House Of Representatives of Indiana for one!) More generally, it's often useful to monkeypatch "constants" at runtime, for testing purposes (and for less justifiable purposes). Why prevent that? I cite the Consenting Adults rule. //arry/

On 01/30/2013 05:53 PM, Larry Hastings wrote:
So make the property access the __dict__: --> class Test(object): ... @property ... def pi(self): ... return self.__dict__['pi'] ... @pi.setter ... def pi(self, new_value): ... self.__dict__['pi'] = new_value ... --> t = Test() --> t <__main__.Test object at 0x7f165d689850> --> t.pi = 3.141596 --> t.pi 3.141596 --> t.__dict__['pi'] = 3 --> t.pi 3 ~Ethan~

On 01/30/2013 08:22 PM, Ethan Furman wrote:
In which case, it behaves exactly like it does today without a property. Okay... so why bother? If your answer is "so it can have code behind it", maybe you find a better example than math.pi, which will never need code behind it. In general, I was proposing we add property support to modules mostly so that new globals could be properties, saving us from adding more accessors to the language. Otherwise I'm gonna have to switch to Eclipse. //arry/

On 01/30/2013 09:04 PM, Larry Hastings wrote:
math.pi wasn't my example, I was just showing how you could use the __dict__ as well. Why bother? Backwards compatibility. I think I missed your main point of __dict__ access, though -- if it is set directly then the property doesn't get the chance to update whatever is supposed to update at the right moment, leading to weird (and most likely buggy) behavior. ~Ethan~

On 1/31/2013 5:00 PM, Chris Angelico wrote:
Mixed float-decimal arithmetic does not work. If the float functions in math used 'math.pi', that would not work, but they do not. On the other hand, the stdlib and test suite do have a few expressions like 2.0 * math.pi and math.pi / 180.0 that would have to be changed to 2 * math.pi and math.pi / 180. (This might be a reasonable change anyway.) Non-float inputs to math functions like sin are apparently converted to float when possible (I tested int, Decimal, and Fraction with sin). So if you can otherwise avoid float +-*/ decimal, the answer might be yes. -- Terry Jan Reedy

On Thu, Jan 31, 2013 at 2:42 AM, Larry Hastings <larry@hastings.org> wrote:
Looking at the problem from a different direction: Currently, modules are *instances* of a normal type (types.ModuleType). Thus, anything stored in their global namespace is like anything else stored in a normal instance dictionary: no descriptor behaviour. The request in this thread is basically for a way to: 1. Define a custom type 2. Put an instance of that type in sys.modules instead of the ordinary module object Now here's the thing: we already support this, because the import system is designed to cope with modules replacing "sys.modules[__name__]" while they're being loaded. The way this happens is that, after we finish loading a module, we usually don't trust what the loader gave us. Instead, we go look at what's in sys.modules under the name being loaded. So if, in your module code, you do this: import sys, types class MyPropertyUsingModule(types.ModuleType): def __init__(self, original): # Keep a reference to the original module to avoid the # destructive cleanup of the global namespace self._original = original @property def myglobal(self): return theglobal @myglobal.setter def myglobal(self, value): global theglobal theglobal = value sys.modules[__name__] = MyPropertyUsingModule(sys.modules[__name__]) Then what you end up with in sys.modules is a module with a global property, "myglobal". I'd prefer to upgrade this from "begrudged backwards compatibility hack" to "supported feature", rather than doing anything more complicated. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Thu, Jan 31, 2013 at 1:56 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
+1 At this point I don't see this behavior of the import system changing, even for Python 4. Making it part of the spec is the best fit for this class of problem (not-terribly-sophisticated solution for a relatively uncommon case). Otherwise we'd need a way to allow a module definition (.py, etc.) to dictate which class to use, which seems unnecessary and even overly complicated given the scale of the target audience. That said, Larry's original proposal relates to sys, a built-in module written in C (in CPython of course). In that case the solution is not quite the same, since module initialization interacts with sys.modules differently. [1][2] Accommodating the original request would require more work, whether to muck with the import C-API or making sys an instance of another type, as someone suggested. -eric [1] See http://mail.python.org/pipermail/python-dev/2012-November/122599.html [2] http://bugs.python.org/msg174704

On Fri, Feb 1, 2013 at 5:56 AM, Eric Snow <ericsnowcurrently@gmail.com> wrote:
sys is already special cased so heavily in the interpreter initialization, making it a little more special really wouldn't bother me much :) As far as customising other extension modules goes, the same "replace yourself in sys.modules" trick should still work, you'd just do it via the C API from your module init function. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Fri, Feb 1, 2013 at 12:58 PM, Eric Snow <ericsnowcurrently@gmail.com> wrote:
Oh, you're right, they're currently responsible for creating their own module object (I was thinking of the way I *wished* extension module imports worked). Returning the module object from the init function still works though - we just need to make sure the extension loader can cope with the init function returning a non-module object :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Thu, Jan 31, 2013 at 5:50 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
One benefit of this is that sys.modules could be made a [carefully done] property that wraps interp->modules, making sys.modules truly replaceable, which would simplify a number of different places where we've had to work around it. -eric

On 30.01.2013 02:06, Larry Hastings wrote:
Would be nice, but I'm not sure how you'd implement this, since module contents are accessed directly via the module dictionary, so the attribute lookup hook to add the property magic is missing. Overall, it would be great to have modules behave more like classes. This idea has been floating around for years, but hasn't gone far due to the above direct content dict access approach taken by the Python code. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jan 30 2013)
::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

On Wed, Jan 30, 2013 at 11:06 AM, Larry Hastings <larry@hastings.org> wrote:
As MAL notes, the issues with such an approach are: - code executed at module scope - code in inner scopes that uses "global" - code that uses globals() - code that directly modifies a module's __dict__ There is too much code that expects to be able to modify a module's namespace directly without going through the attribute access machinery. However, a slightly more practical suggestion might be: 1. Officially bless the practice of placing class instances in sys.modules (currently this is tolerated, since it's the only way to manage things like lazy module loading, but not officially recommended as the way to achieve "module properties") 2. Change sys from a module object to an ordinary class instance Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Nick Coghlan, 30.01.2013 10:54:
The Cython project has been wanting this feature for years. We even considered writing our own Module (sub-)type for this, but didn't get ourselves convinced that all of the involved hassle was really worth it. The main drive behind it is full control over setters to allow for safe and fast internal C level access to module globals (which usually don't change from the outside but may...). Currently, users help themselves by explicitly declaring globals as static internal names that are invisible to external Python code. Allowing regular objects in sys.modules would be one way to do it, but these things are a lot more involved at the C level than at the Python level due to the C level module setup procedure. I wouldn't mind letting such a feature appear at the C level first, even though the Python syntax would be pretty obvious anyway. It's not like people would commonly mess around with sys.__dict__. (Although, many C modules have a Python module wrapper these days, not sure if and how this should get passed through.) Stefan

On 01/30/2013 01:54 AM, Nick Coghlan wrote:
Of those four issues, the latter two are wontfix. Code that futzes with an object's __dict__ bypasses the property machinery but this is already viewed as acceptable. Obviously the point of the proposal is to change the behavior of the first two. Whether this is manageable additional complexity, or fast enough, remains to be seen--which is why this is in ideas not dev. Also, I'm not sure there are any existing globals that we'd want to convert into properties. Assuming this is only used for new globals, this change hopefully wouldn't break existing code. (Fingers crossed.) //arry/

On 01/30/2013 04:45 PM, Steven D'Aprano wrote:
Well, hmm. The thing is, properties--at least the existing implementation with classes--doesn't mesh well with direct access via the dict. So, right now,
math.__dict__['pi'] 3.141592653589793
If we change math.pi to be a property it wouldn't be in the dict anymore. So that has the possibility of breaking code. We could ameliorate it with
math.__dict__['pi'] = math.pi
But if the user assigns a different value to math.__dict__['pi'], math.pi will diverge, which again could break code. (Who might try to assign a different value to pi? The 1897 House Of Representatives of Indiana for one!) More generally, it's often useful to monkeypatch "constants" at runtime, for testing purposes (and for less justifiable purposes). Why prevent that? I cite the Consenting Adults rule. //arry/

On 01/30/2013 05:53 PM, Larry Hastings wrote:
So make the property access the __dict__: --> class Test(object): ... @property ... def pi(self): ... return self.__dict__['pi'] ... @pi.setter ... def pi(self, new_value): ... self.__dict__['pi'] = new_value ... --> t = Test() --> t <__main__.Test object at 0x7f165d689850> --> t.pi = 3.141596 --> t.pi 3.141596 --> t.__dict__['pi'] = 3 --> t.pi 3 ~Ethan~

On 01/30/2013 08:22 PM, Ethan Furman wrote:
In which case, it behaves exactly like it does today without a property. Okay... so why bother? If your answer is "so it can have code behind it", maybe you find a better example than math.pi, which will never need code behind it. In general, I was proposing we add property support to modules mostly so that new globals could be properties, saving us from adding more accessors to the language. Otherwise I'm gonna have to switch to Eclipse. //arry/

On 01/30/2013 09:04 PM, Larry Hastings wrote:
math.pi wasn't my example, I was just showing how you could use the __dict__ as well. Why bother? Backwards compatibility. I think I missed your main point of __dict__ access, though -- if it is set directly then the property doesn't get the chance to update whatever is supposed to update at the right moment, leading to weird (and most likely buggy) behavior. ~Ethan~

On 1/31/2013 5:00 PM, Chris Angelico wrote:
Mixed float-decimal arithmetic does not work. If the float functions in math used 'math.pi', that would not work, but they do not. On the other hand, the stdlib and test suite do have a few expressions like 2.0 * math.pi and math.pi / 180.0 that would have to be changed to 2 * math.pi and math.pi / 180. (This might be a reasonable change anyway.) Non-float inputs to math functions like sin are apparently converted to float when possible (I tested int, Decimal, and Fraction with sin). So if you can otherwise avoid float +-*/ decimal, the answer might be yes. -- Terry Jan Reedy

On Thu, Jan 31, 2013 at 2:42 AM, Larry Hastings <larry@hastings.org> wrote:
Looking at the problem from a different direction: Currently, modules are *instances* of a normal type (types.ModuleType). Thus, anything stored in their global namespace is like anything else stored in a normal instance dictionary: no descriptor behaviour. The request in this thread is basically for a way to: 1. Define a custom type 2. Put an instance of that type in sys.modules instead of the ordinary module object Now here's the thing: we already support this, because the import system is designed to cope with modules replacing "sys.modules[__name__]" while they're being loaded. The way this happens is that, after we finish loading a module, we usually don't trust what the loader gave us. Instead, we go look at what's in sys.modules under the name being loaded. So if, in your module code, you do this: import sys, types class MyPropertyUsingModule(types.ModuleType): def __init__(self, original): # Keep a reference to the original module to avoid the # destructive cleanup of the global namespace self._original = original @property def myglobal(self): return theglobal @myglobal.setter def myglobal(self, value): global theglobal theglobal = value sys.modules[__name__] = MyPropertyUsingModule(sys.modules[__name__]) Then what you end up with in sys.modules is a module with a global property, "myglobal". I'd prefer to upgrade this from "begrudged backwards compatibility hack" to "supported feature", rather than doing anything more complicated. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Thu, Jan 31, 2013 at 1:56 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
+1 At this point I don't see this behavior of the import system changing, even for Python 4. Making it part of the spec is the best fit for this class of problem (not-terribly-sophisticated solution for a relatively uncommon case). Otherwise we'd need a way to allow a module definition (.py, etc.) to dictate which class to use, which seems unnecessary and even overly complicated given the scale of the target audience. That said, Larry's original proposal relates to sys, a built-in module written in C (in CPython of course). In that case the solution is not quite the same, since module initialization interacts with sys.modules differently. [1][2] Accommodating the original request would require more work, whether to muck with the import C-API or making sys an instance of another type, as someone suggested. -eric [1] See http://mail.python.org/pipermail/python-dev/2012-November/122599.html [2] http://bugs.python.org/msg174704

On Fri, Feb 1, 2013 at 5:56 AM, Eric Snow <ericsnowcurrently@gmail.com> wrote:
sys is already special cased so heavily in the interpreter initialization, making it a little more special really wouldn't bother me much :) As far as customising other extension modules goes, the same "replace yourself in sys.modules" trick should still work, you'd just do it via the C API from your module init function. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Fri, Feb 1, 2013 at 12:58 PM, Eric Snow <ericsnowcurrently@gmail.com> wrote:
Oh, you're right, they're currently responsible for creating their own module object (I was thinking of the way I *wished* extension module imports worked). Returning the module object from the init function still works though - we just need to make sure the extension loader can cope with the init function returning a non-module object :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Thu, Jan 31, 2013 at 5:50 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
One benefit of this is that sys.modules could be made a [carefully done] property that wraps interp->modules, making sys.modules truly replaceable, which would simplify a number of different places where we've had to work around it. -eric
participants (10)
-
Chris Angelico
-
Eric Snow
-
Ethan Furman
-
Larry Hastings
-
M.-A. Lemburg
-
Nick Coghlan
-
Stefan Behnel
-
Steven D'Aprano
-
Terry Reedy
-
Yuval Greenfield