Mailman 3 Allow mutable builtin types (optionally) - Python-ideas

newer
Add an optional type file for Type...

Allow mutable builtin types (optionally)

Eloi Gaudry

7 May 2018 7 May '18

5:37 p.m.

Hi, I'd like to bring back this discussion (from 2005, by Greg): https://bugs.python.org/issue1229239 Briefly, non-heap types cannot have their attributes changed by Python code. This makes sense for python builtin types, but not for the types defined in extension/modules. As we have been using this patch for the very same reasons and for more than 10 years, I think it might make sense to reconsider the discussion that Greg started. The main question at that time was "why not using a heap type instead ?" (because heap-type do not have this limitation). But I think that the right question could have been "why imposing such a restriction on non-heap types as defined in (non Python core) extensions ?". I mean, to my knowledge, there is no reason why a type should be allocated on the heap (https://docs.python.org/2/c-api/typeobj.html) to be able to change its attributes at Python level. I'm not saying that all non-heap types should be able to do so, just that it would make sense to allow this behavior, as an option (bit flag). At the implementation level, the changes needed are really limited (about a few lines): - Include/object.h - Objects/typeobject.c: Eloi

Show replies by date

Petr Viktorin

7 May 7 May

9:23 p.m.

On 05/07/18 11:37, Eloi Gaudry wrote:

...

Hi,

I'd like to bring back this discussion (from 2005, by Greg): https://bugs.python.org/issue1229239

Briefly, non-heap types cannot have their attributes changed by Python code. This makes sense for python builtin types, but not for the types defined in extension/modules.

As we have been using this patch for the very same reasons and for more than 10 years, I think it might make sense to reconsider the discussion that Greg started.

The main question at that time was "why not using a heap type instead ?" (because heap-type do not have this limitation).

But I think that the right question could have been "why imposing such a restriction on non-heap types as defined in (non Python core) extensions ?".

I mean, to my knowledge, there is no reason why a type should be allocated on the heap (https://docs.python.org/2/c-api/typeobj.html) to be able to change its attributes at Python level.

One reason is sub-interpreter support: you can have multiple interpreters per process, and those shouldn't influence each other. (see https://docs.python.org/3/c-api/init.html#sub-interpreter-support) With heap types, each sub-interpreter can have its own copy of the type object. But with builtins, changes done in one interpreter would be visible in all the others.

...

I'm not saying that all non-heap types should be able to do so, just that it would make sense to allow this behavior, as an option (bit flag).

At the implementation level, the changes needed are really limited (about a few lines): - Include/object.h - Objects/typeobject.c:

Eloi

Eloi Gaudry

8 May 8 May

9:26 a.m.

On Mon, 2018-05-07 at 15:23 -0400, Petr Viktorin wrote:

...

On 05/07/18 11:37, Eloi Gaudry wrote:

...
I mean, to my knowledge, there is no reason why a type should be allocated on the heap (https://docs.python.org/2/c-api/typeobj.html ) to be able to change its attributes at Python level.

One reason is sub-interpreter support: you can have multiple interpreters per process, and those shouldn't influence each other. (see https://docs.python.org/3/c-api/init.html#sub-interpreter-suppor t)

With heap types, each sub-interpreter can have its own copy of the type object. But with builtins, changes done in one interpreter would be visible in all the others.

Eloi Gaudry

21 Jun 21 Jun

5:26 p.m.

This request didn't have a lot of traction, but I still consider this is something that would need to be supported (2 lines of code to be changed; no regression so far with python 2 and python 3). My main points are: - HEAP_TYPE is not really used (as anyone being using it ?) - HEAP_TYPE serves other purposes - extension would benefit for allowing direct access to any of its type attributes Petr, what do you think ? Eloi ________________________________ From: Python-ideas on behalf of Eloi Gaudry Sent: Tuesday, May 8, 2018 9:26:47 AM To: encukou@gmail.com; python-ideas@python.org Subject: Re: [Python-ideas] Allow mutable builtin types (optionally) On Mon, 2018-05-07 at 15:23 -0400, Petr Viktorin wrote:

...

On 05/07/18 11:37, Eloi Gaudry wrote:

...
I mean, to my knowledge, there is no reason why a type should be allocated on the heap (https://docs.python.org/2/c-api/typeobj.html ) to be able to change its attributes at Python level.

One reason is sub-interpreter support: you can have multiple interpreters per process, and those shouldn't influence each other. (see https://docs.python.org/3/c-api/init.html#sub-interpreter-suppor t)

With heap types, each sub-interpreter can have its own copy of the type object. But with builtins, changes done in one interpreter would be visible in all the others.

Eloi Gaudry

26 Jun 26 Jun

4:27 p.m.

some literature: https://mail.python.org/pipermail/python-dev/2008-February/077180.html https://mail.python.org/pipermail/python-dev/2008-February/077180.html https://mail.python.org/pipermail/python-dev/2008-February/077169.html where it is stated that python C struct type should not be able to have their attributes changed. but the extension needs is clearly not taken into account. ________________________________ From: Python-ideas on behalf of Eloi Gaudry Sent: Thursday, June 21, 2018 5:26:37 PM To: python-ideas@python.org; encukou@gmail.com Subject: Re: [Python-ideas] Allow mutable builtin types (optionally) This request didn't have a lot of traction, but I still consider this is something that would need to be supported (2 lines of code to be changed; no regression so far with python 2 and python 3). My main points are: - HEAP_TYPE is not really used (as anyone being using it ?) - HEAP_TYPE serves other purposes - extension would benefit for allowing direct access to any of its type attributes Petr, what do you think ? Eloi ________________________________ From: Python-ideas on behalf of Eloi Gaudry Sent: Tuesday, May 8, 2018 9:26:47 AM To: encukou@gmail.com; python-ideas@python.org Subject: Re: [Python-ideas] Allow mutable builtin types (optionally) On Mon, 2018-05-07 at 15:23 -0400, Petr Viktorin wrote:

...

On 05/07/18 11:37, Eloi Gaudry wrote:

...
I mean, to my knowledge, there is no reason why a type should be allocated on the heap (https://docs.python.org/2/c-api/typeobj.html ) to be able to change its attributes at Python level.

One reason is sub-interpreter support: you can have multiple interpreters per process, and those shouldn't influence each other. (see https://docs.python.org/3/c-api/init.html#sub-interpreter-suppor t)

With heap types, each sub-interpreter can have its own copy of the type object. But with builtins, changes done in one interpreter would be visible in all the others.

Eloi Gaudry

4:45 p.m.

the origin of this feature disappearing for built-in types: http://bugs.jython.org/issue1058 ''' object.__set/delattr__ allow modification of built in types, this is known as the Carlo Verre hack: Jython 2.3a0+ (trunk:4630:4631M, Jun 14 2008, 20:07:38) [Java HotSpot(TM) Client VM (Apple Inc.)] on java1.5.0_13 Type "help", "copyright", "credits" or "license" for more information.

...

...
...
object.__setattr__(str, 'lower', str.upper) 'dammit Carlo!'.lower() 'DAMMIT CARLO!' '''

but I do not see any reason why having an explicit flag for python extensions written in C to declare their types as static struct, and still be able to change their __setattr__, __getattr__, etc. slots would not make sense. extensions and core types have not the same constraints and purposes, this should be reflected on the capabilities the first would have somewhere then. ________________________________ From: Eloi Gaudry Sent: Tuesday, June 26, 2018 4:27:18 PM To: python-ideas@python.org Subject: Re: [Python-ideas] Allow mutable builtin types (optionally) some literature: https://mail.python.org/pipermail/python-dev/2008-February/077180.html https://mail.python.org/pipermail/python-dev/2008-February/077180.html https://mail.python.org/pipermail/python-dev/2008-February/077169.html where it is stated that python C struct type should not be able to have their attributes changed. but the extension needs is clearly not taken into account. ________________________________ From: Python-ideas on behalf of Eloi Gaudry Sent: Thursday, June 21, 2018 5:26:37 PM To: python-ideas@python.org; encukou@gmail.com Subject: Re: [Python-ideas] Allow mutable builtin types (optionally) This request didn't have a lot of traction, but I still consider this is something that would need to be supported (2 lines of code to be changed; no regression so far with python 2 and python 3). My main points are: - HEAP_TYPE is not really used (as anyone being using it ?) - HEAP_TYPE serves other purposes - extension would benefit for allowing direct access to any of its type attributes Petr, what do you think ? Eloi ________________________________ From: Python-ideas on behalf of Eloi Gaudry Sent: Tuesday, May 8, 2018 9:26:47 AM To: encukou@gmail.com; python-ideas@python.org Subject: Re: [Python-ideas] Allow mutable builtin types (optionally) On Mon, 2018-05-07 at 15:23 -0400, Petr Viktorin wrote:

...

On 05/07/18 11:37, Eloi Gaudry wrote:

...
I mean, to my knowledge, there is no reason why a type should be allocated on the heap (https://docs.python.org/2/c-api/typeobj.html ) to be able to change its attributes at Python level.

One reason is sub-interpreter support: you can have multiple interpreters per process, and those shouldn't influence each other. (see https://docs.python.org/3/c-api/init.html#sub-interpreter-suppor t)

With heap types, each sub-interpreter can have its own copy of the type object. But with builtins, changes done in one interpreter would be visible in all the others.

Guido van Rossum

7:19 p.m.

Hey Eloi, I think you need to just give up on this. Nobody here seems to support or understand your use case. At this point you are repeating yourself (again claiming there is no good reason for the prohibition and that it's only a few lines of code to change) and you can be assured that the response will also be the same. --Guido On Tue, Jun 26, 2018 at 8:00 AM Eloi Gaudry wrote:

...

the origin of this feature disappearing for built-in types:

http://bugs.jython.org/issue1058

'''

object.__set/delattr__ allow modification of built in types, this is known as the Carlo Verre hack:

Jython 2.3a0+ (trunk:4630:4631M, Jun 14 2008, 20:07:38) [Java HotSpot(TM) Client VM (Apple Inc.)] on java1.5.0_13 Type "help", "copyright", "credits" or "license" for more information.

...
...
...
object.__setattr__(str, 'lower', str.upper) 'dammit Carlo!'.lower() 'DAMMIT CARLO!' '''

but I do not see any reason why having an explicit flag for python extensions written in C to declare their types as static struct, and still be able to change their __setattr__, __getattr__, etc. slots would not make sense.

extensions and core types have not the same constraints and purposes, this should be reflected on the capabilities the first would have somewhere then.

------------------------------ *From:* Eloi Gaudry *Sent:* Tuesday, June 26, 2018 4:27:18 PM *To:* python-ideas@python.org *Subject:* Re: [Python-ideas] Allow mutable builtin types (optionally)

some literature:

https://mail.python.org/pipermail/python-dev/2008-February/077180.html

https://mail.python.org/pipermail/python-dev/2008-February/077180.html https://mail.python.org/pipermail/python-dev/2008-February/077169.html

where it is stated that python C struct type should not be able to have their attributes changed.

but the extension needs is clearly not taken into account.

------------------------------ *From:* Python-ideas on behalf of Eloi Gaudry *Sent:* Thursday, June 21, 2018 5:26:37 PM *To:* python-ideas@python.org; encukou@gmail.com *Subject:* Re: [Python-ideas] Allow mutable builtin types (optionally)

This request didn't have a lot of traction, but I still consider this is something that would need to be supported (2 lines of code to be changed; no regression so far with python 2 and python 3).

My main points are:

- HEAP_TYPE is not really used (as anyone being using it ?)

- HEAP_TYPE serves other purposes

- extension would benefit for allowing direct access to any of its type attributes

Petr, what do you think ?

Eloi ------------------------------ *From:* Python-ideas on behalf of Eloi Gaudry *Sent:* Tuesday, May 8, 2018 9:26:47 AM *To:* encukou@gmail.com; python-ideas@python.org *Subject:* Re: [Python-ideas] Allow mutable builtin types (optionally)

On Mon, 2018-05-07 at 15:23 -0400, Petr Viktorin wrote:

...
On 05/07/18 11:37, Eloi Gaudry wrote:

...
I mean, to my knowledge, there is no reason why a type should be allocated on the heap (https://docs.python.org/2/c-api/typeobj.html ) to be able to change its attributes at Python level.

One reason is sub-interpreter support: you can have multiple interpreters per process, and those shouldn't influence each other. (see https://docs.python.org/3/c-api/init.html#sub-interpreter-suppor t)

With heap types, each sub-interpreter can have its own copy of the type object. But with builtins, changes done in one interpreter would be visible in all the others.

Yes, this could be a reason, but if you don't rely on such a feature neither implicitly nor explicitly ?

I mean, our types are built-in and should be considered as immutable across interpreters. And we (as most users I guess) are only running one interpreter.

In case several intepreters are used, it would make sense to have a non-heap type that would be seen as a singleton across all of them, no ? _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

-- --Guido van Rossum (python.org/~guido)

Eloi Gaudry

9:49 p.m.

Hi Guido, I would like to be sure that the lack of support would not be the result of my inability to sum-up my use case. This is why I gave some links to illustrate - the reason why the behavior was changed a decade ago - that such a possibility was actually needed by other extension developers (where their built-in types would benefit from being able to redefine some method dynamically) - that python core developers and python extension developers can have different needs and objectives (which was the main reason why I was submitting this to the mailing-list again) I feel sorry if that only resulted in looking like I was repeating myself. Have a good day, Eloi From: Guido van Rossum Sent: Tuesday, June 26, 2018 7:20 PM To: Eloi Gaudry Cc: Python-Ideas ; Serhiy Storchaka Subject: Re: [Python-ideas] Allow mutable builtin types (optionally) Hey Eloi, I think you need to just give up on this. Nobody here seems to support or understand your use case. At this point you are repeating yourself (again claiming there is no good reason for the prohibition and that it's only a few lines of code to change) and you can be assured that the response will also be the same. --Guido On Tue, Jun 26, 2018 at 8:00 AM Eloi Gaudry mailto:Eloi.Gaudry@fft.be> wrote: the origin of this feature disappearing for built-in types: http://bugs.jython.org/issue1058 ''' object.__set/delattr__ allow modification of built in types, this is known as the Carlo Verre hack: Jython 2.3a0+ (trunk:4630:4631M, Jun 14 2008, 20:07:38) [Java HotSpot(TM) Client VM (Apple Inc.)] on java1.5.0_13 Type "help", "copyright", "credits" or "license" for more information.

...

...
...
object.__setattr__(str, 'lower', str.upper) 'dammit Carlo!'.lower() 'DAMMIT CARLO!' '''

but I do not see any reason why having an explicit flag for python extensions written in C to declare their types as static struct, and still be able to change their __setattr__, __getattr__, etc. slots would not make sense. extensions and core types have not the same constraints and purposes, this should be reflected on the capabilities the first would have somewhere then. ________________________________ From: Eloi Gaudry Sent: Tuesday, June 26, 2018 4:27:18 PM To: python-ideas@python.orgmailto:python-ideas@python.org Subject: Re: [Python-ideas] Allow mutable builtin types (optionally) some literature: https://mail.python.org/pipermail/python-dev/2008-February/077180.html https://mail.python.org/pipermail/python-dev/2008-February/077169.html where it is stated that python C struct type should not be able to have their attributes changed. but the extension needs is clearly not taken into account. ________________________________ From: Python-ideas mailto:fft.be@python.org> on behalf of Eloi Gaudry mailto:Eloi.Gaudry@fft.be> Sent: Thursday, June 21, 2018 5:26:37 PM To: python-ideas@python.orgmailto:python-ideas@python.org; encukou@gmail.commailto:encukou@gmail.com Subject: Re: [Python-ideas] Allow mutable builtin types (optionally) This request didn't have a lot of traction, but I still consider this is something that would need to be supported (2 lines of code to be changed; no regression so far with python 2 and python 3). My main points are: - HEAP_TYPE is not really used (as anyone being using it ?) - HEAP_TYPE serves other purposes - extension would benefit for allowing direct access to any of its type attributes Petr, what do you think ? Eloi ________________________________ From: Python-ideas mailto:fft.be@python.org> on behalf of Eloi Gaudry mailto:Eloi.Gaudry@fft.be> Sent: Tuesday, May 8, 2018 9:26:47 AM To: encukou@gmail.commailto:encukou@gmail.com; python-ideas@python.orgmailto:python-ideas@python.org Subject: Re: [Python-ideas] Allow mutable builtin types (optionally) On Mon, 2018-05-07 at 15:23 -0400, Petr Viktorin wrote:

...

On 05/07/18 11:37, Eloi Gaudry wrote:

...
I mean, to my knowledge, there is no reason why a type should be allocated on the heap (https://docs.python.org/2/c-api/typeobj.html ) to be able to change its attributes at Python level.

One reason is sub-interpreter support: you can have multiple interpreters per process, and those shouldn't influence each other. (see https://docs.python.org/3/c-api/init.html#sub-interpreter-suppor t)

With heap types, each sub-interpreter can have its own copy of the type object. But with builtins, changes done in one interpreter would be visible in all the others.

Yes, this could be a reason, but if you don't rely on such a feature neither implicitly nor explicitly ? I mean, our types are built-in and should be considered as immutable across interpreters. And we (as most users I guess) are only running one interpreter. In case several intepreters are used, it would make sense to have a non-heap type that would be seen as a singleton across all of them, no ? _______________________________________________ Python-ideas mailing list Python-ideas@python.orgmailto:Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ -- --Guido van Rossum (python.org/~guidohttp://python.org/~guido)

Brett Cannon

8:53 p.m.

On Thu, Jun 21, 2018, 12:27 Eloi Gaudry, wrote:

...

This request didn't have a lot of traction, but I still consider this is something that would need to be supported

Please be careful about using the word "need" as it comes off as demanding instead of as a suggestion. -Brett (2 lines of code to be changed; no regression so far with python 2 and

...

python 3).

My main points are:

- HEAP_TYPE is not really used (as anyone being using it ?)

- HEAP_TYPE serves other purposes

- extension would benefit for allowing direct access to any of its type attributes

...

Petr, what do you think ?

Eloi ------------------------------ *From:* Python-ideas on behalf of Eloi Gaudry *Sent:* Tuesday, May 8, 2018 9:26:47 AM *To:* encukou@gmail.com; python-ideas@python.org *Subject:* Re: [Python-ideas] Allow mutable builtin types (optionally)

On Mon, 2018-05-07 at 15:23 -0400, Petr Viktorin wrote:

...
On 05/07/18 11:37, Eloi Gaudry wrote:

...
I mean, to my knowledge, there is no reason why a type should be allocated on the heap (https://docs.python.org/2/c-api/typeobj.html ) to be able to change its attributes at Python level.

One reason is sub-interpreter support: you can have multiple interpreters per process, and those shouldn't influence each other. (see https://docs.python.org/3/c-api/init.html#sub-interpreter-suppor t)

With heap types, each sub-interpreter can have its own copy of the type object. But with builtins, changes done in one interpreter would be visible in all the others.

Yes, this could be a reason, but if you don't rely on such a feature neither implicitly nor explicitly ?

I mean, our types are built-in and should be considered as immutable across interpreters. And we (as most users I guess) are only running one interpreter.

In case several intepreters are used, it would make sense to have a non-heap type that would be seen as a singleton across all of them, no ? _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

Eloi Gaudry

9:38 p.m.

Hi Brett, Sorry about that, I did not mean to be rude. What I wanted to says is: 1. That I relied on such a feature 2. Other people on this mailing-list already asked something similar at several occasions 3. HEAPTYPE would not always be a solution 4. Then I thought this was something that would indeed need more discussion and might get acceptance if discussed once again. Eloi From: Brett Cannon Sent: Tuesday, June 26, 2018 8:53 PM To: Eloi Gaudry Cc: encukou@gmail.com; python-ideas@python.org Subject: Re: [Python-ideas] Allow mutable builtin types (optionally) On Thu, Jun 21, 2018, 12:27 Eloi Gaudry, mailto:Eloi.Gaudry@fft.be> wrote: This request didn't have a lot of traction, but I still consider this is something that would need to be supported Please be careful about using the word "need" as it comes off as demanding instead of as a suggestion. -Brett (2 lines of code to be changed; no regression so far with python 2 and python 3). My main points are: - HEAP_TYPE is not really used (as anyone being using it ?) - HEAP_TYPE serves other purposes - extension would benefit for allowing direct access to any of its type attributes Petr, what do you think ? Eloi ________________________________ From: Python-ideas mailto:fft.be@python.org> on behalf of Eloi Gaudry mailto:Eloi.Gaudry@fft.be> Sent: Tuesday, May 8, 2018 9:26:47 AM To: encukou@gmail.commailto:encukou@gmail.com; python-ideas@python.orgmailto:python-ideas@python.org Subject: Re: [Python-ideas] Allow mutable builtin types (optionally) On Mon, 2018-05-07 at 15:23 -0400, Petr Viktorin wrote:

...

On 05/07/18 11:37, Eloi Gaudry wrote:

...
I mean, to my knowledge, there is no reason why a type should be allocated on the heap (https://docs.python.org/2/c-api/typeobj.html ) to be able to change its attributes at Python level.

One reason is sub-interpreter support: you can have multiple interpreters per process, and those shouldn't influence each other. (see https://docs.python.org/3/c-api/init.html#sub-interpreter-suppor t)

With heap types, each sub-interpreter can have its own copy of the type object. But with builtins, changes done in one interpreter would be visible in all the others.

Yes, this could be a reason, but if you don't rely on such a feature neither implicitly nor explicitly ? I mean, our types are built-in and should be considered as immutable across interpreters. And we (as most users I guess) are only running one interpreter. In case several intepreters are used, it would make sense to have a non-heap type that would be seen as a singleton across all of them, no ? _______________________________________________ Python-ideas mailing list Python-ideas@python.orgmailto:Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ _______________________________________________ Python-ideas mailing list Python-ideas@python.orgmailto:Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

2128

Age (days ago)

2178

Last active (days ago)

List overview

Download

9 comments

4 participants

participants (4)

Brett Cannon
Eloi Gaudry
Guido van Rossum
Petr Viktorin

Allow mutable builtin types (optionally)

tags

participants (4)