State of PEP-3118 (memoryview part)
State of PEP-3118 (memoryview part) Hello, In Python 3.3 most issues with the memoryview object have been fixed in a recent commit (3f9b3b6f7ff0). Many features have been added, see: http://docs.python.org/dev/whatsnew/3.3.html The underlying problems with memoryview were intricate and required a long discussion (issue #10181) that led to a complete rewrite of memoryobject.c. We have several options with regard to 2.7 and 3.2: 1) Won't fix. 2) Backport the changes and disable as much of the new functionality as possible. 3) Backport all of it (this would be the least amount of work and could be done relatively quickly). 4) Nick suggested another option: put a module with the new functionality on PyPI. This would be quite a bit of work, and personally I don't have time for that. Options 2) and 3) would ideally entail one backwards incompatible bugfix: In 2.7 and 3.2 assignment to a memoryview with format 'B' rejects integers but accepts byte objects, but according to the struct syntax mandated by the PEP it should be the other way round. It would be nice to get some opinions and ideas, especially of course from the release managers. Stefan Krah
On Sun, 26 Feb 2012 14:27:21 +0100
Stefan Krah
The underlying problems with memoryview were intricate and required a long discussion (issue #10181) that led to a complete rewrite of memoryobject.c.
We have several options with regard to 2.7 and 3.2:
1) Won't fix.
Given the extent of the rewrite, this one has my preference. Regards Antoine.
+1 for won't fix.
On Feb 26, 2012 9:46 PM, "Antoine Pitrou"
On Sun, 26 Feb 2012 14:27:21 +0100 Stefan Krah
wrote: The underlying problems with memoryview were intricate and required a long discussion (issue #10181) that led to a complete rewrite of memoryobject.c.
We have several options with regard to 2.7 and 3.2:
1) Won't fix.
Given the extent of the rewrite, this one has my preference.
Regards
Antoine.
_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/anacrolix%40gmail.com
Stefan Krah wrote:
Options 2) and 3) would ideally entail one backwards incompatible bugfix: In 2.7 and 3.2 assignment to a memoryview with format 'B' rejects integers but accepts byte objects, but according to the struct syntax mandated by the PEP it should be the other way round.
Maybe a compromise could be made to accept both in the backport? That would avoid breaking old code while allowing code that does the right thing to work. -- Greg
Stefan, thank you for the massive rewrite. On 2/26/2012 11:21 AM, Paul Moore wrote:
On 26 February 2012 13:41, Antoine Pitrou
wrote: We have several options with regard to 2.7 and 3.2:
1) Won't fix.
Given the extent of the rewrite, this one has my preference.
+1 (although I'd word it as "fixed in 3.3" rather than "won't fix").
I agree with 3.3 only. My suggestion: when you close the issues, change Versions to 3.3 and Resolution to 'fixed'. On the main issue, change Type to 'enhancement'. Add a message to others saying something like "This was fixed for 3.3 in #xxxxx by rewriting and enhancing memoryview. The new version was not backported because it would either add new features, which is forbidden for bugfix releases, or would require substantial work to try to disable them without introducing bugs, without a guarantee that that would work" -- Terry Jan Reedy
Greg Ewing
Options 2) and 3) would ideally entail one backwards incompatible bugfix: In 2.7 and 3.2 assignment to a memoryview with format 'B' rejects integers but accepts byte objects, but according to the struct syntax mandated by the PEP it should be the other way round.
Maybe a compromise could be made to accept both in the backport? That would avoid breaking old code while allowing code that does the right thing to work.
This could definitely be done. But backporting is beginning to look unlikely, since we currently have three +1 for "too complex to backport". I'm not strongly in favor of backporting myself. The main reason for me would be to prevent having additional 2->3 or 3->2 porting obstacles. Stefan Krah
On 2/29/2012 2:34 PM, Stefan Krah wrote:
Greg Ewing
wrote: Options 2) and 3) would ideally entail one backwards incompatible bugfix: In 2.7 and 3.2 assignment to a memoryview with format 'B' rejects integers but accepts byte objects, but according to the struct syntax mandated by the PEP it should be the other way round.
Maybe a compromise could be made to accept both in the backport? That would avoid breaking old code while allowing code that does the right thing to work.
This *almost* sounds like a feature addition.
This could definitely be done. But backporting is beginning to look unlikely, since we currently have three +1 for "too complex to backport".
I'm not strongly in favor of backporting myself. The main reason for me would be to prevent having additional 2->3 or 3->2 porting obstacles.
Stefan Krah
-- Terry Jan Reedy
[erroneouly hit send button before instead of edit menu above it] On 2/29/2012 2:34 PM, Stefan Krah wrote:
Greg Ewing
wrote: Options 2) and 3) would ideally entail one backwards incompatible bugfix: In 2.7 and 3.2 assignment to a memoryview with format 'B' rejects integers but accepts byte objects, but according to the struct syntax mandated by the PEP it should be the other way round.
If implementation and PEP conflict, the normal question is 'what does the doc say?' as doc takes precedent over PEP. However, in this case the 'MemoryView objects' section under 'Concrete objects' says nothing about the above that I could see and refers to Buffer Protocal in Abstract Objects Layer. I did not see anything there either, but could have missed it.
Maybe a compromise could be made to accept both in the backport? That would avoid breaking old code while allowing code that does the right thing to work.
This looks a bit like an enhancement ;-)
This could definitely be done. But backporting is beginning to look unlikely, since we currently have three +1 for "too complex to backport".
My comment was more 'unnecessary to backpart'. This is based on the following thoughts (which could have mistakes). * I do not see enough benefit that I could wish you to write or anyone else to review a bugfix patch. I would in no way stop you if this continue to itch you ;-). * Sorting out bugfix changes from feature looks complex and possibly contentious and might take some time to discuss. * 3.2.3 is, I presume, less than a month away, and if that is missed, the next and last bugfix will be 3.2.4 at about the same time as 3.3.0. At that time, the full new memoryview version would be a better target. * As for porting, my impression is that the PEP directly affects only C code and Python code using ctypes and only some fraction of those. If the bugfix-only patch is significantly different from complete patch, porting to 3.2 would be significantly different from porting to 3.3. So I can foresee a temptation to just port to 3.3 anyway.
I'm not strongly in favor of backporting myself. The main reason for me would be to prevent having additional 2->3 or 3->2 porting obstacles.
-- Terry Jan Reedy
On Thu, Mar 1, 2012 at 11:48 AM, Terry Reedy
* As for porting, my impression is that the PEP directly affects only C code and Python code using ctypes and only some fraction of those. If the bugfix-only patch is significantly different from complete patch, porting to 3.2 would be significantly different from porting to 3.3. So I can foresee a temptation to just port to 3.3 anyway.
memoryview as it exists in 2.7 and 3.2 misbehaves when used with certain buffer exporters - while Antoine bashed it into shape (mostly) for 1D views into 1D objects, it's rather temperamental if you try to go beyond that. So it affects the Python level as well, in terms of what objects are likely to upset memoryview. Still, I think backporting would be a lot of work for relatively small benefit, so it ends up in my "with infinite resources, sure, but with limited resources, there are more fruitful things to be doing" pile. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Terry Reedy
Options 2) and 3) would ideally entail one backwards incompatible bugfix: In 2.7 and 3.2 assignment to a memoryview with format 'B' rejects integers but accepts byte objects, but according to the struct syntax mandated by the PEP it should be the other way round.
If implementation and PEP conflict, the normal question is 'what does the doc say?' as doc takes precedent over PEP. However, in this case the 'MemoryView objects' section under 'Concrete objects' says nothing about the above that I could see and refers to Buffer Protocal in Abstract Objects Layer. I did not see anything there either, but could have missed it.
For the C-API, it's here: http://docs.python.org/py3k/c-api/buffer.html#the-buffer-structure const char *format A NULL terminated string in struct module style syntax giving the contents of the elements available through the buffer. If this is NULL, "B" (unsigned bytes) is assumed. Unfortunately, the memoryview documentation itself gives examples like these: http://docs.python.org/py3k/library/stdtypes.html#typememoryview data = bytearray(b'abcefg') v = memoryview(data) v[0] = b'z' # That's where the struct module would throw an error.
* 3.2.3 is, I presume, less than a month away, and if that is missed, the next and last bugfix will be 3.2.4 at about the same time as 3.3.0.
That would be too soon indeed.
* As for porting, my impression is that the PEP directly affects only C code and Python code using ctypes and only some fraction of those. If the bugfix-only patch is significantly different from complete patch, porting to 3.2 would be significantly different from porting to 3.3. So I can foresee a temptation to just port to 3.3 anyway.
The general problem is this: If someone supports 2 and 3 and uses the single codebase approach, it's unlikely that new features will ever be used. Even if a new 3.3 project is started that needs to be backwards compatible, I can imagine that people will shun anything that 3to2 isn't able to handle (out of the box). This would be less of an issue if the officially sanctioned way of porting were the "separate branches with 2to3 (or 3to2) for the initial conversion" approach. Stefan Krah
participants (7)
-
Antoine Pitrou
-
Greg Ewing
-
Matt Joiner
-
Nick Coghlan
-
Paul Moore
-
Stefan Krah
-
Terry Reedy