Intention to accept PEP 552 soon (deterministic pyc files)
I've read the current version of PEP 552 over and I think everything looks good for acceptance. I believe there are no outstanding objections (or they have been adequately addressed in responses). Therefore I intend to accept PEP 552 this Friday, unless grave objections are raised on this mailing list (python-dev). Congratulations Benjamin. Gotta love those tristate options! -- --Guido van Rossum (python.org/~guido)
It's Friday! There have been no further comments. PEP 550 is now accepted. Congrats, Benjamin! Go ahead and send your implementation for review. --Guido On Tue, Sep 26, 2017 at 1:47 PM, Guido van Rossum <guido@python.org> wrote:
I've read the current version of PEP 552 over and I think everything looks good for acceptance. I believe there are no outstanding objections (or they have been adequately addressed in responses).
Therefore I intend to accept PEP 552 this Friday, unless grave objections are raised on this mailing list (python-dev).
Congratulations Benjamin. Gotta love those tristate options!
-- --Guido van Rossum (python.org/~guido)
-- --Guido van Rossum (python.org/~guido)
Hi all, Do the accepted PEP were 552, not 550? Thanks, Louie. 2017-09-29 22:40 GMT+08:00 Guido van Rossum <guido@python.org>:
It's Friday!
There have been no further comments. PEP 550 is now accepted.
Congrats, Benjamin! Go ahead and send your implementation for review.
--Guido
On Tue, Sep 26, 2017 at 1:47 PM, Guido van Rossum <guido@python.org> wrote:
I've read the current version of PEP 552 over and I think everything looks good for acceptance. I believe there are no outstanding objections (or they have been adequately addressed in responses).
Therefore I intend to accept PEP 552 this Friday, unless grave objections are raised on this mailing list (python-dev).
Congratulations Benjamin. Gotta love those tristate options!
-- --Guido van Rossum (python.org/~guido)
-- --Guido van Rossum (python.org/~guido)
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/me%40louie.lu
Let me try that again. There have been no further comments. PEP 552 is now accepted. Congrats, Benjamin! Go ahead and send your implementation for review.Oops. Let me try that again. PS. PEP 550 is still unaccepted, awaiting a new revision from Yury and Elvis. -- --Guido van Rossum (python.org/~guido <http://python.org/%7Eguido>)
Thanks, Guido and everyone who submitted feedback! I guess I know what I'll be doing this weekend. On Fri, Sep 29, 2017, at 08:18, Guido van Rossum wrote:
Let me try that again.
There have been no further comments. PEP 552 is now accepted.
Congrats, Benjamin! Go ahead and send your implementation for review.Oops. Let me try that again.
PS. PEP 550 is still unaccepted, awaiting a new revision from Yury and> Elvis.
-- --Guido van Rossum (python.org/~guido <http://python.org/%7Eguido>) _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/benjamin%40python.org
BTW, if you find the bytecode-specific APIs are sub-par while trying to update them, let me know as I have been toying with cleaning them up and centralizing them under importlib for a while and just never gotten around to sitting down and coming up with a better design that warranted putting the time into it. :) On Fri, 29 Sep 2017 at 09:17 Benjamin Peterson <benjamin@python.org> wrote:
Thanks, Guido and everyone who submitted feedback!
I guess I know what I'll be doing this weekend.
On Fri, Sep 29, 2017, at 08:18, Guido van Rossum wrote:
Let me try that again.
There have been no further comments. PEP 552 is now accepted.
Congrats, Benjamin! Go ahead and send your implementation for review.Oops. Let me try that again.
PS. PEP 550 is still unaccepted, awaiting a new revision from Yury and Elvis.
-- --Guido van Rossum (python.org/~guido <http://python.org/%7Eguido>) _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/benjamin%40python.org
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org
What do you mean by bytecode-specific APIs? The internal importlib ones? On Fri, Sep 29, 2017, at 09:38, Brett Cannon wrote:
BTW, if you find the bytecode-specific APIs are sub-par while trying to update them, let me know as I have been toying with cleaning them up and centralizing them under importlib for a while and just never gotten around to sitting down and coming up with a better design that warranted putting the time into it. :)
On Fri, 29 Sep 2017 at 09:17 Benjamin Peterson <benjamin@python.org> wrote:
Thanks, Guido and everyone who submitted feedback!
I guess I know what I'll be doing this weekend.
On Fri, Sep 29, 2017, at 08:18, Guido van Rossum wrote:
Let me try that again.
There have been no further comments. PEP 552 is now accepted.
Congrats, Benjamin! Go ahead and send your implementation for review.Oops. Let me try that again.
PS. PEP 550 is still unaccepted, awaiting a new revision from Yury and Elvis.
-- --Guido van Rossum (python.org/~guido <http://python.org/%7Eguido>) _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/benjamin%40python.org
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org
On Sat, 30 Sep 2017 at 18:46 Benjamin Peterson <benjamin@python.org> wrote:
What do you mean by bytecode-specific APIs? The internal importlib ones?
There's that, but more specifically py_compile and compileall. -Brett
On Fri, Sep 29, 2017, at 09:38, Brett Cannon wrote:
BTW, if you find the bytecode-specific APIs are sub-par while trying to update them, let me know as I have been toying with cleaning them up and centralizing them under importlib for a while and just never gotten around to sitting down and coming up with a better design that warranted putting the time into it. :)
On Fri, 29 Sep 2017 at 09:17 Benjamin Peterson <benjamin@python.org> wrote:
Thanks, Guido and everyone who submitted feedback!
I guess I know what I'll be doing this weekend.
On Fri, Sep 29, 2017, at 08:18, Guido van Rossum wrote:
Let me try that again.
There have been no further comments. PEP 552 is now accepted.
Congrats, Benjamin! Go ahead and send your implementation for review.Oops. Let me try that again.
PS. PEP 550 is still unaccepted, awaiting a new revision from Yury and Elvis.
-- --Guido van Rossum (python.org/~guido <http://python.org/%7Eguido>) _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe:
https://mail.python.org/mailman/options/python-dev/benjamin%40python.org
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org
On Sep 29, 2017 18:21, "Guido van Rossum" <guido@python.org> wrote: PS. PEP 550 is still unaccepted, awaiting a new revision from Yury and Elvis. This is getting really off-topic, but I do have updates to add to PEP 555 if there is interest in that. IMO, 555 is better and most likely faster than 550, but on the other hand, the issues with PEP 550 are most likely not going to be a problem for me personally. -- Koos
Your PEP is currently incomplete. If you don't finish it, it is not even a contender. But TBH it's not my favorite anyway, so you could also just withdraw it. On Oct 1, 2017 9:13 AM, "Koos Zevenhoven" <k7hoven@gmail.com> wrote:
On Sep 29, 2017 18:21, "Guido van Rossum" <guido@python.org> wrote:
PS. PEP 550 is still unaccepted, awaiting a new revision from Yury and Elvis.
This is getting really off-topic, but I do have updates to add to PEP 555 if there is interest in that. IMO, 555 is better and most likely faster than 550, but on the other hand, the issues with PEP 550 are most likely not going to be a problem for me personally.
-- Koos
On Oct 1, 2017 19:26, "Guido van Rossum" <guido@python.org> wrote: Your PEP is currently incomplete. If you don't finish it, it is not even a contender. But TBH it's not my favorite anyway, so you could also just withdraw it. I can withdraw it if you ask me to, but I don't want to withdraw it without any reason. I haven't changed my mind about the big picture. OTOH, PEP 521 is elegant and could be used to implement PEP 555, but 521 is almost certainly less performant and has some problems regarding context manager wrappers that use composition instead of inheritance. -- Koos On Oct 1, 2017 9:13 AM, "Koos Zevenhoven" <k7hoven@gmail.com> wrote:
On Sep 29, 2017 18:21, "Guido van Rossum" <guido@python.org> wrote:
PS. PEP 550 is still unaccepted, awaiting a new revision from Yury and Elvis.
This is getting really off-topic, but I do have updates to add to PEP 555 if there is interest in that. IMO, 555 is better and most likely faster than 550, but on the other hand, the issues with PEP 550 are most likely not going to be a problem for me personally.
-- Koos
On Sun, Oct 1, 2017 at 1:52 PM, Koos Zevenhoven <k7hoven@gmail.com> wrote:
On Oct 1, 2017 19:26, "Guido van Rossum" <guido@python.org> wrote:
Your PEP is currently incomplete. If you don't finish it, it is not even a contender. But TBH it's not my favorite anyway, so you could also just withdraw it.
I can withdraw it if you ask me to, but I don't want to withdraw it without any reason. I haven't changed my mind about the big picture. OTOH, PEP 521 is elegant and could be used to implement PEP 555, but 521 is almost certainly less performant and has some problems regarding context manager wrappers that use composition instead of inheritance.
It is my understanding that PEP 521 (which proposes to add optional __suspend__ and __resume__ methods to the context manager protocol, to be called whenever a frame is suspended or resumed inside a `with` block) is no longer a contender because it would be way too slow. I haven't read it recently or thought about it, so I don't know what the second issue you mention is about (though it's presumably about the `yield` in a context manager implemented using a generator decorated with `@contextlib.contextmanager`). So it's really between PEP 550 and PEP 555. And there are currently too many parts of PEP 555 that are left to the imagination of the reader. So, again, I ask you to put up or shut up. It's your choice. If you don't want to do the work completing the PEP you might as well withdraw (once I am satisfied with Yury's PEP I will just accept it when there's no contender). If you do complete it I will probably still choose PEP 550 -- but at the moment the choice would be between something I understand completely and something that's too poorly specified to be able to reason about it. --Guido
-- Koos
On Oct 1, 2017 9:13 AM, "Koos Zevenhoven" <k7hoven@gmail.com> wrote:
On Sep 29, 2017 18:21, "Guido van Rossum" <guido@python.org> wrote:
PS. PEP 550 is still unaccepted, awaiting a new revision from Yury and Elvis.
This is getting really off-topic, but I do have updates to add to PEP 555 if there is interest in that. IMO, 555 is better and most likely faster than 550, but on the other hand, the issues with PEP 550 are most likely not going to be a problem for me personally.
-- Koos
-- --Guido van Rossum (python.org/~guido)
One more thing. I would really appreciate it if you properly wrapped lines in your PEP around column 72 instead of using a single line per paragraph. This is the standard convention, see the template in PEP 12. On Sun, Oct 1, 2017 at 8:42 PM, Guido van Rossum <guido@python.org> wrote:
On Sun, Oct 1, 2017 at 1:52 PM, Koos Zevenhoven <k7hoven@gmail.com> wrote:
On Oct 1, 2017 19:26, "Guido van Rossum" <guido@python.org> wrote:
Your PEP is currently incomplete. If you don't finish it, it is not even a contender. But TBH it's not my favorite anyway, so you could also just withdraw it.
I can withdraw it if you ask me to, but I don't want to withdraw it without any reason. I haven't changed my mind about the big picture. OTOH, PEP 521 is elegant and could be used to implement PEP 555, but 521 is almost certainly less performant and has some problems regarding context manager wrappers that use composition instead of inheritance.
It is my understanding that PEP 521 (which proposes to add optional __suspend__ and __resume__ methods to the context manager protocol, to be called whenever a frame is suspended or resumed inside a `with` block) is no longer a contender because it would be way too slow. I haven't read it recently or thought about it, so I don't know what the second issue you mention is about (though it's presumably about the `yield` in a context manager implemented using a generator decorated with `@contextlib.contextmanager`).
So it's really between PEP 550 and PEP 555. And there are currently too many parts of PEP 555 that are left to the imagination of the reader. So, again, I ask you to put up or shut up. It's your choice. If you don't want to do the work completing the PEP you might as well withdraw (once I am satisfied with Yury's PEP I will just accept it when there's no contender). If you do complete it I will probably still choose PEP 550 -- but at the moment the choice would be between something I understand completely and something that's too poorly specified to be able to reason about it.
--Guido
-- Koos
On Oct 1, 2017 9:13 AM, "Koos Zevenhoven" <k7hoven@gmail.com> wrote:
On Sep 29, 2017 18:21, "Guido van Rossum" <guido@python.org> wrote:
PS. PEP 550 is still unaccepted, awaiting a new revision from Yury and Elvis.
This is getting really off-topic, but I do have updates to add to PEP 555 if there is interest in that. IMO, 555 is better and most likely faster than 550, but on the other hand, the issues with PEP 550 are most likely not going to be a problem for me personally.
-- Koos
-- --Guido van Rossum (python.org/~guido)
-- --Guido van Rossum (python.org/~guido)
On Mon, Oct 2, 2017 at 6:42 AM, Guido van Rossum <guido@python.org> wrote:
On Sun, Oct 1, 2017 at 1:52 PM, Koos Zevenhoven <k7hoven@gmail.com> wrote:
On Oct 1, 2017 19:26, "Guido van Rossum" <guido@python.org> wrote:
Your PEP is currently incomplete. If you don't finish it, it is not even a contender. But TBH it's not my favorite anyway, so you could also just withdraw it.
I can withdraw it if you ask me to, but I don't want to withdraw it without any reason. I haven't changed my mind about the big picture. OTOH, PEP 521 is elegant and could be used to implement PEP 555, but 521 is almost certainly less performant and has some problems regarding context manager wrappers that use composition instead of inheritance.
It is my understanding that PEP 521 (which proposes to add optional __suspend__ and __resume__ methods to the context manager protocol, to be called whenever a frame is suspended or resumed inside a `with` block) is no longer a contender because it would be way too slow. I haven't read it recently or thought about it, so I don't know what the second issue you mention is about (though it's presumably about the `yield` in a context manager implemented using a generator decorated with `@contextlib.contextmanager`).
Well, it's not completely unrelated to that. The problem I'm talking about is perhaps most easily seen from a simple context manager wrapper that uses composition instead of inheritance: class Wrapper: def __init__(self): self._wrapped = SomeContextManager() def __enter__(self): print("Entering context") return self._wrapped.__enter__() def __exit__(self): self._wrapped.__exit__() print("Exited context") Now, if the wrapped contextmanager becomes a PEP 521 one with __suspend__ and __resume__, the Wrapper class is broken, because it does not respect __suspend__ and __resume__. So actually this is a backwards compatiblity issue. But if the wrapper is made using inheritance, the problem goes away: class Wrapper(SomeContextManager): def __enter__(self): print("Entering context") return super().__enter__() def __exit__(self): super().__exit__() print("Exited context") Now the wrapper cleanly inherits the new optional __suspend__ and __resume__ from the wrapped context manager type. ––Koos
-- + Koos Zevenhoven + http://twitter.com/k7hoven +
Please start a new thread on python-dev. It's unrelated to "deterministic pyc files". Victor
Guido van Rossum wrote:
There have been no further comments. PEP 552 is now accepted.
Congrats, Benjamin! Go ahead and send your implementation for review.Oops. Let me try that again.
While I'm very glad PEP 552 has been accepted, it occurs to me that it will now be more difficult to parse the various pyc file formats from Python. E.g. I used to be able to just open the pyc in binary mode, read all the bytes, and then lop off the first 8 bytes to get to the code object. With the addition of the source file size, I now have to (maybe, if I have to also read old-style pyc files) lop off the front 12 bytes, but okay. With PEP 552, I have to do a lot more work to just get at the code object. How many bytes at the front of the file do I need to skip past? What about all the metadata at the front of the pyc, how do I interpret that if I want to get at it from Python code? Should the PEP 552 implementation add an API, probably to importlib.util, that would understand all current and future formats? Something like this perhaps? class PycFileSpec: magic_number: bytes timestamp: Optional[bytes] # maybe an int? datetime? source_size: Optional[bytes] bit_field: Optional[bytes] code_object: bytes def parse_pyc(path: str) -> PycFileSpec: Cheers, -Barry
It's really not that hard. You just check the magic number and if it's the new one, skip 4 words. No need to understand the internals of the header. On Oct 3, 2017 08:06, "Barry Warsaw" <barry@python.org> wrote:
Guido van Rossum wrote:
There have been no further comments. PEP 552 is now accepted.
Congrats, Benjamin! Go ahead and send your implementation for review.Oops. Let me try that again.
While I'm very glad PEP 552 has been accepted, it occurs to me that it will now be more difficult to parse the various pyc file formats from Python. E.g. I used to be able to just open the pyc in binary mode, read all the bytes, and then lop off the first 8 bytes to get to the code object. With the addition of the source file size, I now have to (maybe, if I have to also read old-style pyc files) lop off the front 12 bytes, but okay.
With PEP 552, I have to do a lot more work to just get at the code object. How many bytes at the front of the file do I need to skip past? What about all the metadata at the front of the pyc, how do I interpret that if I want to get at it from Python code?
Should the PEP 552 implementation add an API, probably to importlib.util, that would understand all current and future formats? Something like this perhaps?
class PycFileSpec: magic_number: bytes timestamp: Optional[bytes] # maybe an int? datetime? source_size: Optional[bytes] bit_field: Optional[bytes] code_object: bytes
def parse_pyc(path: str) -> PycFileSpec:
Cheers, -Barry
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ guido%40python.org
On Tue, 3 Oct 2017 08:15:04 -0700 Guido van Rossum <gvanrossum@gmail.com> wrote:
It's really not that hard. You just check the magic number and if it's the new one, skip 4 words. No need to understand the internals of the header.
Still, I agree with Barry that an API would be nice. Regards Antoine.
On Oct 3, 2017 08:06, "Barry Warsaw" <barry@python.org> wrote:
Guido van Rossum wrote:
There have been no further comments. PEP 552 is now accepted.
Congrats, Benjamin! Go ahead and send your implementation for review.Oops. Let me try that again.
While I'm very glad PEP 552 has been accepted, it occurs to me that it will now be more difficult to parse the various pyc file formats from Python. E.g. I used to be able to just open the pyc in binary mode, read all the bytes, and then lop off the first 8 bytes to get to the code object. With the addition of the source file size, I now have to (maybe, if I have to also read old-style pyc files) lop off the front 12 bytes, but okay.
With PEP 552, I have to do a lot more work to just get at the code object. How many bytes at the front of the file do I need to skip past? What about all the metadata at the front of the pyc, how do I interpret that if I want to get at it from Python code?
Should the PEP 552 implementation add an API, probably to importlib.util, that would understand all current and future formats? Something like this perhaps?
class PycFileSpec: magic_number: bytes timestamp: Optional[bytes] # maybe an int? datetime? source_size: Optional[bytes] bit_field: Optional[bytes] code_object: bytes
def parse_pyc(path: str) -> PycFileSpec:
Cheers, -Barry
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ guido%40python.org
I'm fine with adding an API, though I don't think that an API that knows about all current (historic) and future formats belongs in importlib.util -- that module only concerns itself with the *current* format. In terms of the API design I'd make take an IO[bytes] and just read and parse the header, so after that you can use marshal.load() straight from the file object. File size, mtime and bitfield should be represented as ints (the parser should take care of endianness).The hash should be a bytes. On Tue, Oct 3, 2017 at 8:24 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:
It's really not that hard. You just check the magic number and if it's
On Tue, 3 Oct 2017 08:15:04 -0700 Guido van Rossum <gvanrossum@gmail.com> wrote: the
new one, skip 4 words. No need to understand the internals of the header.
Still, I agree with Barry that an API would be nice.
Regards
Antoine.
On Oct 3, 2017 08:06, "Barry Warsaw" <barry@python.org> wrote:
Guido van Rossum wrote:
There have been no further comments. PEP 552 is now accepted.
Congrats, Benjamin! Go ahead and send your implementation for review.Oops. Let me try that again.
While I'm very glad PEP 552 has been accepted, it occurs to me that it will now be more difficult to parse the various pyc file formats from Python. E.g. I used to be able to just open the pyc in binary mode, read all the bytes, and then lop off the first 8 bytes to get to the code object. With the addition of the source file size, I now have to (maybe, if I have to also read old-style pyc files) lop off the front
bytes, but okay.
With PEP 552, I have to do a lot more work to just get at the code object. How many bytes at the front of the file do I need to skip
12 past?
What about all the metadata at the front of the pyc, how do I interpret that if I want to get at it from Python code?
Should the PEP 552 implementation add an API, probably to importlib.util, that would understand all current and future formats? Something like this perhaps?
class PycFileSpec: magic_number: bytes timestamp: Optional[bytes] # maybe an int? datetime? source_size: Optional[bytes] bit_field: Optional[bytes] code_object: bytes
def parse_pyc(path: str) -> PycFileSpec:
Cheers, -Barry
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ guido%40python.org
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ guido%40python.org
-- --Guido van Rossum (python.org/~guido)
03.10.17 18:15, Guido van Rossum пише:
It's really not that hard. You just check the magic number and if it's the new one, skip 4 words. No need to understand the internals of the header.
Hence you should know all old magic numbers to determine if the read magic number is the new one. Right?
On Tue, Oct 3, 2017, at 08:03, Barry Warsaw wrote:
Guido van Rossum wrote:
There have been no further comments. PEP 552 is now accepted.
Congrats, Benjamin! Go ahead and send your implementation for review.Oops. Let me try that again.
While I'm very glad PEP 552 has been accepted, it occurs to me that it will now be more difficult to parse the various pyc file formats from Python. E.g. I used to be able to just open the pyc in binary mode, read all the bytes, and then lop off the first 8 bytes to get to the code object. With the addition of the source file size, I now have to (maybe, if I have to also read old-style pyc files) lop off the front 12 bytes, but okay.
With PEP 552, I have to do a lot more work to just get at the code object. How many bytes at the front of the file do I need to skip past? What about all the metadata at the front of the pyc, how do I interpret that if I want to get at it from Python code?
As Guido points out, the header is just now always 4 32-bit words rather than 3. Not long ago we underwent the transition from 2-3 words without widespread disaster.
Should the PEP 552 implementation add an API, probably to importlib.util, that would understand all current and future formats? Something like this perhaps?
class PycFileSpec: magic_number: bytes timestamp: Optional[bytes] # maybe an int? datetime? source_size: Optional[bytes] bit_field: Optional[bytes] code_object: bytes
def parse_pyc(path: str) -> PycFileSpec:
I'm not sure turning the implementation details of our internal formats into APIs is the way to go.
On Oct 3, 2017, at 13:29, Benjamin Peterson <benjamin@python.org> wrote:
I'm not sure turning the implementation details of our internal formats into APIs is the way to go.
I still think an API in the stdlib would be useful and appropriate, but it’s not like this couldn’t be done as a 3rd party module. -Barry
On Wed, 4 Oct 2017 10:14:22 -0400 Barry Warsaw <barry@python.org> wrote:
On Oct 3, 2017, at 13:29, Benjamin Peterson <benjamin@python.org> wrote:
I'm not sure turning the implementation details of our internal formats into APIs is the way to go.
I still think an API in the stdlib would be useful and appropriate, but it’s not like this couldn’t be done as a 3rd party module.
It can also be an implementation-specific API for which we don't guarantee anything in the future. The consenting adults rule would apply. Regards Antoine.
On Wed, Oct 4, 2017, 09:00 Antoine Pitrou, <solipsis@pitrou.net> wrote:
On Wed, 4 Oct 2017 10:14:22 -0400 Barry Warsaw <barry@python.org> wrote:
On Oct 3, 2017, at 13:29, Benjamin Peterson <benjamin@python.org> wrote:
I'm not sure turning the implementation details of our internal formats into APIs is the way to go.
I still think an API in the stdlib would be useful and appropriate, but it’s not like this couldn’t be done as a 3rd party module.
It can also be an implementation-specific API for which we don't guarantee anything in the future. The consenting adults rule would apply.
I've toyed with the idea of coming up with an API for bytecode files, but having lived through the last file format change I could never come up with one that didn't just chop off the header or would be a maintenance burden. But having an API that followed just what was in that Python release like the ast module would solve that problem. Then 3rd-party code could wrap it it smooth out differences between versions. -Brett
Regards
Antoine.
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org
On Wed, Oct 4, 2017, at 07:14, Barry Warsaw wrote:
On Oct 3, 2017, at 13:29, Benjamin Peterson <benjamin@python.org> wrote:
I'm not sure turning the implementation details of our internal formats into APIs is the way to go.
I still think an API in the stdlib would be useful and appropriate, but it’s not like this couldn’t be done as a 3rd party module.
It might be helpful to enumerate the usecases for such an API. Perhaps a narrow, specialized API could satisfy most needs in a supportable way.
On Oct 4, 2017, at 13:53, Benjamin Peterson <benjamin@python.org> wrote:
It might be helpful to enumerate the usecases for such an API. Perhaps a narrow, specialized API could satisfy most needs in a supportable way.
Currently `python -m dis thing.py` compiles the source then disassembles it. It would be kind of cool if you could pass a .pyc file to -m dis, in which case you’d need to unpack the header to get to the code object. A naive implementation would unpack the magic number and refuse to disassemble any files that don’t match whatever that version of Python understands. A more robust (possibly 3rd party) implementation could potentially disassemble a range of magic numbers and formats, and an API to get at the code object and metadata would help. I was thinking about the bytecode hacking that some debuggers do. This API would help them support multiple versions of Python. They could use the API to discover what pyc format was in use, extract the code object, hack the bytecode and possibly rewrite a new PEP 3147 style pyc file with the debugger bytecodes inserted. Third party bytecode optimizers could use the API to unpack multiple versions of pyc files, do their optimizations, and rewrite new files with the proper format. Cheers, -Barry
Honestly I think the API for accessing historic pyc headers should itself also be 3rd party. CPython itself should not bother (backwards compatibility with pyc files has never been a feature). On Thu, Oct 5, 2017 at 6:44 AM, Barry Warsaw <barry@python.org> wrote:
On Oct 4, 2017, at 13:53, Benjamin Peterson <benjamin@python.org> wrote:
It might be helpful to enumerate the usecases for such an API. Perhaps a narrow, specialized API could satisfy most needs in a supportable way.
Currently `python -m dis thing.py` compiles the source then disassembles it. It would be kind of cool if you could pass a .pyc file to -m dis, in which case you’d need to unpack the header to get to the code object. A naive implementation would unpack the magic number and refuse to disassemble any files that don’t match whatever that version of Python understands. A more robust (possibly 3rd party) implementation could potentially disassemble a range of magic numbers and formats, and an API to get at the code object and metadata would help.
I was thinking about the bytecode hacking that some debuggers do. This API would help them support multiple versions of Python. They could use the API to discover what pyc format was in use, extract the code object, hack the bytecode and possibly rewrite a new PEP 3147 style pyc file with the debugger bytecodes inserted.
Third party bytecode optimizers could use the API to unpack multiple versions of pyc files, do their optimizations, and rewrite new files with the proper format.
Cheers, -Barry
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ guido%40python.org
-- --Guido van Rossum (python.org/~guido)
On 5 October 2017 at 23:44, Barry Warsaw <barry@python.org> wrote:
On Oct 4, 2017, at 13:53, Benjamin Peterson <benjamin@python.org> wrote:
It might be helpful to enumerate the usecases for such an API. Perhaps a narrow, specialized API could satisfy most needs in a supportable way.
Currently `python -m dis thing.py` compiles the source then disassembles it. It would be kind of cool if you could pass a .pyc file to -m dis, in which case you’d need to unpack the header to get to the code object. A naive implementation would unpack the magic number and refuse to disassemble any files that don’t match whatever that version of Python understands. A more robust (possibly 3rd party) implementation could potentially disassemble a range of magic numbers and formats, and an API to get at the code object and metadata would help.
I was thinking about the bytecode hacking that some debuggers do. This API would help them support multiple versions of Python. They could use the API to discover what pyc format was in use, extract the code object, hack the bytecode and possibly rewrite a new PEP 3147 style pyc file with the debugger bytecodes inserted.
Third party bytecode optimizers could use the API to unpack multiple versions of pyc files, do their optimizations, and rewrite new files with the proper format.
Actually doing that properly also requires keeping track of which opcodes were valid in different versions of the eval loop, so as Guido suggests, such an abstraction layer would make the most sense as a third party project that tracked: - the magic number for each CPython feature release (plus the 3.5.3+ anomaly) - the pyc header format for each CPython feature release - the valid opcode set for each CPython feature release - any other version dependent variations (e.g. the expected stack layout for BUILD_MAP changed in Python 3.5, when the evaluation order for dict displays was updated to be key then value, rather than the other way around) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
26.09.17 23:47, Guido van Rossum пише:
I've read the current version of PEP 552 over and I think everything looks good for acceptance. I believe there are no outstanding objections (or they have been adequately addressed in responses).
Therefore I intend to accept PEP 552 this Friday, unless grave objections are raised on this mailing list (python-dev).
Congratulations Benjamin. Gotta love those tristate options!
While PEP 552 is accepted, I would want to see some changes. 1. Increase the size of the constant part of the signature to at least 32 bits. Currently only the third and forth bytes are constant, and they are '\r\n', that is often occurred in text files. The first two bytes can be different in every Python version. This make hard detecting pyc files by utilities like file (1). 2. Split the "version" of pyc files by "major" and "minor" parts. Every major version is incompatible with other major versions, the interpreter accepts only one particular major version. It can't be changed in a bugfix release. But all minor versions inside the same major version are forward and backward compatible. The interpreter should be able to execute pyc file with arbitrary minor version, but it can use minor version of pyc file to handle errors in older versions. Minor version can be changed in a bugfix release. I hope this can help us with issues like https://bugs.python.org/issue29537. Currently 3.5 supports two magic numbers. If we change the pyc format, it would be easy to make the above changes.
On Oct 3, 2017 9:55 AM, "Serhiy Storchaka" <storchaka@gmail.com> wrote: While PEP 552 is accepted, I would want to see some changes. 1. Increase the size of the constant part of the signature to at least 32 bits. Currently only the third and forth bytes are constant, and they are '\r\n', that is often occurred in text files. The first two bytes can be different in every Python version. This make hard detecting pyc files by utilities like file (1). 2. Split the "version" of pyc files by "major" and "minor" parts. Every major version is incompatible with other major versions, the interpreter accepts only one particular major version. It can't be changed in a bugfix release. But all minor versions inside the same major version are forward and backward compatible. The interpreter should be able to execute pyc file with arbitrary minor version, but it can use minor version of pyc file to handle errors in older versions. Minor version can be changed in a bugfix release. I hope this can help us with issues like https://bugs.python.org/issue29537. Currently 3.5 supports two magic numbers. If we change the pyc format, it would be easy to make the above changes. IIUC the PEP doesn't commit to any particular magic word format, so this can be negotiated separately, on the tracker (unless there's a PEP specifying the internal structure of the magic word?). --Guido
participants (11)
-
Antoine Pitrou
-
Barry Warsaw
-
Benjamin Peterson
-
Brett Cannon
-
Guido van Rossum
-
Guido van Rossum
-
Koos Zevenhoven
-
Louie Lu
-
Nick Coghlan
-
Serhiy Storchaka
-
Victor Stinner