On Sat, May 8, 2021 at 2:09 PM Pablo Galindo Salgado <pablogsal@gmail.com> wrote:

> Why not put in it -O instead?  Then -O means lose asserts and lose fine-grained tracebacks, while -OO continues to also

strip out doc strings.

What if someone wants to keep asserts but do not want the extra data?

exactly my theme.  our existing -O and -OO already don't serve all user needs.  (I've witnessed people who need asserts but don't want docstrings wasting ram jump through hacky hoops to do that).  Complicating these options more by combining additional actions on them them doesn't help.

The reason we have -O and -OO generate their own special opt-1 and opt-2 pyc files is because both of those change the generated bytecode and overall flow of the program by omitting instructions and data.  code using those will run slightly faster as there are fewer instructions.

The change we're talking about here doesn't do that.  It just adds additional metadata to whatever instructions are generated.  So it doesn't feel -O related.

While some people aren't going to like the overhead, I'm happy not offering the choice.

> Greg, what do you think if instead of not writing it to the pyc file with -OO or adding a header entry to decide to read/write, we place None in the field? That way we can

> leverage the option that we intend to add to deactivate displaying the traceback new information to reduce the data in the pyc files. The only problem
> is that there will be still a tiny bit of overhead: an extra object per code object (None), but that's much much better than something that scales with the
> number of instructions :)
> 
> What's your opinion on this?

I don't understand the pyc structure enough to comment on how that works,

but that sounds fine from a way to store less data if these are stored as a side table rather than intermingled with each instruction itself.  If anyone even cares about storing less data.  I would not activate generation of that in py_compile and compileall based on the -X flag to disable display of tracebacks though.  A flag changing a setting of the current runtime regarding traceback printing detail level should not change the metadata in pyc files it emits.  I realize -O and -OO behave this way, but I don't view those as a great example. We're not writing new uniquely named pyc files, I suggest making this an explicit option for py_compile and compileall if we're going to support generation of pyc files without column data at all.

I'm unclear on what the specific goals are with all of these option possibilities.

Who non-hypothetically cares about a 22% pyc file size increase?  I don't think we should be concerned.  I'm in favor of always writing them and the 20% size increase that results in.  If pyc size is an issue that should be its own separate enhancement PEP.  When it comes to pyc files there is more data we may want to store in the future for performance reasons - I don't see them shrinking without an independent effort.

Caring about additional data retained in memory at runtime makes more sense to me as ram cost is much greater than storage cost and is paid repeatedly per process.  Storing an additional reference to None on code objects where a column information table is perfectly fine.  That can be a -X style interpreter startup option.  It isn't something that needs to impacted by the pyc files.  Pass that option to the interpreter, and it just discards column info tables on code objects after loading them or compiling them.  If people want to optimize for a shared pyc situation with memory mapping techniques, that is also something that should be a separate enhancement PEP and not involved here.  People writing code to use the column information should always check it for None first, that'd be something we document with the new feature.

-gps

 

On Sat, 8 May 2021 at 22:05, Ethan Furman <ethan@stoneleaf.us> wrote:

On 5/8/21 1:31 PM, Pablo Galindo Salgado wrote:
 >> We can't piggy back on -OO as the only way to disable this, it needs to
 >> have an option of its own.  -OO is unusable as code that relies on "doc"
 >> strings as application data such as http://www.dabeaz.com/ply/ply.html
 >> exists.
 >
 > -OO is the only sensible way to disable the data. There are two things to disable:
 >
 > * The data in pyc files
 > * Printing the exception highlighting

Why not put in it -O instead?  Then -O means lose asserts and lose fine-grained tracebacks, while -OO continues to also
strip out doc strings.

--
~Ethan~
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/BEE4BGUZHXBTVDPOW5R4DC3S463XC3EJ/
Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/WK4KXZPOSWYMI3C5AILQCEYVZRCDFL7N/
Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/Q5RCT3VMQFPZTJJKB666PML4K2KZ2HJW/
Code of Conduct: http://python.org/psf/codeofconduct/