Hi everyone,
Thanks again for all your comments on PEP 611.
I would like to ask a favour; please be more specific in your comments.
Ideally state which part of the PEP you are disagreeing with and why you
disagree with the relevant part of the rationale/motivation.
Also, when asking for limits to be raised or removed entirely, could you
state what you perceive to be the costs and benefits of larger limits.
What do you believe is an acceptable cost in memory or runtime for
larger limits?
For example, you might say that the limit of one million lines of code
per module is too small, and that it is worth a small, say 1%, impact on
speed to allow a larger of limit of 100 million.
If you believe a limit would have no cost, then please give a
explanation of why that is so.
Merely saying that you would like a larger limit is pointless.
If there were no cost to arbitrarily large limits, then I wouldn't have
proposed the PEP in the first place.
Bear in mind that the costs of higher limits are paid by everyone, but
the benefits are gained by few.
Cheers,
Mark.
Hi Everyone,
Thanks for all your feedback on my proposed PEP. I've editing the PEP in
light of all your comments and it is now hopefully more precise and with
better justification.
https://github.com/python/peps/pull/1249
Cheers,
Mark.
I want to question two specific limits.
(a) Limiting the number of classes, in order to potentially save space in
object headers, sounds like a big C API change, and I think it's better to
lift this idea out of PEP 611 and debate the and cons separately.
(b) Why limit coroutines? It's just another Python object and has no
operating resources associated with it. Perhaps your definition of
coroutine is different, and you are thinking of OS threads?
--
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-…>
Hi guys,
during the last few weeks I have been struggling quite much
in order to make PySide run with Python 3.8 at all.
The expected problems were refcounting leaks due to changed
handling of heaptypes. But in fact, the runtime behavior was
much worse, because I always got negative refcounts!
After exhaustive searching through the different 3.8 commits, I could
isolate the three problems with logarithmic search.
The hard problem was this:
Whenever PySide creates a new type, it crashes in PyType_Ready.
The reason is the existence of the Py_TPFLAGS_METHOD_DESCRIPTOR
flag.
During the PyType_Ready call, the function mro() is called.
This mro() call results in a negative refcount, because something
behaves differently since this flag is set by default in mro().
When I patched this flag away during the type_new call, everything
worked ok. I don't understand why this problem affects PySide
at all. Here is the code that would normally be only the newType line:
// PYSIDE-939: This is a temporary patch that circumvents the problem
// with Py_TPFLAGS_METHOD_DESCRIPTOR until this is finally solved.
PyObject *ob_PyType_Type = reinterpret_cast<PyObject *>(&PyType_Type);
PyObject *mro = PyObject_GetAttr(ob_PyType_Type,
Shiboken::PyName::mro());
auto hold = Py_TYPE(mro)->tp_flags;
Py_TYPE(mro)->tp_flags &= ~Py_TPFLAGS_METHOD_DESCRIPTOR;
auto *newType = reinterpret_cast<SbkObjectType *>(type_new(metatype,
args, kwds));
Py_TYPE(mro)->tp_flags = hold;
I would really like to understand the reason for this unexpected effect.
Does this ring a bell? I have no clue what is wrong with PySide, if it
is wrong at all.
Thanks -- Chris
--
Christian Tismer :^) tismer(a)stackless.com
Software Consulting : http://www.stackless.com/
Karl-Liebknecht-Str. 121 : https://github.com/PySide
14482 Potsdam : GPG key -> 0xFB7BEE0E
phone +49 173 24 18 776 fax +49 (30) 700143-0023
The Python Steering Council discussed PEP 611 at today’s meeting. Here is our feedback so far:
* The Steering Council reserves the right to be the BDFL-Delegate for this PEP
* The PEP should clearly delineate two aspects:
- A language generic part (i.e. applies to all implementations of Python) which exposes implementation limits in a way that Python code can read at runtime. Think sys.getrecursionlimit(), sys.maxsize, and sys.implementation
- An implementation-specific part where the PEP would propose specific limits for the CPython implementation. Other implementations of the Python language would be free to adjust such limits up or down as they deem appropriate.
* We encourage the PEP authors and proponents to gather actual performance data that can be used to help us evaluate whether this PEP is a good idea or not.
By all means, continue to discuss and refine the PEP. When the PEP author is ready for a pronouncement, you can email the Steering Council.
Cheers,
-Barry
as an aside in the portion of PEP 611 proposing a limit to the number of classes, the proposal also mentioned reducing the size of the reference count field to achieve part of its object size savings.
one of the mechanisms proposed was to implement a "saturating" reference count, though it would be different from conventional saturating arithmetic in that any object that reaches the maximum refcount would become immortal, since we could not trust decref from there to account for all references.
something that may be worth considering or testing is: what would the cost, both in performance and interpreter code size, of implementing this logic on every single incref and decref?
Hi Everyone,
I am proposing a new PEP, still in draft form, to impose a limit of one
million on various aspects of Python programs, such as the lines of code
per module.
Any thoughts or feedback?
The PEP:
https://github.com/markshannon/peps/blob/one-million/pep-1000000.rst
Cheers,
Mark.
Full text
*********
PEP: 1000000
Title: The one million limit
Author: Mark Shannon <mark(a)hotpy.org>
Status: Active
Type: Enhancement
Content-Type: text/x-rst
Created: 03-Dec-2019
Post-History:
Abstract
========
This PR proposes a limit of one million (1 000 000) for various aspects
of Python code and its implementation.
The Python language does not specify limits for many of its features.
Not having any limit to these values seems to enhance programmer freedom,
at least superficially, but in practice the CPython VM and other Python
virtual
machines have implicit limits or are forced to assume that the limits are
astronomical, which is expensive.
This PR lists a number of features which are to have a limit of one
million.
If a language feature is not listed but appears unlimited and must be
finite,
for physical reasons if no other, then a limit of one million should be
assumed.
Motivation
==========
There are many values that need to be represented in a virtual machine.
If no limit is specified for these values, then the representation must
either be inefficient or vulnerable to overflow.
The CPython virtual machine represents values like line numbers,
stack offsets and instruction offsets by 32 bit values. This is
inefficient, and potentially unsafe.
It is inefficient as actual values rarely need more than a dozen or so
bits to represent them.
It is unsafe as malicious or poorly generated code could cause values to
exceed 2\ :sup:`32`.
For example, line numbers are represented by 32 bit values internally.
This is inefficient, given that modules almost never exceed a few
thousand lines.
Despite being inefficent, is is still vulnerable to overflow as
it is easy for an attacker to created a module with billions of newline
characters.
Memory access is usually a limiting factor in the performance of modern
CPUs.
Better packing of data structures enhances locality and reduces memory
bandwith,
at a modest increase in ALU usage (for shifting and masking).
Being able to safely store important values in 20 bits would allow
memory savings
in several data structures including, but not limited to:
* Frame objects
* Object headers
* Code objects
There is also the potential for a more efficient instruction format,
speeding up interpreter dispatch.
Rationale
=========
Imposing a limit on values such as lines of code in a module, and the
number of local variables,
has significant advantages for ease of implementation and efficiency of
virtual machines.
If the limit is sufficiently large, there is no adverse effect on users
of the language.
By selecting a fixed but large limit for these values,
it is possible to have both safety and efficiency whilst causing no
inconvience to human programmers
and only very rare problems for code generators.
One million
-----------
The Java Virtual Machine (JVM) [1]_ specifies a limit of 2\ :sup:`16`-1
(65535) for many program
elements similar to those covered here.
This limit enables limited values to fit in 16 bits, which is a very
efficient machine representation.
However, this limit is quite easily exceeded in practice by code
generators and
the author is aware of existing Python code that already exceeds 2\
:sup:`16` lines of code.
A limit of one million fits into 20 bits which, although not as
convenient for machine representation,
is still reasonably compact. Three signed valuses in the range -1000_000
to +1000_000 can fit into a 64 bit word.
A limit of one million is small enough for efficiency advantages (only
20 bits),
but large enough not to impact users (no one has ever written a module
of one million lines).
The value "one million" is very easy to remember.
Isn't this "640K ought to be enough for anybody" again?
-------------------------------------------------------
The infamous 640K memory limit was a limit on machine usable resources.
The proposed one million limit is a limit on human generated code.
While it is possible that generated code could exceed the limit,
it is easy for a code generator to modify its output to conform.
The author has hit the 64K limit in the JVM on at least two occasions
when generating Java code.
The workarounds were relatively straightforward and
probably wouldn't have been necessary with a limit of one million
bytecodes or lines of code.
Specification
=============
This PR proposes that the following language features and runtime values
be limited to one million.
* The number of source code lines in a module
* The number of bytecode instructions in a code object.
* The sum of local variables and stack usage for a code object.
* The number of distinct names in a code object
* The number of constants in a code object.
* The number of classes in a running interpreter.
* The number of live coroutines in a running interpreter.
The advantages for CPython of imposing these limits:
----------------------------------------------------
Line of code in a module and code object restrictions.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
When compiling source code to bytecode or modifying bytecode for
profiling or debugging,
an intermediate form is required. The limiting line numbers and operands
to 20 bits,
instructions can be represented in a compact 64 bit form allowing
very fast passes over the instruction sequence.
Having 20 bit operands (21 bits for relative branches) allows instructions
to fit into 32 bits without needing additional ``EXTENDED_ARG``
instructions.
This improves dispatch, as the operand is strictly local to the instruction.
Using super-instructions would make that the 32 bit format
almost as compact as the 16 bit format, and significantly faster.
Total number of classes in a running interpreter
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This limit has to the potential to reduce the size of object headers
considerably.
Currently objects have a two word header, for objects without references
(int, float, str, etc.) or a four word header for objects with references.
By reducing the maximum number of classes, the space for the class reference
can be reduced from 64 bits to fewer than 32 bits allowing a much more
compact header.
For example, a super-compact header format might look like this:
.. code-block::
struct header {
uint32_t gc_flags:6; /* Needs finalisation, might be part of a
cycle, etc. */
uint32_t class_id:26; /* Can be efficiently mapped to address
by ensuring suitable alignment of classes */
uint32_t refcount; /* Limited memory or saturating */
}
This format would reduce the size of a Python object without slots, on a
64 bit machine, from 40 to 16 bytes.
Note that there are two ways to use a 32 bit refcount on a 64 bit machine.
One is to limit each sub-interpreter to 32Gb of memory.
The other is to use a saturating reference count, which would be a
little bit slower, but allow unlimited memory allocation.
Backwards Compatibility
=======================
It is hypothetically possible that some machine generated code exceeds
one or more of the above limits.
The author believes that to be highly unlikely and easily fixed by
modifying the output stage of the code generator.
Security Implications
=====================
Minimal. This reduces the attack surface of any Python virtual machine
by a small amount.
Reference Implementation
========================
None, as yet. This will be implemented in CPython, once the PEP has been
accepted.
Rejected Ideas
==============
None, as yet.
Open Issues
===========
None, as yet.
References
==========
.. [1] The Java Virtual Machine specification
https://docs.oracle.com/javase/specs/jvms/se8/jvms8.pdf
Copyright
=========
This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.
..
Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
coding: utf-8
End:
Python 3.8.1rc1 is the release candidate of the first maintenance release of Python 3.8.
The Python 3.8 series is the newest feature release of the Python language, and it contains many new features and optimizations. You can find Python 3.8.1rc1 here:
https://www.python.org/downloads/release/python-381rc1/ <https://www.python.org/downloads/release/python-381rc1/>
Assuming no critical problems are found prior to 2019-12-16, the scheduled release date for 3.8.1 as well as Ned Deily's birthday, no code changes are planned between this release candidate and the final release.
That being said, please keep in mind that this is a pre-release of 3.8.1 and as such its main purpose is testing.
See the “What’s New in Python 3.8 <https://docs.python.org/3.8/whatsnew/3.8.html>” document for more information about features included in the 3.8 series. Detailed information about all changes made in 3.8.0 can be found in its change log.
Maintenance releases for the 3.8 series will continue at regular bi-monthly intervals, with 3.8.2 planned for February 2020.
We hope you enjoy Python 3.8!
Thanks to all of the many volunteers who help make Python Development and these releases possible! Please consider supporting our efforts by volunteering yourself or through organization contributions to the Python Software Foundation.
https://www.python.org/psf/ <https://www.python.org/psf/>
Hi everyone,
Thanks for all your feedback so far.
Previously I asked for "more precise" feedback, which has been
interpreted as evidence backed feedback. That's not what I meant.
My fault for not being clearer.
Your opinions without any justifications are welcome, but I need precision.
For example, saying "I don't want any limits ever for anything" is
precise, but saying "A limit of 1 million is OK provided the performance
improvements justify it" is not.
"A limit of 1 million is OK provided a speed up of 50% can be shown" is
precise, if a bit of a challenge :)
To start this off, here are my opinion of the tradeoff:
Almost any performance gain, even 0.1% is worth the, IMO, slight
inconvenience of 1 million limits.
The reason I believe this is that a 0.1% speedup benefits all Python
applications and libraries everywhere and forever, whereas the
inconvenience will be felt by a handful of developers, very rarely.
Another thing I would like feedback on this:
My justification for a single limit of one million across the board is
to ease memorization and learning.
Is that sufficient justification, or would differing limits be better?
Thanks once again,
Mark.
P.S. On the subject of tradeoffs, here's a bonus question:
What, in your opinion, increase in memory consumption is acceptable for
a 1% improvement in speed, or vice versa?
