I proposed https://bugs.python.org/issue39352 to remove the formatter
module. It's deprecated since Python 3.4. The main reason why it's
still around is the PEP 4 rule:
"In order to facilitate writing code that works in both Python 2 & 3
simultaneously, any module that exists in both Python 3.5 and Python
2.7 will not be removed from the standard library until Python 2.7 is
no longer supported as specified by PEP 373."
Python 2.7 is not longer supported. So can we now remove the formatter module?
The module has no test, I failed to find any user in a dummy GitHub
code search and I didn't know that this module even existed.
We can still revert the module before 3.9.0 final release if someone
shows up and asks to keep it for one more cycle.
My intent here is to reduce the size of the Python standard library to
reduce the maintenance burden. Python became quite large and it's more
and more expensive to maintain it. Wait... I'm not asking to remove
all modules of the standard library :-) The decision must be taken on
a case by case basis, for each module. Here I'm only taking about the
Night gathers, and now my watch begins. It shall not end until my death.
I'm looking at CPython's behavior when an addition is called. From what
I understand, binary_op1
eventually called, and it calls either slotv or slotw, which seems to be
the binaryfunc defined as nb_add in the field tp_as_number of
respectively v / w.
I have a few questions:
1) In the default case, tp_as_number->nb_add is defined by the function
itself stemming from the macro expansion SLOT1BINFULL
defined in typeobject.c. Both binary_op1(v, w) and slot_nb_add(v, w)
appear to perform similar checks (if their second argument is a subtype
of the first, etc), to decide if v's add or w's reverse add must be
called and in which order. I find this repetition weird, and I guess I'm
missing something... Any ideas?
2) From the SLOT1BINFULL macro, both __add__ and __radd__ are defined by
the slot_nb_add function (with some argument swapping done by
wrap_binaryfunc_l / wrap_binaryfunc_r). If I want to define a different
behavior for the reverse operator during a definition with a
PyTypeObject, I guess I should add an "__radd__" method?
3) If I create a user-defined class A, having different methods __add__
and __radd__, these methods are added in A's dictionary. From what I
understand, the function update_one_slot
is then called to change A's tp_as_number->nb_add to point to the
methods defined by __add__ and __radd__? From the code documentation, I
think that "a wrapper for the special methods is installed". Where
exactly is this wrapper applied, and how does it know when to dispatch
to __add__ or __radd__?
I think that PEP 573 is ready to be accepted, to greatly improve the state
of extension modules in CPython 3.9.
It has come a long way since the original proposal and went through several
iterations and discussions by various interested people, effectively
reducing its scope quite a bit. So this is the last call for comments on
the latest version of the PEP, before I will pronounce on it. Please keep
the discussion in this thread.
On windows i want to play with the grammar file, but according to this
> For Windows, there is no officially supported way of running pgen.
However, you can clone my fork
<https://github.com/tonybaloney/cpython/tree/pcbuildregen> and run
build.bat --regen from within the PCBuild directory.
But i don't want to work from his fork. Is there an official way of
the files? I just want to have my own keywords. As far as i've understood,
rebuilding won't do it.
I'm experimenting with package development on different versions of
Python in different virtualenvs. After running "make" I don't do "make
install", but rather I set up virtualenvs by running
/path/to/source/python -m venv env_dir. This works for as long as I
don't need to compile extensions. Once I do that I'm running into
trouble because there is no python3-config binary in the venv, so it
uses the "system" python3-config which of course returns results for the
This seems to go against the idea of an encapsulated and self-contained
environment. And the venv created straight from the "not-installed"
source tree works so well that having a venv/bin/python3-config whose
output points into the source tree seems a logical step. Is this an
omission or is there a rationale for not doing it?
Of course I can "properly" install different Python versions by using
different "configure --prefix" directories. But I like the venv so much
in general that this rubs me the wrong way.
I started to modify Python internals to pass explicitly the Python
thread state ("tstate") to internal C a functions:
Until subinterpreters will be fully implemented, it's unclear if
getting the current Python thread state using _PyThreadState_GET()
macro (which uses an atomic read) will remain efficient. For example,
right now, the "GIL state" API doesn't support subinterpreters: fixing
this may require to add a lock somewhere which may make
_PyThreadState_GET() less efficient. Sorry, I don't have numbers,
since I didn't experiment to implement these changes yet: I was
blocked by other issues. We can only guess at this point.
To me, it sounds like a good practice to pass the current Python
thread state to internal C functions. It seems like most core
developers agreed with that in my previous python-dev thread "Pass the
Python thread state to internal C functions":
The question is now if we should "propagate" tstate to function calls
in the latest VECTORCALL calling convention (which is currently
private). Petr Viktorin plans to make VECTORCALL APIs public in Python
3.9, as planned in the PEP 590:
I added explicitly Stefan Behnel in copy, since Cython should be
directly impacted by such change. Cython is the kind of project which
may benefit of having tstate accessible directly.
I started to move more and more things from "globals" to "per
interpreter". For example, the garbage collector is now "per
interpreter" (lives in PyThreadState). Small integer singletons are
now also "per singleton": int("1") are now different objects in each
interpreter, whereas they were shared previously. Later, even "None"
singleton (and all other singletons) should be made "per interpreter".
Getting a "per interpreter" object requires to state from the Python
thread state: call _PyThreadState_GET(). Avoiding _PyThreadState_GET()
calls reduces any risk of making Python slower with incoming
For the long term, the goal is that each subinterpreter has its own
isolated world: no Python object should be shared, no state should be
shared. The intent is to avoid any need of locking, to maximize
performance when running interpreters in parallel. Obviously, each
interpreter will have its own private GIL ;-) Py_INCREF() and
Py_DECREF() would require slow atomic operations if Python objects are
shared. If objects are not shared between interpreters, Py_INCREF()
and Py_DECREF() can remain as fast as they are today. Any additional
locking may kill performance.
Night gathers, and now my watch begins. It shall not end until my death.
Recently I've been facing a really weird bug where a Python program
was randomly segfaulting during the finalization, the program was
using some C extensions via Cython.
Debugging the issue I realized that during the deallocation of one of
the Python objects the deallocation function was trying to release a
pointer that was surprisingly assigned to NULL. The pointer was at
the same time held by another Python object that was an attribute of
the Python object that had the deallocation function, something like
my_type * value
self._foo = Foo()
self._foo->value = initialize()
Seems that randomly the instance of the object Foo held by the Bar
object was deallocated by the CPython interpreter before the Foo
deallocation, so after being deallocated - and zeroing the memory
space of the instance of Foo - the execution of the
`destroy(self._foo->value)` was in fact given as a parameter a NULL
address and raising a segfault.
It was a surprise for me, If I'm not missing something the
deallocation of the Foo instance happened even though there was still
an active reference held by the Bar object.
As a kind of double-checking I changed the program for making an
explicit `gc.collect()` before the last line of the Python program. As
a result, I couldn't reproduce the segfault, which theoretically would
mean that objects were deallocated "in order".
So my question would be, could CPython deallocate the objects during
the finalization step without considering the dependencies between
If this is not the right list to make this kind of questions, just let
me know what would be the best place for making this kind of questions
Thanks in advance,