In bpo-29400, it was proposed to add the ability to trace not only
function calls but also instructions at the bytecode level. I like the
idea, but I don't see how to extend sys.settrace() to add a new
"trace_instructions: bool" optional (keyword-only?) parameter without
breaking the backward compatibility. Should we add a new function
Would it be possible to make the API "future-proof", so we can extend
it again later?
I almost never used sys.settrace(), so I prefer to ask on this python-dev list.
Copy of the George King's message:
After reviewing the thread, I'm reminded that the main design problem
concerns preserving behavior of this idiom:
"old=sys.gettrace(); ...; sys.settrace(old)"
If we add more state, i.e. the `trace_instructions` bool, then the
above idiom no longer correctly stores/reloads the full state.
Here are the options that I see:
1. New APIs:
* `gettrace() -> Callable # no change.`
* `settrace(tracefunc: Callable) -> None # no change.`
* `gettraceinst() -> Callable # new.`
* `settraceinst(tracefunc: Callable) -> None # new.`
* `settrace()` raises if `gettraceinst()` is not None.
* `settraceinst()` raises if `gettrace()` is not None.
2. Add keyword arg to `settrace`.
* `gettrace() -> Callable # no change.`
* `settrace(tracefunc: Callable,
trace_instructions:Optional[bool]=False) -> None # added keyword.`
* `gettracestate() -> Dict[str, Any] # new.`
* `gettracestate()` returns the complete trace-related state:
currently, `tracefunc` and `trace_instructions` fields.
* `gettrace()` raises an exception if any of the trace state does not
match the old behavior, i.e. if `trace_instructions` is set.
3. New API, but with extensible keyword args:
* `settracestate(tracefunc: Callable,
trace_instructions:Optional[bool]=False) -> None # new.`
* `gettracestate() -> Dict[str, Any] # new.`
* `settrace()` raises if `gettracestate()` is not None.
* `settracestate()` raises if `gettrace()` is not None.
As I see it:
* approach #1 is more conservative because it does not modify the old API.
* approach #2 is more extensible because all future features can be
represented by the state dictionary.
* approach #3 has both of these strengths, but is also the most work.
Examples of possible future features via keywords:
* branch-level tracing, as briefly disscussed above.
* instruction filtering set, which could be a more general version of
the branch tracing.
I intend to prototype one or both of these, but I'm also feeling a bit
of time pressure to get the basic feature on track for 3.7.
settraceinst() doesn't seem "future-proof" to me.
Python 3 added a __traceback__ attribute to exception objects. I guess
that it was added to be able to get the original traceback when an
exception is re-raised. Artifical example (real code is more complex
with subfunctions and conditional code):
except Exception as exc:
raise exc # keep original traceback
The problem is that Exception.__traceback__ creates reference cycles.
If you store an exception in a local variable, suddently, all local
variables of all frames (of the current traceback) become part of a
giant reference cycle. Sometimes all these variables are kept alive
very long (until you exit Python?), at least, much longer than the
"expected" lifetime (destroyed "magically" local variables when you
exit the function).
The reference cycle is:
1) frame -> local variable -> variable which stores the exception
2) exception -> traceback -> frame
exception -> ... -> frame -> ... -> same exception
Breaking manually the reference cycle is complex. First, you must be
aware of the reference cycle! Second, you have to identify which
functions of your large application create the reference cycle: this
task is long and painful even with good tooling. Finally, you have to
explicitly clear variables or attributes to break the reference cycle
asyncio.Future.set_exception() keeps an exception object and its
traceback object alive: asyncio creates reference cycles *by design*.
Enjoy! asyncio tries hard to reduce the consequence of reference
cycles, or even try to break cycles, using hacks like "self = None" in
methods... Setting self to None is really surprising and requires a
comment explainaing the hack.
Last years, I fixed many reference cycles in various parts of the
Python 3 standard library. Sometimes, it takes years to become aware
of the reference cycle and finally fix it.
For example, recently, I worked on fixing all "dangling threads"
leaked by tests of the Python test suite, and I found and fixed many
reference cycles which probably existed since Python 3 was created
(forked from Python 2):
* socket.create_connection(): commit
* concurrent.futures.ThreadPoolExecutor: commit
* pydoc: commit 4cab2cd0c05fcda5fcb128c9eb230253fff88c21, bpo-31238
* xmlrpc.server: commit 84524454d0ba77d299741c47bd0f5841ac3f66ce, bpo-31247
* test_ssl: commit 868710158910fa38e285ce0e6d50026e1d0b2a8c, bpo-31323
* test_threading: commit 3d284c081fc3042036adfe1bf2ce92c34d743b0b, bpo-31234
Another example of a recent change fixing a reference cycle, by Antoine Pitrou:
* multiprocessing: commit 79d37ae979a65ada0b2ac820279ccc3b1cd41ba6, bpo-30775
For socket.create_connection(), I discovered the reference cycle
because a test started to log a warning about dangling thred. The
warning was introduced indirectly by a change which modified
support.HOST value from '127.0.0.1' to 'localhost'... It's hard to see
to link between support.HOST value and a reference cycle. Full story:
Again, it's just yet another random example of a very tricky reference
cycle bug caused by Exception.__traceback__.
Ideally, CPython 3.x should never create reference cycles. Removing
Exception.__traceback__ is the obvious "fix" for the issue. But I
expect that slowly, a lot of code started to rely on the attribute,
maybe even for good reasons :-)
A more practical solution would be to log a warning. Maybe the garbage
collector can emit a warning if it detects an exception part of a
reference cycle? Or maybe detect frames?
If the GC cannot do it, maybe we might use a debug thread (enabled
manually) which checks manually if an exception is part of a reference
cycle using gc.get_objects(): check if an exception remains alive
longer than X seconds? I had the same idea for asyncio, to detect
reference cycles or if a task is never "awaited", but I never
implemented the idea.
While I can't attend to sprint, I saw etherpad and I found
Neil Schemenauer and Eric Snow will work on startup time.
I want to share my current knowledge about startup time.
For bare (e.g. `python -c pass`) startup time, I'm waiting C
implementation of ABC.
But application startup time is more important. And we can improve
them with optimize importing common stdlib.
Current `python -v` is not useful to optimize import.
So I use this patch to profile import time.
With this profile, I tried optimize `python -c 'import asyncio'`, logging
With this small patch:
logging: 14.9ms -> 12.9ms
asyncio: 62.1ms -> 58.2ms
http.client: 43.8ms -> 36.1ms
I haven't created pull request yet.
(Can I create without issue, as trivial patch?)
I'm very busy these days, maybe until December.
But I hope this report helps people working on optimizing startup time.
INADA Naoki <songofacandy(a)gmail.com>
Last week, the National Security Authority of Slovakia contacted the
Python Security Response Team (PSRT) to report that the Python Package
Index (PyPI) was hosting malicious packages. Installing these packages
send user data to a HTTP server, but also install the expected module
so it was an easy to notice the attack.
Kudos to them to report the issue!
It's not a compromise of the PyPI server nor a third-party project,
but the "typo squatting" issue which is known since at least June 2016
(for PyPI). The issue is not specific to Python, npmjs.com or
rubygems.org are vulnerable to the same issue.
For example, a malicious package used the names "urllib" (no 3) and
"urlib3" (1 L) instead of "urllib3" (2 L). These packages were
downloaded by users, so the attack was effective.
More information on typo squatting and Python package security:
The PRST contacted PyPI administrators and all identified packages
were taken down, only 1h10 after the PSRT received the email from the
National Security Authority of Slovakia!
The typo squatting issue is known and discussed, but not solution was
found yet. See for example this warehouse issue:
It seems like the consensus is that pip is not responsible to detect
malicious code, it's more the responsability of PyPI.
The problem is to decide how to detect malicious code and/or prevent
typo squatting on PyPI.
The issue has been discussed privately on the PSRT list last week. The
National Security Authority of Slovakia just published their advisory,
and a public discussion started on reddit:
I consider that it's now time to find a solution on the public
python-dev mailing list.
Let's try to find a solution!
Can we learn something from the Update Framework (TUF)?
with these security issues on their package manager?
See also my other notes on Python security and the list of known
I've written a short PEP about an import extension to allow pycs to be
more deterministic by optional replacing the timestamp with a hash of
the source file: https://www.python.org/dev/peps/pep-0552/
Thanks for reading,
P.S. I came up with the idea for this PEP while awake.
I've written a PEP proposing a language change:
The TL;DR summary: add support for property objects to modules. I've
already posted a prototype.
How's that sound?
The docs have this rule for slot initialization for the benefit of Cygwin:
- PyType_GenericNew, /* tp_new */
+ noddy_NoddyType.tp_new = PyType_GenericNew;
+ if (PyType_Ready(&noddy_NoddyType) < 0)
This is absolutely not required by C99 (and probably never was).
'PyType_GenericNew' is an address constant, and MSVC supports it just
fine -- at least since VS 2008.
Does anyone know if Cygwin still misbehaves? I would like to get rid
of this arcane rule.