Mailman 3 September 2017 - Python-Dev

Re: [Python-Dev] API design question: how to extend sys.settrace()?
by George King 27 Sep '17

27 Sep '17

Victor, thank you for starting this conversation. Nick, I just looked at your patch and I think it is a better solution than mine, because it does not involve adding to or changing the sys API. I will close my pull request. The reason for my interest in this area is that I’m experimenting with a code coverage tool that does per-opcode tracing. I just updated it to use the new f_trace_opcodes feature and it *almost* worked: Nick’s implementation calls the opcode trace before updating the line number, whereas my version updated line numbers first, so the event stream is slightly different. See: https://github.com/python/cpython/commit/5a8516701f5140c8c989c40e261a4f4e20… From my perspective it makes more sense to do the update to f_lineno first, followed by the opcode trace, because then line events and opcode events corresponding to the same opcode will have the same line number; as it is they come out different. Is the current order intentional? Otherwise I’ll submit a patch. Either way, I’m really happy that this functionality has made it into master! Thanks, George

2 1

New security-announce@python.org mailing list
by Barry Warsaw 27 Sep '17

27 Sep '17

I’m happy to announce the availability of a new mailing list, with the mission of providing security announcements to the Python community from the Python Security Response Team (PSRT): security-announce(a)python.org You can sign up in the usual Mailman way: https://mail.python.org/mailman/listinfo/security-announce This joins our suite of security related forums. As always, if you believe you’ve found a security issue in Python, you should contact the PSRT directly and securely via: security(a)python.org For more information on how you can contact us, see: https://www.python.org/news/security/ We also have a public security-focused discussion mailing list that you can subscribe and contribute to. security-sig(a)python.org https://mail.python.org/mailman/listinfo/security-sig Please don’t report security vulnerabilities here, since this is a publicly archived mailing list. We welcome you to collaborate here to help make Python and its ecosystem even more secure than it already is. Once a security vulnerability is identified and fixed, it becomes public knowledge. Generally, these are captured in a ReadTheDocs site for posterity: https://python-security.readthedocs.io/ This new security-announce mailing list fills a void — one-way communication about security related matters from the PSRT back to the community. This is an area that we’ve not done a great job at, frankly, and this new announcement list is intended to improve that situation. The PSRT will use this low traffic, high value forum as the primary way the PSRT will communicate security issues of high importance back to the wider Python community. All follow-ups to postings to this list are redirected to the security-sig mailing list. Cheers, -Barry (on behalf of the PSRT)

4 5

API design question: how to extend sys.settrace()?
by Victor Stinner 27 Sep '17

27 Sep '17

Hi, In bpo-29400, it was proposed to add the ability to trace not only function calls but also instructions at the bytecode level. I like the idea, but I don't see how to extend sys.settrace() to add a new "trace_instructions: bool" optional (keyword-only?) parameter without breaking the backward compatibility. Should we add a new function instead? Would it be possible to make the API "future-proof", so we can extend it again later? I almost never used sys.settrace(), so I prefer to ask on this python-dev list. Copy of the George King's message: https://bugs.python.org/issue29400#msg298441 """ After reviewing the thread, I'm reminded that the main design problem concerns preserving behavior of this idiom: "old=sys.gettrace(); ...; sys.settrace(old)" If we add more state, i.e. the `trace_instructions` bool, then the above idiom no longer correctly stores/reloads the full state. Here are the options that I see: 1. New APIs: * `gettrace() -> Callable # no change.` * `settrace(tracefunc: Callable) -> None # no change.` * `gettraceinst() -> Callable # new.` * `settraceinst(tracefunc: Callable) -> None # new.` Behavior: * `settrace()` raises if `gettraceinst()` is not None. * `settraceinst()` raises if `gettrace()` is not None. 2. Add keyword arg to `settrace`. * `gettrace() -> Callable # no change.` * `settrace(tracefunc: Callable, trace_instructions:Optional[bool]=False) -> None # added keyword.` * `gettracestate() -> Dict[str, Any] # new.` Behavior: * `gettracestate()` returns the complete trace-related state: currently, `tracefunc` and `trace_instructions` fields. * `gettrace()` raises an exception if any of the trace state does not match the old behavior, i.e. if `trace_instructions` is set. 3. New API, but with extensible keyword args: * `settracestate(tracefunc: Callable, trace_instructions:Optional[bool]=False) -> None # new.` * `gettracestate() -> Dict[str, Any] # new.` Behavior: * `settrace()` raises if `gettracestate()` is not None. * `settracestate()` raises if `gettrace()` is not None. As I see it: * approach #1 is more conservative because it does not modify the old API. * approach #2 is more extensible because all future features can be represented by the state dictionary. * approach #3 has both of these strengths, but is also the most work. Examples of possible future features via keywords: * branch-level tracing, as briefly disscussed above. * instruction filtering set, which could be a more general version of the branch tracing. I intend to prototype one or both of these, but I'm also feeling a bit of time pressure to get the basic feature on track for 3.7. """ settraceinst() doesn't seem "future-proof" to me. Victor

5 4

Evil reference cycles caused Exception.__traceback__
by Victor Stinner 25 Sep '17

25 Sep '17

Hi, Python 3 added a __traceback__ attribute to exception objects. I guess that it was added to be able to get the original traceback when an exception is re-raised. Artifical example (real code is more complex with subfunctions and conditional code): try: ... except Exception as exc: ... raise exc # keep original traceback The problem is that Exception.__traceback__ creates reference cycles. If you store an exception in a local variable, suddently, all local variables of all frames (of the current traceback) become part of a giant reference cycle. Sometimes all these variables are kept alive very long (until you exit Python?), at least, much longer than the "expected" lifetime (destroyed "magically" local variables when you exit the function). The reference cycle is: 1) frame -> local variable -> variable which stores the exception 2) exception -> traceback -> frame exception -> ... -> frame -> ... -> same exception Breaking manually the reference cycle is complex. First, you must be aware of the reference cycle! Second, you have to identify which functions of your large application create the reference cycle: this task is long and painful even with good tooling. Finally, you have to explicitly clear variables or attributes to break the reference cycle manually. asyncio.Future.set_exception() keeps an exception object and its traceback object alive: asyncio creates reference cycles *by design*. Enjoy! asyncio tries hard to reduce the consequence of reference cycles, or even try to break cycles, using hacks like "self = None" in methods... Setting self to None is really surprising and requires a comment explainaing the hack. Last years, I fixed many reference cycles in various parts of the Python 3 standard library. Sometimes, it takes years to become aware of the reference cycle and finally fix it. For example, recently, I worked on fixing all "dangling threads" leaked by tests of the Python test suite, and I found and fixed many reference cycles which probably existed since Python 3 was created (forked from Python 2): * socket.create_connection(): commit acb9fa79fa6453c2bbe3ccfc9cad2837feb90093, bpo-31234 * concurrent.futures.ThreadPoolExecutor: commit bc61315377056fe362b744d9c44e17cd3178ce54, bpo-31249 * pydoc: commit 4cab2cd0c05fcda5fcb128c9eb230253fff88c21, bpo-31238 * xmlrpc.server: commit 84524454d0ba77d299741c47bd0f5841ac3f66ce, bpo-31247 Other examples: * test_ssl: commit 868710158910fa38e285ce0e6d50026e1d0b2a8c, bpo-31323 * test_threading: commit 3d284c081fc3042036adfe1bf2ce92c34d743b0b, bpo-31234 Another example of a recent change fixing a reference cycle, by Antoine Pitrou: * multiprocessing: commit 79d37ae979a65ada0b2ac820279ccc3b1cd41ba6, bpo-30775 For socket.create_connection(), I discovered the reference cycle because a test started to log a warning about dangling thred. The warning was introduced indirectly by a change which modified support.HOST value from '127.0.0.1' to 'localhost'... It's hard to see to link between support.HOST value and a reference cycle. Full story: https://bugs.python.org/issue29639#msg302087 Again, it's just yet another random example of a very tricky reference cycle bug caused by Exception.__traceback__. Ideally, CPython 3.x should never create reference cycles. Removing Exception.__traceback__ is the obvious "fix" for the issue. But I expect that slowly, a lot of code started to rely on the attribute, maybe even for good reasons :-) A more practical solution would be to log a warning. Maybe the garbage collector can emit a warning if it detects an exception part of a reference cycle? Or maybe detect frames? If the GC cannot do it, maybe we might use a debug thread (enabled manually) which checks manually if an exception is part of a reference cycle using gc.get_objects(): check if an exception remains alive longer than X seconds? I had the same idea for asyncio, to detect reference cycles or if a task is never "awaited", but I never implemented the idea. Victor

9 19

To reduce Python "application" startup time
by INADA Naoki 23 Sep '17

23 Sep '17

Hi, While I can't attend to sprint, I saw etherpad and I found Neil Schemenauer and Eric Snow will work on startup time. I want to share my current knowledge about startup time. For bare (e.g. `python -c pass`) startup time, I'm waiting C implementation of ABC. But application startup time is more important. And we can improve them with optimize importing common stdlib. Current `python -v` is not useful to optimize import. So I use this patch to profile import time. https://gist.github.com/methane/e688bb31a23bcc437defcea4b815b1eb With this profile, I tried optimize `python -c 'import asyncio'`, logging and http.client. https://gist.github.com/methane/1ab97181e74a33592314c7619bf34233#file-0-opt… With this small patch: logging: 14.9ms -> 12.9ms asyncio: 62.1ms -> 58.2ms http.client: 43.8ms -> 36.1ms I haven't created pull request yet. (Can I create without issue, as trivial patch?) I'm very busy these days, maybe until December. But I hope this report helps people working on optimizing startup time. Regards, INADA Naoki <songofacandy(a)gmail.com>

12 16

Summary of Python tracker Issues
by Python tracker 22 Sep '17

22 Sep '17

ACTIVITY SUMMARY (2017-09-15 - 2017-09-22) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 6196 (+25) closed 37107 (+42) total 43303 (+67) Open issues with patches: 2376 Issues opened (56) ================== #13487: inspect.getmodule fails when module imports change sys.modules https://bugs.python.org/issue13487 reopened by merwok #31486: calling a _json.Encoder object raises a SystemError in case ob https://bugs.python.org/issue31486 opened by Oren Milman #31488: IDLE: Update feature classes when options are changed. https://bugs.python.org/issue31488 opened by terry.reedy #31489: Signal delivered to a subprocess triggers parent's handler https://bugs.python.org/issue31489 opened by Ilya.Kulakov #31490: assertion failure in ctypes in case an _anonymous_ attr appear https://bugs.python.org/issue31490 opened by Oren Milman #31491: Add is_closing() to asyncio.StreamWriter. https://bugs.python.org/issue31491 opened by aymeric.augustin #31492: assertion failures in case a module has a bad __name__ attribu https://bugs.python.org/issue31492 opened by Oren Milman #31493: IDLE cond context: fix code update and font update timers https://bugs.python.org/issue31493 opened by terry.reedy #31494: Valgrind suppression file https://bugs.python.org/issue31494 opened by Aaron Michaux #31495: Wrong offset with IndentationError ("expected an indented bloc https://bugs.python.org/issue31495 opened by blueyed #31496: IDLE: test_configdialog failed https://bugs.python.org/issue31496 opened by serhiy.storchaka #31500: IDLE: Tiny font on HiDPI display https://bugs.python.org/issue31500 opened by serhiy.storchaka #31502: IDLE: Config dialog again deletes custom themes and keysets. https://bugs.python.org/issue31502 opened by terry.reedy #31503: Enhance dir(module) to be informed by __all__ by updating modu https://bugs.python.org/issue31503 opened by codypiersall #31504: Documentation for return value for string.rindex is missing wh https://bugs.python.org/issue31504 opened by kgashok #31505: assertion failure in json, in case _json.make_encoder() receiv https://bugs.python.org/issue31505 opened by Oren Milman #31506: Improve the error message logic for object_new & object_init https://bugs.python.org/issue31506 opened by ncoghlan #31507: email.utils.parseaddr has no docstring https://bugs.python.org/issue31507 opened by mark.dickinson #31508: Running test_ttk_guionly logs "test_widgets.py:1562: UserWarni https://bugs.python.org/issue31508 opened by haypo #31509: test_subprocess hangs randomly on AMD64 Windows10 3.x https://bugs.python.org/issue31509 opened by haypo #31510: test_many_processes() of test_multiprocessing_spawn failed on https://bugs.python.org/issue31510 opened by haypo #31511: test_normalization: test.support.open_urlresource() doesn't ha https://bugs.python.org/issue31511 opened by haypo #31512: Add non-elevated symlink support for dev mode Windows 10 https://bugs.python.org/issue31512 opened by vidartf #31516: current_thread() becomes "dummy" thread during shutdown https://bugs.python.org/issue31516 opened by pitrou #31517: MainThread association logic is fragile https://bugs.python.org/issue31517 opened by pitrou #31518: ftplib, urllib2, poplib, httplib, urllib2_localnet use ssl.PRO https://bugs.python.org/issue31518 opened by doko #31520: ResourceWarning: unclosed <socket.socket [closed] fd=3, ...> w https://bugs.python.org/issue31520 opened by haypo #31521: segfault in PyBytes_AsString https://bugs.python.org/issue31521 opened by Tim Smith #31522: _mboxMMDF.get_string() fails to pass param to get_bytes() https://bugs.python.org/issue31522 opened by bpoaugust #31523: Windows build file fixes https://bugs.python.org/issue31523 opened by steve.dower #31524: mailbox._mboxMMDF.get_message throws away From envelope https://bugs.python.org/issue31524 opened by bpoaugust #31526: Allow setting timestamp in gzip-compressed tarfiles https://bugs.python.org/issue31526 opened by randombit #31527: Report details of excess arguments in object_new & object_init https://bugs.python.org/issue31527 opened by ncoghlan #31528: Let ConfigParser parse systemd units https://bugs.python.org/issue31528 opened by johnlinp #31529: IDLE: Add docstrings and tests for editor.py reload functions https://bugs.python.org/issue31529 opened by csabella #31530: Python 2.7 readahead feature of file objects is not thread saf https://bugs.python.org/issue31530 opened by haypo #31531: crash and SystemError in case of a bad zipimport._zip_director https://bugs.python.org/issue31531 opened by Oren Milman #31534: python 3.6.2 installation failed 0x80070002 error https://bugs.python.org/issue31534 opened by roie #31535: configparser unable to write comment with a upper cas letter https://bugs.python.org/issue31535 opened by philippewagnieres(a)hispeed.ch #31536: `make regen-all` triggers wholesale rebuild https://bugs.python.org/issue31536 opened by pitrou #31537: Bug in readline module documentation example https://bugs.python.org/issue31537 opened by bazwal #31538: mailbox does not treat external factories the same https://bugs.python.org/issue31538 opened by bpoaugust #31539: asyncio.sleep may sleep less time then it should https://bugs.python.org/issue31539 opened by germn #31540: Adding context in concurrent.futures.ProcessPoolExecutor https://bugs.python.org/issue31540 opened by tomMoral #31541: Mock called_with does not ensure self/cls argument is used https://bugs.python.org/issue31541 opened by jonathan.huot #31542: pth files in site-packages of venvs are executed twice https://bugs.python.org/issue31542 opened by Antony.Lee #31543: Optimize wrapper descriptors using FASTCALL https://bugs.python.org/issue31543 opened by haypo #31544: gettext.Catalog title is not flagged as a class https://bugs.python.org/issue31544 opened by linkid #31545: Fixing documentation for timedelta. https://bugs.python.org/issue31545 opened by musically_ut #31546: PyOS_InputHook is not called when waiting for input() in Windo https://bugs.python.org/issue31546 opened by yhlam #31547: IDLE: Save definitions added to user keysets https://bugs.python.org/issue31547 opened by terry.reedy #31548: test_os fails on Windows if current directory contains spaces https://bugs.python.org/issue31548 opened by serhiy.storchaka #31549: test_strptime and test_time fail on non-English Windows https://bugs.python.org/issue31549 opened by serhiy.storchaka #31550: Inconsistent error message for TypeError with subscripting https://bugs.python.org/issue31550 opened by Anthony Sottile #31551: test_distutils fails if current directory contains spaces https://bugs.python.org/issue31551 opened by serhiy.storchaka #31552: IDLE: Convert browswers to use ttk.Treeview https://bugs.python.org/issue31552 opened by terry.reedy Most recent 15 issues with no replies (15) ========================================== #31551: test_distutils fails if current directory contains spaces https://bugs.python.org/issue31551 #31549: test_strptime and test_time fail on non-English Windows https://bugs.python.org/issue31549 #31548: test_os fails on Windows if current directory contains spaces https://bugs.python.org/issue31548 #31547: IDLE: Save definitions added to user keysets https://bugs.python.org/issue31547 #31545: Fixing documentation for timedelta. https://bugs.python.org/issue31545 #31542: pth files in site-packages of venvs are executed twice https://bugs.python.org/issue31542 #31541: Mock called_with does not ensure self/cls argument is used https://bugs.python.org/issue31541 #31540: Adding context in concurrent.futures.ProcessPoolExecutor https://bugs.python.org/issue31540 #31537: Bug in readline module documentation example https://bugs.python.org/issue31537 #31535: configparser unable to write comment with a upper cas letter https://bugs.python.org/issue31535 #31534: python 3.6.2 installation failed 0x80070002 error https://bugs.python.org/issue31534 #31531: crash and SystemError in case of a bad zipimport._zip_director https://bugs.python.org/issue31531 #31528: Let ConfigParser parse systemd units https://bugs.python.org/issue31528 #31526: Allow setting timestamp in gzip-compressed tarfiles https://bugs.python.org/issue31526 #31520: ResourceWarning: unclosed <socket.socket [closed] fd=3, ...> w https://bugs.python.org/issue31520 Most recent 15 issues waiting for review (15) ============================================= #31550: Inconsistent error message for TypeError with subscripting https://bugs.python.org/issue31550 #31545: Fixing documentation for timedelta. https://bugs.python.org/issue31545 #31543: Optimize wrapper descriptors using FASTCALL https://bugs.python.org/issue31543 #31536: `make regen-all` triggers wholesale rebuild https://bugs.python.org/issue31536 #31530: Python 2.7 readahead feature of file objects is not thread saf https://bugs.python.org/issue31530 #31529: IDLE: Add docstrings and tests for editor.py reload functions https://bugs.python.org/issue31529 #31518: ftplib, urllib2, poplib, httplib, urllib2_localnet use ssl.PRO https://bugs.python.org/issue31518 #31516: current_thread() becomes "dummy" thread during shutdown https://bugs.python.org/issue31516 #31512: Add non-elevated symlink support for dev mode Windows 10 https://bugs.python.org/issue31512 #31508: Running test_ttk_guionly logs "test_widgets.py:1562: UserWarni https://bugs.python.org/issue31508 #31507: email.utils.parseaddr has no docstring https://bugs.python.org/issue31507 #31506: Improve the error message logic for object_new & object_init https://bugs.python.org/issue31506 #31505: assertion failure in json, in case _json.make_encoder() receiv https://bugs.python.org/issue31505 #31504: Documentation for return value for string.rindex is missing wh https://bugs.python.org/issue31504 #31502: IDLE: Config dialog again deletes custom themes and keysets. https://bugs.python.org/issue31502 Top 10 most discussed issues (10) ================================= #31500: IDLE: Tiny font on HiDPI display https://bugs.python.org/issue31500 19 msgs #31443: Possibly out of date C extension documentation https://bugs.python.org/issue31443 14 msgs #31504: Documentation for return value for string.rindex is missing wh https://bugs.python.org/issue31504 13 msgs #31530: Python 2.7 readahead feature of file objects is not thread saf https://bugs.python.org/issue31530 13 msgs #31484: Cache single-character strings outside of the Latin1 range https://bugs.python.org/issue31484 12 msgs #31539: asyncio.sleep may sleep less time then it should https://bugs.python.org/issue31539 12 msgs #31506: Improve the error message logic for object_new & object_init https://bugs.python.org/issue31506 11 msgs #31536: `make regen-all` triggers wholesale rebuild https://bugs.python.org/issue31536 8 msgs #31517: MainThread association logic is fragile https://bugs.python.org/issue31517 7 msgs #31496: IDLE: test_configdialog failed https://bugs.python.org/issue31496 6 msgs Issues closed (38) ================== #13224: Change str(x) to return only the qualname for some types https://bugs.python.org/issue13224 closed by merwok #15472: Itertools doc summary table misdocuments some arguments https://bugs.python.org/issue15472 closed by merwok #26510: [argparse] Add required argument to add_subparsers https://bugs.python.org/issue26510 closed by merwok #27541: Repr of collection's subclasses https://bugs.python.org/issue27541 closed by serhiy.storchaka #29916: No explicit documentation for PyGetSetDef and getter and sette https://bugs.python.org/issue29916 closed by serhiy.storchaka #29966: typing.get_type_hints doesn't really work for classes with For https://bugs.python.org/issue29966 closed by levkivskyi #30228: Open a file in text mode requires too many syscalls https://bugs.python.org/issue30228 closed by haypo #30381: test_smtpnet.test_connect_using_sslcontext_verified() randomly https://bugs.python.org/issue30381 closed by haypo #30648: test_logout() of test_imaplib.RemoteIMAP_STARTTLSTest failed r https://bugs.python.org/issue30648 closed by haypo #30658: Buildbot: don't sent email notification for custom builders https://bugs.python.org/issue30658 closed by haypo #30848: test_multiprocessing_forkserver hangs on AMD64 FreeBSD CURRENT https://bugs.python.org/issue30848 closed by haypo #30866: Add _testcapi.stack_pointer() to measure the C stack consumpti https://bugs.python.org/issue30866 closed by haypo #31008: FAIL: test_wait_for_handle (test.test_asyncio.test_windows_eve https://bugs.python.org/issue31008 closed by haypo #31016: [Regression] sphinx shows an EOF error when using python2.7 fr https://bugs.python.org/issue31016 closed by benjamin.peterson #31293: crashes in multiply_float_timedelta() and in truedivide_timede https://bugs.python.org/issue31293 closed by serhiy.storchaka #31315: assertion failure in imp.create_dynamic(), when spec.name is n https://bugs.python.org/issue31315 closed by serhiy.storchaka #31346: Prefer PROTOCOL_TLS_CLIENT/SERVER https://bugs.python.org/issue31346 closed by christian.heimes #31376: test_multiprocessing_spawn randomly hangs AMD64 FreeBSD 10.x S https://bugs.python.org/issue31376 closed by haypo #31381: Unable to read the project file "pythoncore.vcxproj". https://bugs.python.org/issue31381 closed by denis-osipov #31386: Make return types of wrap_bio and wrap_socket customizable https://bugs.python.org/issue31386 closed by christian.heimes #31404: undefined behavior and crashes in case of a bad sys.modules https://bugs.python.org/issue31404 closed by eric.snow #31410: int.__repr__() is slower than repr() https://bugs.python.org/issue31410 closed by serhiy.storchaka #31431: SSL: check_hostname should imply CERT_REQUIRED https://bugs.python.org/issue31431 closed by christian.heimes #31477: IDLE 'strip trailing whitespace' changes multiline strings https://bugs.python.org/issue31477 closed by terry.reedy #31479: Always reset the signal alarm in tests https://bugs.python.org/issue31479 closed by haypo #31482: random.seed() doesn't work with bytes and version=1 https://bugs.python.org/issue31482 closed by rhettinger #31487: Improve f-strings documentation wrt format specifiers https://bugs.python.org/issue31487 closed by Mariatta #31497: Add _PyType_Name() https://bugs.python.org/issue31497 closed by serhiy.storchaka #31498: Default values for zero in time.strftime() https://bugs.python.org/issue31498 closed by denis-osipov #31499: ElementTree crash while cleaning up ParseError https://bugs.python.org/issue31499 closed by haypo #31501: Operator precedence description for arithmetic operators https://bugs.python.org/issue31501 closed by rhettinger #31513: Document structure of memo dictionary to enable more advanced https://bugs.python.org/issue31513 closed by r.david.murray #31514: There is a problem with the setdefault type conversion in the https://bugs.python.org/issue31514 closed by steven.daprano #31515: Doc for os.makedirs is inconsistent with actual behaviour https://bugs.python.org/issue31515 closed by fpom #31519: Negative number raised to even power is negative (-1 ** even = https://bugs.python.org/issue31519 closed by r.david.murray #31525: require sqlite3_prepare_v2 https://bugs.python.org/issue31525 closed by benjamin.peterson #31532: Py_GetPath, Py_SetPath memory corruption due to mixed PyMem_Ne https://bugs.python.org/issue31532 closed by benjamin.peterson #31533: Dead link in SSLContext.set_ciphers documentation https://bugs.python.org/issue31533 closed by Mariatta

1 0

SK-CSIRT identified malicious software libraries in the official Python package repository, PyPI
by Victor Stinner 22 Sep '17

22 Sep '17

Hi, Last week, the National Security Authority of Slovakia contacted the Python Security Response Team (PSRT) to report that the Python Package Index (PyPI) was hosting malicious packages. Installing these packages send user data to a HTTP server, but also install the expected module so it was an easy to notice the attack. Advisory: http://www.nbu.gov.sk/skcsirt-sa-20170909-pypi/ Kudos to them to report the issue! It's not a compromise of the PyPI server nor a third-party project, but the "typo squatting" issue which is known since at least June 2016 (for PyPI). The issue is not specific to Python, npmjs.com or rubygems.org are vulnerable to the same issue. For example, a malicious package used the names "urllib" (no 3) and "urlib3" (1 L) instead of "urllib3" (2 L). These packages were downloaded by users, so the attack was effective. More information on typo squatting and Python package security: https://python-security.readthedocs.io/packages.html#pypi-typo-squatting The PRST contacted PyPI administrators and all identified packages were taken down, only 1h10 after the PSRT received the email from the National Security Authority of Slovakia! The typo squatting issue is known and discussed, but not solution was found yet. See for example this warehouse issue: https://github.com/pypa/warehouse/issues/2151 It seems like the consensus is that pip is not responsible to detect malicious code, it's more the responsability of PyPI. The problem is to decide how to detect malicious code and/or prevent typo squatting on PyPI. The issue has been discussed privately on the PSRT list last week. The National Security Authority of Slovakia just published their advisory, and a public discussion started on reddit: https://news.ycombinator.com/item?id=15256121 I consider that it's now time to find a solution on the public python-dev mailing list. Let's try to find a solution! Can we learn something from the Update Framework (TUF)? How does Javascript, Ruby, Perl and other programming languages deal with these security issues on their package manager? See also my other notes on Python security and the list of known CPython vulnerabilities: https://python-security.readthedocs.io/ Victor

3 5

Please accept my PR on issue bpo-31504
by ashok bakthavathsalam 20 Sep '17

20 Sep '17

Dear Sir / Madam, Against the issue https://bugs.python.org/issue31504 (related to an earlier issue https://bugs.python.org/issue24243) please accept my Pull Request. https://github.com/python/cpython/pull/3642 Thanks and regards, Ashok Bakthavathsalam

1 0

PEP 552: deterministic pycs
by Benjamin Peterson 20 Sep '17

20 Sep '17

Hello, I've written a short PEP about an import extension to allow pycs to be more deterministic by optional replacing the timestamp with a hash of the source file: https://www.python.org/dev/peps/pep-0552/ Thanks for reading, Benjamin P.S. I came up with the idea for this PEP while awake.

8 25

PEP 549: Instance Properties (aka: module properties)
by Larry Hastings 20 Sep '17

20 Sep '17

I've written a PEP proposing a language change: https://www.python.org/dev/peps/pep-0549/ The TL;DR summary: add support for property objects to modules. I've already posted a prototype. How's that sound? //arry/

13 36