Python-Dev October 2018

python-dev@python.org

75 participants
52 discussions

The future of the wchar_t cache
by Serhiy Storchaka 23 Oct '18

23 Oct '18

Currently the PyUnicode object contains two caches: for UTF-8 representation and for wchar_t representation. They are needed not for optimization but for supporting C API which returns borrowed references for such representations. The UTF-8 cache always was in unicode objects (but in Python 2 it was not a UTF-8 cache, but a 8-bit representation cache). Initially it was needed for compatibility with 8-bit str, for implementing the "s" and "z" format units in PyArg_Parse(). Now it is used also for PyUnicode_AsUTF8() and PyUnicode_AsUTF8AndSize(). The wchar_t cache was added with PEP 393 in 3.3 as a replacement for the former Py_UNICODE representation. Now Py_UNICODE is defined as an alias of wchar_t, and the C API which returned a pointer to Py_UNICODE content returns now a pointer to the cached wchar_t representation. It is the "u" and "Z" format units in PyArg_Parse(), PyUnicode_AsUnicode(), PyUnicode_AsUnicodeAndSize(), PyUnicode_GET_SIZE(), PyUnicode_GET_DATA_SIZE(), PyUnicode_AS_UNICODE(), PyUnicode_AS_DATA(). All this increase the size of the unicode object. It includes the constant overhead of additional pointer and size fields, and the overhead of the cached representation proportional to the string length. The following table contains number of bytes per character for different kinds, with and without filling specified caches. raw +utf8 +wchar_t +utf8+wchar_t Windows Linux Windows Linux ASCII 1 1 3 5 3 5 UCS1 1 2-3 3 5 4-5 6-7 UCS2 2 3-5 2 6 3-5 7-9 UCS4 4 5-8 6-8 4 7-12 5-8 There is also a new C API added in 3.3 for getting wchar_t representation without using the cache: PyUnicode_AsWideChar() and PyUnicode_AsWideCharString(). Currently it uses the cache, this has benefits and disadvantages. Old Py_UNICODE based API is deprecated, and will be removed eventually. I want to ask about the future of the wchar_t cache. Is the benefit of caching the wchar_t representation larger the disadvantage of spending more memory? The wchar_t representation is so natural for Windows API as the UTF8 representation for POSIX API. But in all other cases it is just waste of memory. Are there reasons of keeping the wchar_t cache after removing the deprecated API? I have rewrote PyUnicode_AsWideChar() and PyUnicode_AsWideCharString(). They were implemented via the old Py_UNICODE based API, and now they don't use deprecated functions. They still use the wchar_t cache if it was created by previous use of the deprecated API, but don't create it themselves. Is this the correct decision? https://bugs.python.org/issue30863

6 17

bpo-34837: Multiprocessing.Pool API Extension - Pass Data to Workers w/o Globals
by Sean Harrington 22 Oct '18

22 Oct '18

I am proposing an extension to the multiprocessing.Pool API that allows for an alternative way to pass data to Pool worker processes, *without* using globals. A PR has been opened <https://github.com/python/cpython/pull/9627>, extensive test coverage is also included, with all tests & CI passing on github. Please see this blog post <https://thelaziestprogrammer.com/python/multiprocessing-pool-expect-initret…> for details, motivation, and use cases of the API extension before reading on. In *short*, the implementation of the feature works as follows: 1. Exposes a kwarg on Pool.__init__ called `expect_initret`, that defaults to False. When set to True: 1. Capture the return value of the initializer kwarg of Pool 2. Pass this value to the function being applied, as a kwarg. Again, in *short,* the motivation of the feature is to provide an explicit "flow of data" from parent process to worker process, and to avoid being *forced* to using the *global* keyword in initializer, or being *forced* to create global variables in the parent process. The interface is 100% backwards compatible through Python3.x (and perhaps beyond).

9 37

wininst-*.exe files in Lib/distutils/command
by VanL 22 Oct '18

22 Oct '18

Hi all, I am looking into an issue associated with the wininst-*.exe files in the distutils/command subdirectory. It looks like these are the executable stubs used to create self-extracting zips for installation - but I am not 100% sure. It also looks like they include the calls to standard Windows functions to display the installer window. I have a couple questions I need help with: 1) Am I correct about the function, and if not, what are they? 2) Where did these come from, and where is their source code? Thanks, Van

4 6

Some PRs to merge?
by Stephane Wirtel 22 Oct '18

22 Oct '18

Hi all, How are you? I am fine ;-) and you? So, on this morning I was playing with the github interface and the pull requests of CPython and I have discovered the advanced search of Github and I think this one is really useful for us and certainly for the core-dev. So, I was interested by somes PRs. PRs with this status: * open * review is approved * status of the CI is 'success' * has labels "awaiting merge", "CLA signed" and -"DO-NOT-MERGE" total: 49 PRs In the GitHub interface, here is the criteria is:open is:pr review:approved status:success label:"awaiting merge" -label:"DO-NOT-MERGE" label:""LA signed"" But if you want to see the result in your browser, just click on this link. https://github.com/python/cpython/pulls?utf8=%E2%9C%93&q=is%3Aopen+is%3Apr+… Here are the numbers: * just open: 959 is:open * and with label "CLA signed": 900 label:"CLA signed" * and with label "awaiting merge": 169 label:"awaiting merge" * and without label "DO-NOT-MERGE": 152 -label:"DO-NOT-MERGE" * with CI is happy ;-): 112 status:success * with review is approved: 49 review:approved total: 49 PRs could be merged. I was really surprised by this tool, (doc: https://help.github.com/articles/searching-issues-and-pull-requests/) But I was thinking about one thing, how can I help the core-devs to merge these PRs? Each week, I can send a report to this ML with the "mergeable" PRs. This kind of report could be useful for you? Have a nice day, Stéphane -- Stéphane Wirtel - https://wirtel.be - @matrixise

5 6

Servicing pypi.python.org
by Facundo Batista 20 Oct '18

20 Oct '18

Hola! tl;dr: can we have a (semi)permanent redirect from pypi.python.org to pypi.org? I own/maintain a project called `fades`, which is an automatic virtualenv creator/manager [0]. While it relies on `pip` itself for the heavy work on the virtualenv itself, `fades` checks if all required dependencies can be served [1] *before* starting to do the rest of the work, which saves a lot of time. I just noticed that this is broken, because `pypi.python.org` is dead, and the service should be queried under `pypi.org` directly. We probably should have switched to the new domain months ago, but you know how it's open source, it frequently happens that one has to live first and that leaves no time for fun work. So, can we have a (semi)permanent redirect from pypi.python.org to pypi.org? This will help `fades` directly (even if we fix this and release soon, there will be a lot of older fades in the wild hitting the de[recated URL) and probably other projects using the old URL. I suspect that having a generic redirect will not be burdensome server side. Can we? Can we? P-p-please??? :) Thanks in advance. [0] fades.readthedocs.io [1] using URLs like `https://pypi.python.org/pypi/Twisted/json` -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org.ar/ Twitter: @facundobatista

4 4

Re: [Python-Dev] [RELEASE] Python 3.7.1 and 3.6.7 are now available
by Terry Reedy 20 Oct '18

20 Oct '18

On 10/20/2018 1:37 PM, Ned Deily wrote: > We are also happy to announce the availability of Python 3.6.7, the next > maintenance release of Python 3.6: > https://www.python.org/downloads/release/python-367/ Am I correct in thinking that there will be just one more maintenance release before going on security-only status? -- Terry Jan Reedy

2 1

[RELEASE] Python 3.7.1 and 3.6.7 are now available
by Ned Deily 20 Oct '18

20 Oct '18

On behalf of the Python development community and the Python 3.7 release team, we are pleased to announce the availability of Python 3.7.1. Python 3.7.1 is the first maintenance release of the newest feature release of the Python language. You can find Python 3.7.1 here: https://www.python.org/downloads/release/python-371/ See the What’s New In Python 3.7 document for more information about the many new features and optimizations included in the 3.7 series. Detailed information about the changes made in 3.7.1 can be found in its change log: https://docs.python.org/3.7/whatsnew/3.7.html https://docs.python.org/3.7/whatsnew/changelog.html#python-3-7-1-final We are also happy to announce the availability of Python 3.6.7, the next maintenance release of Python 3.6: https://www.python.org/downloads/release/python-367/ Thanks to all of the many volunteers who help make Python Development and these releases possible! Please consider supporting our efforts by volunteering yourself or through organization contributions to the Python Software Foundation. -- Ned Deily nad(a)python.org -- []

1 0

Summary of Python tracker Issues
by Python tracker 19 Oct '18

19 Oct '18

ACTIVITY SUMMARY (2018-10-12 - 2018-10-19) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 6835 (+11) closed 39943 (+50) total 46778 (+61) Open issues with patches: 2737 Issues opened (41) ================== #34783: [3.7] segmentation-fault/core dump when try to run non-existin https://bugs.python.org/issue34783 reopened by ned.deily #34909: StrEnum subclasses cannot be created https://bugs.python.org/issue34909 reopened by ned.deily #34968: loop.call_soon_threadsafe should be documented to be re-entran https://bugs.python.org/issue34968 opened by njs #34969: Add --fast, --best to the gzip CLI https://bugs.python.org/issue34969 opened by matrixise #34970: Protect tasks weak set manipulation in asyncio.all_tasks() https://bugs.python.org/issue34970 opened by asvetlov #34971: add support for tls/ssl sessions in asyncio https://bugs.python.org/issue34971 opened by RemiCardona #34973: Crash in bytes constructor with mutating list https://bugs.python.org/issue34973 opened by serhiy.storchaka #34975: start_tls() difficult when using asyncio.start_server() https://bugs.python.org/issue34975 opened by icgood #34976: IDLE: Replace the search dialog with a search bar https://bugs.python.org/issue34976 opened by taleinat #34977: Release Windows Store app containing Python https://bugs.python.org/issue34977 opened by steve.dower #34978: check type of object in fix_dict.py in 2to3 https://bugs.python.org/issue34978 opened by devarakondapranav #34979: Python throws “SyntaxError: Non-UTF-8 code start with \xe8.. https://bugs.python.org/issue34979 opened by ausaki #34980: KillPython target doesn't detect 64-bit processes https://bugs.python.org/issue34980 opened by jkloth #34981: Unable to install Python from web-based installer and executab https://bugs.python.org/issue34981 opened by skycraper #34983: expose symtable.Symbol.is_nonlocal() https://bugs.python.org/issue34983 opened by pablogsal #34984: Improve error messages in bytes and bytearray constructors https://bugs.python.org/issue34984 opened by serhiy.storchaka #34985: python finds test modules from the wrong directory during PGO https://bugs.python.org/issue34985 opened by Kal Sze2 #34987: A possible null pointer dereference in _pickle.c's save_reduce https://bugs.python.org/issue34987 opened by ZackerySpytz #34990: year 2038 problem in compileall.py https://bugs.python.org/issue34990 opened by bmwiedemann #34991: variable type list [] referential integrity data loss https://bugs.python.org/issue34991 opened by alan.pan #34993: asyncio.streams.FlowControlMixin should be part of the API https://bugs.python.org/issue34993 opened by xitop #34995: functools.cached_property does not maintain the wrapped method https://bugs.python.org/issue34995 opened by mwilbz #34996: Add name to process and thread pool https://bugs.python.org/issue34996 opened by Raz Manor #35000: aexit called after loop close https://bugs.python.org/issue35000 opened by pdxjohnny #35003: Provide an option to venv to put files in a bin/ directory on https://bugs.python.org/issue35003 opened by brett.cannon #35004: Odd behavior when using datetime.timedelta under cProfile https://bugs.python.org/issue35004 opened by beaugunderson #35005: argparse should accept json and yaml argument types https://bugs.python.org/issue35005 opened by derelbenkoenig #35007: Minor change to weakref docs https://bugs.python.org/issue35007 opened by frankmillman #35009: argparse throws UnicodeEncodeError for printing help with unic https://bugs.python.org/issue35009 opened by xtreak #35012: [3.7] test_multiprocessing_spawn hangs randomly on AppVeyor https://bugs.python.org/issue35012 opened by vstinner #35015: availability directive breaks po files https://bugs.python.org/issue35015 opened by mdk #35017: socketserver accept a last request after shutdown https://bugs.python.org/issue35017 opened by beledouxdenis #35018: Sax parser provides no user access to lexical handlers https://bugs.python.org/issue35018 opened by Jonathan.Gossage #35019: Minor Bug found in asyncio - Python 3.5.3 https://bugs.python.org/issue35019 opened by bassford #35020: Add multisort recipe to sorting docs https://bugs.python.org/issue35020 opened by xtreak #35021: Assertion failures in datetimemodule.c. https://bugs.python.org/issue35021 opened by twouters #35022: MagicMock should support `__fspath__` https://bugs.python.org/issue35022 opened by Maxime Belanger #35024: Incorrect logging in importlib when '.pyc' file creation fails https://bugs.python.org/issue35024 opened by qagren #35025: Compiling `timemodule.c` can fail on macOS due to availability https://bugs.python.org/issue35025 opened by Maxime Belanger #35026: Winreg's documentation lacks mentioning required permission at https://bugs.python.org/issue35026 opened by georgefischhof #35027: distutils.core.setup does not raise TypeError when if classifi https://bugs.python.org/issue35027 opened by TilmanKrummeck Most recent 15 issues with no replies (15) ========================================== #35027: distutils.core.setup does not raise TypeError when if classifi https://bugs.python.org/issue35027 #35026: Winreg's documentation lacks mentioning required permission at https://bugs.python.org/issue35026 #35025: Compiling `timemodule.c` can fail on macOS due to availability https://bugs.python.org/issue35025 #35024: Incorrect logging in importlib when '.pyc' file creation fails https://bugs.python.org/issue35024 #35022: MagicMock should support `__fspath__` https://bugs.python.org/issue35022 #35020: Add multisort recipe to sorting docs https://bugs.python.org/issue35020 #35018: Sax parser provides no user access to lexical handlers https://bugs.python.org/issue35018 #35017: socketserver accept a last request after shutdown https://bugs.python.org/issue35017 #35009: argparse throws UnicodeEncodeError for printing help with unic https://bugs.python.org/issue35009 #35007: Minor change to weakref docs https://bugs.python.org/issue35007 #35003: Provide an option to venv to put files in a bin/ directory on https://bugs.python.org/issue35003 #35000: aexit called after loop close https://bugs.python.org/issue35000 #34984: Improve error messages in bytes and bytearray constructors https://bugs.python.org/issue34984 #34983: expose symtable.Symbol.is_nonlocal() https://bugs.python.org/issue34983 #34958: urllib.error.HTTPError.fp is not closed when error is finalize https://bugs.python.org/issue34958 Most recent 15 issues waiting for review (15) ============================================= #35025: Compiling `timemodule.c` can fail on macOS due to availability https://bugs.python.org/issue35025 #35022: MagicMock should support `__fspath__` https://bugs.python.org/issue35022 #35021: Assertion failures in datetimemodule.c. https://bugs.python.org/issue35021 #35020: Add multisort recipe to sorting docs https://bugs.python.org/issue35020 #35019: Minor Bug found in asyncio - Python 3.5.3 https://bugs.python.org/issue35019 #35017: socketserver accept a last request after shutdown https://bugs.python.org/issue35017 #34996: Add name to process and thread pool https://bugs.python.org/issue34996 #34995: functools.cached_property does not maintain the wrapped method https://bugs.python.org/issue34995 #34990: year 2038 problem in compileall.py https://bugs.python.org/issue34990 #34987: A possible null pointer dereference in _pickle.c's save_reduce https://bugs.python.org/issue34987 #34984: Improve error messages in bytes and bytearray constructors https://bugs.python.org/issue34984 #34983: expose symtable.Symbol.is_nonlocal() https://bugs.python.org/issue34983 #34980: KillPython target doesn't detect 64-bit processes https://bugs.python.org/issue34980 #34979: Python throws “SyntaxError: Non-UTF-8 code start with \xe8.. https://bugs.python.org/issue34979 #34976: IDLE: Replace the search dialog with a search bar https://bugs.python.org/issue34976 Top 10 most discussed issues (10) ================================= #34970: Protect tasks weak set manipulation in asyncio.all_tasks() https://bugs.python.org/issue34970 10 msgs #35005: argparse should accept json and yaml argument types https://bugs.python.org/issue35005 10 msgs #34953: Implement `mmap.mmap.__repr__` https://bugs.python.org/issue34953 7 msgs #34979: Python throws “SyntaxError: Non-UTF-8 code start with \xe8.. https://bugs.python.org/issue34979 7 msgs #34990: year 2038 problem in compileall.py https://bugs.python.org/issue34990 7 msgs #35015: availability directive breaks po files https://bugs.python.org/issue35015 7 msgs #34919: Crash caused by certain characters in a string https://bugs.python.org/issue34919 6 msgs #34995: functools.cached_property does not maintain the wrapped method https://bugs.python.org/issue34995 6 msgs #34095: [2.7] Seg fault on archlinux 32 when run tests with xvfb-run https://bugs.python.org/issue34095 5 msgs #34783: [3.7] segmentation-fault/core dump when try to run non-existin https://bugs.python.org/issue34783 5 msgs Issues closed (48) ================== #12572: HP/UX compiler workarounds https://bugs.python.org/issue12572 closed by terry.reedy #16965: 2to3 should rewrite execfile() to open in 'rb' mode https://bugs.python.org/issue16965 closed by serhiy.storchaka #17837: Support for building on ppc64p7 https://bugs.python.org/issue17837 closed by vstinner #22851: 2.7 core crashes with generator.gi_frame.f_restricted https://bugs.python.org/issue22851 closed by serhiy.storchaka #22872: multiprocessing.Queue raises AssertionError https://bugs.python.org/issue22872 closed by serhiy.storchaka #23420: python -m cProfile -s fails with non informative message https://bugs.python.org/issue23420 closed by vstinner #23554: EchoServerClientProtocol class should be called EchoServerProt https://bugs.python.org/issue23554 closed by yselivanov #23831: tkinter canvas lacks of moveto method. https://bugs.python.org/issue23831 closed by serhiy.storchaka #26441: email.charset: to_splittable and from_splittable are not there https://bugs.python.org/issue26441 closed by r.david.murray #30220: Why are custom messages for ValueError, TypeError suppressed i https://bugs.python.org/issue30220 closed by paul.j3 #32789: Note missing from logging.debug() docs https://bugs.python.org/issue32789 closed by vinay.sajip #32912: Raise non-silent warning for invalid escape sequences https://bugs.python.org/issue32912 closed by serhiy.storchaka #34158: Documentation of datetime '%z' format code is odd https://bugs.python.org/issue34158 closed by ned.deily #34475: functools.partial objects have no __qualname__ attribute https://bugs.python.org/issue34475 closed by serhiy.storchaka #34565: Launcher does not validate major versions https://bugs.python.org/issue34565 closed by ned.deily #34590: "Logging HOWTO" should share an example of best practices for https://bugs.python.org/issue34590 closed by vinay.sajip #34741: Get rid of tp_getattro and tp_setattro in pyexpat.xmlparser https://bugs.python.org/issue34741 closed by serhiy.storchaka #34765: Update install-sh https://bugs.python.org/issue34765 closed by vstinner #34844: logging.Formatter enhancement - Checking on style and fmt fiel https://bugs.python.org/issue34844 closed by vinay.sajip #34918: Python 3 tkinter measurement problem https://bugs.python.org/issue34918 closed by serhiy.storchaka #34920: PYTHONWARNINGS entries are escaped https://bugs.python.org/issue34920 closed by martin.panter #34927: Tkinter-related segfault on macOS (regression between 3.7.0 an https://bugs.python.org/issue34927 closed by ned.deily #34930: sha1module: Switch sha1 implementation to sha1dc/hardened sha1 https://bugs.python.org/issue34930 closed by antoine.pietri #34939: Possibly spurious SyntaxError: annotated name can't be global https://bugs.python.org/issue34939 closed by pablogsal #34941: xml.etree.ElementTree findall() fails when using custom TreeBu https://bugs.python.org/issue34941 closed by serhiy.storchaka #34961: Global scoping when shadowing local names in class definitions https://bugs.python.org/issue34961 closed by multun #34967: Sphinx is deprecating add_description_unit https://bugs.python.org/issue34967 closed by mdk #34972: json dump silently converts int keys to string https://bugs.python.org/issue34972 closed by steve.dower #34974: bytes and bytearray constructors replace unexpected exceptions https://bugs.python.org/issue34974 closed by serhiy.storchaka #34982: re.sub() different behavior in 3.7 https://bugs.python.org/issue34982 closed by serhiy.storchaka #34986: python finds test modules from the wrong directory during PGO https://bugs.python.org/issue34986 closed by Kal Sze2 #34988: Rc2 candidates: "gcc" not found on AIX https://bugs.python.org/issue34988 closed by ned.deily #34989: python-gdb.py fails with TypeError("'FakeRepr' object is not s https://bugs.python.org/issue34989 closed by vstinner #34992: many warnings with f28 and gcc 8.1.1 https://bugs.python.org/issue34992 closed by vstinner #34994: sysconfig returns the git commands for GITBRANCH, GITVERSION a https://bugs.python.org/issue34994 closed by ned.deily #34997: test.test_logging.ConfigDictTest.test_out_of_order fails in x8 https://bugs.python.org/issue34997 closed by pablogsal #34998: Logging formatter validation breaks backward ducktyping https://bugs.python.org/issue34998 closed by vinay.sajip #34999: copy.copy and deepcopy do return same logger objects in 3.7 https://bugs.python.org/issue34999 closed by vinay.sajip #35001: ImportFrom level cannot be optional https://bugs.python.org/issue35001 closed by brett.cannon #35002: Potential bug in ConfigParser.read() in python3.6, before os.P https://bugs.python.org/issue35002 closed by BNMetrics #35006: itertools.combinations has wrong type when using the typing pa https://bugs.python.org/issue35006 closed by brett.cannon #35008: Leaks xml.etree.ElementTree.Element.__setsate__() https://bugs.python.org/issue35008 closed by serhiy.storchaka #35010: sort by partially reversed key tuple https://bugs.python.org/issue35010 closed by rhettinger #35011: expat: Restore the use of pyexpatns.h to avoid link time confl https://bugs.python.org/issue35011 closed by gregory.p.smith #35013: Add more type checks for children of xml.etree.ElementTree.Ele https://bugs.python.org/issue35013 closed by serhiy.storchaka #35014: asyncio subprocess accepts string as parameter which lead to U https://bugs.python.org/issue35014 closed by natim #35016: \r to match add into address header with not-ascii character https://bugs.python.org/issue35016 closed by r.david.murray #35023: Missed a key when iterating over dictionary https://bugs.python.org/issue35023 closed by ammar2

1 0

LibSSH Vulnerability
by MRAB 18 Oct '18

18 Oct '18

I wondered if any of you have heard of this: Hacker: I'm logged in. New LibSSH Vulnerability: OK! I believe you. https://www.bleepingcomputer.com/news/security/hacker-im-logged-in-new-libs…

3 2

Arbitrary non-identifier string keys when using **kwargs
by Steven D'Aprano 16 Oct '18

16 Oct '18

While keyword arguments have to be identifiers, using **kwargs allows arbitrary strings which aren't identifiers: py> def spam(**kwargs): ... print(kwargs) ... py> spam(**{"something arbitrary": 1, '\n': 2}) {'something arbitrary': 1, '\n': 2} There is some discussion on Python-Ideas on whether or not that behaviour ought to be considered a language feature, an accident of implementation, or a bug. Can we get some guidence on this please? Thanks, -- Steve

17 25

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

Python-Dev October 2018