Mailman 3 March 2019 - Python-Dev

PEPs from non-core devs now need a sponsor
by Brett Cannon 08 Mar '19

08 Mar '19

The steering council has implemented a new idea called sponsors to the PEP process (added in https://github.com/python/peps/commit/c58d32c33bd06eb386d3f33963a1434510528…). The thinking is that to help make sure PEPs from non-core developers receive appropriate guidance through the PEP process, a core developer needs to sign on to be a sponsor of the PEP. Being a sponsor does *not* preclude the core dev from eventually becoming a co-author or BDFL-delegate later on (but obviously not both), but the expectation is the sponsor is supportive of the idea (because if a single core dev won't sign on to help then what chance does the PEP have of being accepted?). If this doesn't turn out well we can obviously revert this, but hopefully this will make things smoother for those who are new to the PEP process.

14 22

PEP 12 updated with templates for header fields and sections
by Brett Cannon 08 Mar '19

08 Mar '19

https://github.com/python/peps/blob/master/pep-0012.rst now has a complete list of header fields along with format clues for easier copy-and-paste use in creating a new PEP. There is also a section template with one-liner explanations for what each section is for so people don't accidentally leave anything out. They are not in a single, unified template to copy to partially make sure people actually read the PEP before they start writing. :)

3 4

Summary of Python tracker Issues
by Python tracker 08 Mar '19

08 Mar '19

ACTIVITY SUMMARY (2019-03-01 - 2019-03-08) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7031 (+19) closed 40963 (+65) total 47994 (+84) Open issues with patches: 2793 Issues opened (55) ================== #33329: sigaddset() can fail on some signal numbers https://bugs.python.org/issue33329 reopened by larry #36097: Use only public C-API in _xxsubinterpreters module. https://bugs.python.org/issue36097 reopened by eric.snow #36125: Cannot cross-compile to more featureful but same tune https://bugs.python.org/issue36125 reopened by xdegaye #36160: Multiple errors in test_site.py on sysconfig._CONFIG_VARS.clea https://bugs.python.org/issue36160 opened by Ivan.Pozdeev #36161: Use thread-safe functions instead of unsafe ones (crypt, ttyna https://bugs.python.org/issue36161 opened by twouters #36164: Updating Py_InspectFlag programmatically https://bugs.python.org/issue36164 opened by ishimoto #36165: DOC: ssl.rst is missing formatting on two links https://bugs.python.org/issue36165 opened by cheryl.sabella #36166: DOC: Fix markup on function parameter on datamodel.rst https://bugs.python.org/issue36166 opened by cheryl.sabella #36167: DOC: Incorrect capitalization in Programming FAQ https://bugs.python.org/issue36167 opened by cheryl.sabella #36168: DOC: Fix capitalization in string.rst https://bugs.python.org/issue36168 opened by cheryl.sabella #36172: csv module internal consistency https://bugs.python.org/issue36172 opened by Shane Smith #36174: Remove licenseUrl field from nuget packages https://bugs.python.org/issue36174 opened by steve.dower #36176: Make IDLE Autocomplete / Calltip Window Colors Configurable https://bugs.python.org/issue36176 opened by greylaw89 #36179: _hashopenssl has reference leaks in OOM case https://bugs.python.org/issue36179 opened by christian.heimes #36180: mboxMessage.get_payload throws TypeError on malformed content https://bugs.python.org/issue36180 opened by enrico #36182: Path.write_text() docs do not include the case that a file exi https://bugs.python.org/issue36182 opened by lys.nikolaou #36184: [EASY] test_gdb.test_threads() is specific to _POSIX_THREADS, https://bugs.python.org/issue36184 opened by vstinner #36189: DOC: Correct word in tutorial introduction https://bugs.python.org/issue36189 opened by cheryl.sabella #36194: Add "make regen-configure" https://bugs.python.org/issue36194 opened by vstinner #36195: initializer is not a valid param in ThreadPoolExecutor https://bugs.python.org/issue36195 opened by harman786 #36201: AssertCountEqual does not work for nested dictionaries. https://bugs.python.org/issue36201 opened by walterqian #36202: Calling Py_DecodeLocale() before _PyPreConfig_Write() can prod https://bugs.python.org/issue36202 opened by vstinner #36203: PyWeakref_NewRef docs are misleading https://bugs.python.org/issue36203 opened by Maxwell Bernstein #36204: Deprecate calling Py_Main() after Py_Initialize()? Add Py_Init https://bugs.python.org/issue36204 opened by vstinner #36205: Python 3.7 and 3.8 process_time is not reported correctly when https://bugs.python.org/issue36205 opened by Nitapol #36207: robotsparser deny all with some rules https://bugs.python.org/issue36207 opened by quentin-maire #36208: AsyncIO V4MAPPED addresses with V6ONLY. https://bugs.python.org/issue36208 opened by jeeger #36210: ncurses versus cursus integration - platform differences and d https://bugs.python.org/issue36210 opened by Michael.Felt #36211: show full url when execute "make -C Doc/ serve" https://bugs.python.org/issue36211 opened by matrixise #36212: [2.7] Coverity scan: Modules/_hotshot.c , Variable "s1" going https://bugs.python.org/issue36212 opened by cstratak #36213: subprocess.check_output() fails with OSError: [WinError 87] wh https://bugs.python.org/issue36213 opened by Geoff.Alexander #36214: AC_RUN_IFELSE macros not used as arguments of AC_CACHE_VAL et https://bugs.python.org/issue36214 opened by xdegaye #36215: Should AppVeyor run compile Python in debug mode? https://bugs.python.org/issue36215 opened by vstinner #36216: urlsplit does not handle NFKC normalization https://bugs.python.org/issue36216 opened by steve.dower #36217: recvmsg support on Windows https://bugs.python.org/issue36217 opened by chrysn #36218: .sort() segfaults consistently on crafted input https://bugs.python.org/issue36218 opened by Lyn Levenick #36219: Add edit option in IDLE to convert smart quotes to ascii quote https://bugs.python.org/issue36219 opened by rhettinger #36220: LOAD_NAME and LOAD_GLOBAL handle dict subclasses for globals() https://bugs.python.org/issue36220 opened by Kevin Shweh #36221: Setting PYTHONASYNCIODEBUG breaks warnings https://bugs.python.org/issue36221 opened by ods #36225: Lingering subinterpreters should be implicitly cleared on shut https://bugs.python.org/issue36225 opened by ncoghlan #36226: multipart/related header causes false positive StartBoundaryNo https://bugs.python.org/issue36226 opened by cschmidbauer #36227: Add default_namespace argument to xml.etree.ElementTree.tostri https://bugs.python.org/issue36227 opened by Bernt.Røskar.Brenna #36229: Linear-time ops for some mutable collections. https://bugs.python.org/issue36229 opened by brandtbucher #36230: Please sort assertSetEqual's output https://bugs.python.org/issue36230 opened by Jess #36231: no "proper" header files on macOS 10.14 Mojave https://bugs.python.org/issue36231 opened by dimpase #36232: Improve error message on dbm.open https://bugs.python.org/issue36232 opened by rougeth #36233: xml ElementTree quotation marks of xml version string https://bugs.python.org/issue36233 opened by Photoniker #36234: test_os: add tests for invalid uid type https://bugs.python.org/issue36234 opened by vstinner #36235: distutils.sysconfig.customize_compiler() overrides CFLAGS var https://bugs.python.org/issue36235 opened by vstinner #36236: Python crash on macOS when CWD is invalid https://bugs.python.org/issue36236 opened by lkollar #36238: distutils complains "package init file 'xxx/__init__.py' not f https://bugs.python.org/issue36238 opened by jt #36239: gettext: GNUTranslations doesn't parse properly comments in de https://bugs.python.org/issue36239 opened by vstinner #36240: Definitions of time https://bugs.python.org/issue36240 opened by Robert Billing #36241: MD5 checksum is not valid for v2.7.16 "Windows x86-64 MSI inst https://bugs.python.org/issue36241 opened by andrejs-sisojevs-accenture #36243: Python os.listdir fails with FileNotFoundError when directory https://bugs.python.org/issue36243 opened by Geoff.Alexander Most recent 15 issues with no replies (15) ========================================== #36243: Python os.listdir fails with FileNotFoundError when directory https://bugs.python.org/issue36243 #36240: Definitions of time https://bugs.python.org/issue36240 #36238: distutils complains "package init file 'xxx/__init__.py' not f https://bugs.python.org/issue36238 #36232: Improve error message on dbm.open https://bugs.python.org/issue36232 #36231: no "proper" header files on macOS 10.14 Mojave https://bugs.python.org/issue36231 #36229: Linear-time ops for some mutable collections. https://bugs.python.org/issue36229 #36225: Lingering subinterpreters should be implicitly cleared on shut https://bugs.python.org/issue36225 #36221: Setting PYTHONASYNCIODEBUG breaks warnings https://bugs.python.org/issue36221 #36217: recvmsg support on Windows https://bugs.python.org/issue36217 #36215: Should AppVeyor run compile Python in debug mode? https://bugs.python.org/issue36215 #36214: AC_RUN_IFELSE macros not used as arguments of AC_CACHE_VAL et https://bugs.python.org/issue36214 #36212: [2.7] Coverity scan: Modules/_hotshot.c , Variable "s1" going https://bugs.python.org/issue36212 #36211: show full url when execute "make -C Doc/ serve" https://bugs.python.org/issue36211 #36210: ncurses versus cursus integration - platform differences and d https://bugs.python.org/issue36210 #36207: robotsparser deny all with some rules https://bugs.python.org/issue36207 Most recent 15 issues waiting for review (15) ============================================= #36236: Python crash on macOS when CWD is invalid https://bugs.python.org/issue36236 #36235: distutils.sysconfig.customize_compiler() overrides CFLAGS var https://bugs.python.org/issue36235 #36234: test_os: add tests for invalid uid type https://bugs.python.org/issue36234 #36231: no "proper" header files on macOS 10.14 Mojave https://bugs.python.org/issue36231 #36229: Linear-time ops for some mutable collections. https://bugs.python.org/issue36229 #36227: Add default_namespace argument to xml.etree.ElementTree.tostri https://bugs.python.org/issue36227 #36226: multipart/related header causes false positive StartBoundaryNo https://bugs.python.org/issue36226 #36218: .sort() segfaults consistently on crafted input https://bugs.python.org/issue36218 #36216: urlsplit does not handle NFKC normalization https://bugs.python.org/issue36216 #36211: show full url when execute "make -C Doc/ serve" https://bugs.python.org/issue36211 #36210: ncurses versus cursus integration - platform differences and d https://bugs.python.org/issue36210 #36195: initializer is not a valid param in ThreadPoolExecutor https://bugs.python.org/issue36195 #36182: Path.write_text() docs do not include the case that a file exi https://bugs.python.org/issue36182 #36179: _hashopenssl has reference leaks in OOM case https://bugs.python.org/issue36179 #36164: Updating Py_InspectFlag programmatically https://bugs.python.org/issue36164 Top 10 most discussed issues (10) ================================= #33944: Deprecate and remove pth files https://bugs.python.org/issue33944 32 msgs #35967: Better platform.processor support https://bugs.python.org/issue35967 17 msgs #33608: Add a cross-interpreter-safe mechanism to indicate that an obj https://bugs.python.org/issue33608 14 msgs #36218: .sort() segfaults consistently on crafted input https://bugs.python.org/issue36218 10 msgs #36239: gettext: GNUTranslations doesn't parse properly comments in de https://bugs.python.org/issue36239 9 msgs #29515: socket module missing IPPROTO_IPV6, IPPROTO_IPV4 on Windows https://bugs.python.org/issue29515 7 msgs #36204: Deprecate calling Py_Main() after Py_Initialize()? Add Py_Init https://bugs.python.org/issue36204 7 msgs #36227: Add default_namespace argument to xml.etree.ElementTree.tostri https://bugs.python.org/issue36227 6 msgs #36230: Please sort assertSetEqual's output https://bugs.python.org/issue36230 6 msgs #33346: Syntax error with async generator inside dictionary comprehens https://bugs.python.org/issue33346 5 msgs Issues closed (63) ================== #8075: Windows (Vista/7) install error when choosing to compile .py f https://bugs.python.org/issue8075 closed by steve.dower #9566: Compilation warnings under x64 Windows https://bugs.python.org/issue9566 closed by vstinner #10009: Automated MSI installation does not work https://bugs.python.org/issue10009 closed by steve.dower #20906: Issues in Unicode HOWTO https://bugs.python.org/issue20906 closed by vstinner #21478: mock calls don't propagate to parent (autospec) https://bugs.python.org/issue21478 closed by xtreak #24322: Hundreds of linker warnings on Windows https://bugs.python.org/issue24322 closed by steve.dower #29205: col_offset for AsyncFunctionDef AST nodes is wrong https://bugs.python.org/issue29205 closed by serhiy.storchaka #29440: _dbm requires -lgdbm if gdbm is built as static libraries https://bugs.python.org/issue29440 closed by yan12125 #30831: Inconsistent or wrong documentation around Asynchronous Contex https://bugs.python.org/issue30831 closed by yselivanov #32129: IDLE app icon is blurry on macOS with Aqua Tk 8.6 https://bugs.python.org/issue32129 closed by terry.reedy #32856: Optimize the `for y in [x]` idiom in comprehensions https://bugs.python.org/issue32856 closed by serhiy.storchaka #35198: Build issue while compiling cpp files in AIX https://bugs.python.org/issue35198 closed by vstinner #35435: Discourage external links to old docs. https://bugs.python.org/issue35435 closed by mdk #35466: Use a linked list for the ceval pending calls. https://bugs.python.org/issue35466 closed by eric.snow #35740: openssl version 1.1.1 need to be there in cpython-source-deps https://bugs.python.org/issue35740 closed by steve.dower #35808: Let's retire pgen https://bugs.python.org/issue35808 closed by pablogsal #35899: '_is_sunder' function in 'enum' module fails on empty string https://bugs.python.org/issue35899 closed by ethan.furman #35975: Put back the ability to parse files where async/await aren't k https://bugs.python.org/issue35975 closed by gvanrossum #36019: test_urllib fail in s390x buildbots: http://www.example.com/ https://bugs.python.org/issue36019 closed by vstinner #36043: FileCookieJar constructor don't accept PathLike https://bugs.python.org/issue36043 closed by brett.cannon #36091: clean up async generator from types module https://bugs.python.org/issue36091 closed by scotchka #36103: Increase shutil.COPY_BUFSIZE https://bugs.python.org/issue36103 closed by inada.naoki #36108: Avoid failing the build on race condition in clean https://bugs.python.org/issue36108 closed by steve.dower #36114: test_multiprocessing_spawn dumps core in AMD64 FreeBSD CURRENT https://bugs.python.org/issue36114 closed by vstinner #36116: test_multiprocessing_spawn fails on AMD64 Windows8 3.x https://bugs.python.org/issue36116 closed by vstinner #36139: release GIL on mmap dealloc https://bugs.python.org/issue36139 closed by benjamin.peterson #36140: An incorrect check in _msi.c's msidb_getsummaryinformation() https://bugs.python.org/issue36140 closed by steve.dower #36142: Add a new _PyPreConfig step to Python initialization to setup https://bugs.python.org/issue36142 closed by vstinner #36147: [2.7] Coverity scan: Modules/_ctypes/cfield.c , Variable "resu https://bugs.python.org/issue36147 closed by vstinner #36149: use of uninitialised memory in cPickle. https://bugs.python.org/issue36149 closed by twouters #36154: Python quit unexpectedly error https://bugs.python.org/issue36154 closed by ned.deily #36162: error: implicit declaration of function 'sendfile' is invalid https://bugs.python.org/issue36162 closed by inada.naoki #36163: same object, same method, but the is keyword return false. https://bugs.python.org/issue36163 closed by steven.daprano #36169: Add overlap() method to statistics.NormalDist() https://bugs.python.org/issue36169 closed by rhettinger #36170: posix_spawn doesn't exist in 3.7 https://bugs.python.org/issue36170 closed by ned.deily #36171: tkinter scrollbar missing 'state' option https://bugs.python.org/issue36171 closed by cheryl.sabella #36173: apam https://bugs.python.org/issue36173 closed by steven.daprano #36175: Identity of bound methods https://bugs.python.org/issue36175 closed by inada.naoki #36177: test_io: test_daemon_threads_shutdown_stdout_deadlock() fails https://bugs.python.org/issue36177 closed by vstinner #36178: type.__init__ called instead of cls.__init__ when inheriting f https://bugs.python.org/issue36178 closed by steven.daprano #36181: Add mode parameter to PurePath.write_text to allow for 'a' mod https://bugs.python.org/issue36181 closed by serhiy.storchaka #36183: test_gdb failed on AMD64 FreeBSD CURRENT Shared 3.7 and 3.x https://bugs.python.org/issue36183 closed by vstinner #36185: [EASY DOC] typo: Doc/c-api/objbuffer.rst: There is a typo in https://bugs.python.org/issue36185 closed by serhiy.storchaka #36186: [2.7] Coverity scan: Modules/linuxaudiodev.c , fd handle is no https://bugs.python.org/issue36186 closed by vstinner #36187: Get rid of NamedStore https://bugs.python.org/issue36187 closed by serhiy.storchaka #36188: Remove vestiges of Python 2 unbound methods from Python 3 https://bugs.python.org/issue36188 closed by benjamin.peterson #36190: file object method .tell() sometimes returns large number when https://bugs.python.org/issue36190 closed by tim.peters #36191: pubkeys.txt contains bogus keys https://bugs.python.org/issue36191 closed by nanjekyejoannah #36192: Usage of `tmpnam_r` and `tempname` is dangerous, better use `m https://bugs.python.org/issue36192 closed by matrixise #36193: Redirected stderr not reset properly when using logging https://bugs.python.org/issue36193 closed by vinay.sajip #36196: sys.executable does not return python3 executable when using u https://bugs.python.org/issue36196 closed by inada.naoki #36197: Compilation Warning for memoryview.tobytes and _collections._t https://bugs.python.org/issue36197 closed by vstinner #36198: Misleading in library/sets document https://bugs.python.org/issue36198 closed by DeadmanWalking #36199: libzmq.dll causes uncontrollable screen flickering when access https://bugs.python.org/issue36199 closed by eryksun #36200: display index on IndexError https://bugs.python.org/issue36200 closed by Aditya Shankar #36206: re.match() not matching escaped hyphens https://bugs.python.org/issue36206 closed by josh.r #36209: [EASY Doc] Typo in hashlib error message https://bugs.python.org/issue36209 closed by vstinner #36222: ValueError: a coroutine was expected with asyncio.run https://bugs.python.org/issue36222 closed by vstinner #36223: Execution sequence for print function https://bugs.python.org/issue36223 closed by xtreak #36224: Python quit unexpectedly error https://bugs.python.org/issue36224 closed by ned.deily #36228: Support coercion of complex to float/int https://bugs.python.org/issue36228 closed by zach.ware #36237: data_files / Install Additional Files written unclearly such t https://bugs.python.org/issue36237 closed by jt #36242: spam https://bugs.python.org/issue36242 closed by SilentGhost

1 0

Using CLA assistant for Python contributions
by Mariatta 08 Mar '19

08 Mar '19

I've posted in Discourse under core-workflow category <https://discuss.python.org/t/using-cla-assistant-for-python/990>, and this has been previously discussed on the core-workflow mailing list <https://mail.python.org/archives/list/core-workflow@python.org/thread/JBV3X…>, but I feel this affects the wider contributors to Python, so wanted to share it here for more visibility. TL;DR: We'd like to start using CLA assistant for contributions to Python (including CPython, devguide, PEPs, all the bots etc). Ernest had set up our own instance of CLA assistant, and it had been tested by several core developers. We've also consulted The PSF and Van Lindberg for legal advice. Unless I hear strong opposition (with reasons) from Python Steering Council, Python core developers, and active core contributors, I plan to switch us over to to CLA assistant in the coming week (before my OOOS of March 18) How this will affect all contributors to Python old and new: - you will need to sign the CLA again, even if you've signed it before (in bpo). It will take you several clicks, but then you'll do this only once, and it takes effect immediately. (instead of waiting for a PSF staff to check for it) - bpo username will no longer be required when signing the CLA - CLA will be accepted under Apache v2 only (no more Academic Free license) For even more details, please follow the discourse post and the core-workflow mailing list linked above, as well as the "CLA" section of my blog post about Core Python sprint 2018 <https://mariatta.ca/core-sprint-2018-part-2.html#core-sprint-2018-part-2>. Thanks. ᐧ

3 8

configparser: should optionxform be idempotent?
by Inada Naoki 08 Mar '19

08 Mar '19

Hi, all. I came from https://bugs.python.org/issue35838 Since there are no "expert" for configparser in Expert Index, I ask here to make design decision. The default behavior of CofigParser.optionxform is str.lowercase(). This is used to canonicalize option key names. The document of the optionxform shows example overrides it to identity function `lambda option: option`. https://docs.python.org/3/library/configparser.html#configparser.ConfigPars… BPO-35838 is issue about optionxform can be called twice while ConfigParser.read_dict(). If optionxfrom is not idempotent, it creates unexpected option name. https://bugs.python.org/issue35838#msg334439 But even if all APIs calls optionxform exactly once, user may read option name and value, and write updated value with same name. In this case, user read option name already optionxform-ed (canonicalized). So non-idempotent optionxform will break option name. So what should we do about optionxform? a) Document "optionxform must be idempotent". b) Ensure all APIs calls optionxform exactly once, and document "When you get option name from section objects, it is already optionxform-ed. You can not reuse the option name if optionxform is not idempotent, because optionxform will be applied to the name again." I prefer (a) to (b) because it's simple and easy solution. But for some use cases (e.g. read only, write only, use only predefined option name and read only it's value), (b) works. At least issue reporter try this use case and be trapped by this behavior. How do you think? -- Inada Naoki <songofacandy(a)gmail.com>

8 16

Re: [Python-Dev] PEP 581: Using GitHub Issues for CPython
by Matthew Woodcraft 07 Mar '19

07 Mar '19

On 07/03/2019 19.08, Mariatta wrote: > I'd like to formally present to Python-dev PEP 581: Using GitHub Issues > for CPython > > Full text: https://www.python.org/dev/peps/pep-0581/ > > This is my first PEP, and in my opinion it is ready for wider > discussion. One part of this PEP stands out to me: | We should not be moving all open issues to GitHub. Issues with little | or no activity should just be closed. Issues with no decision made for | years should just be closed. I strongly advise against closing bug reports just because they're old. I know that the Python developers value trying to be a welcoming community. To many people, having a bug report that they put some effort into closed for no reason other than the passage of time feels like a slap in the face which stings harder than, for example, intemperate words on a mailing list. This is even more true if there won't be an option to re-open the bug, which seems to be what the PEP is saying will be the case. If a bug has been around for a long time and hasn't been fixed, the most useful information for the bug tracker to contain is "this bug has been around for a long time and it hasn't been fixed". Leaving the bug open is the simplest way to achieve that. (I think the above only goes for issues which are actually reporting bugs. Wishlist items are a different matter.) -M-

3 3

Re: [Python-Dev] PEP 581: Using GitHub Issues for CPython
by Mariatta 07 Mar '19

07 Mar '19

On Thu, Mar 7, 2019 at 12:36 PM Manuel Cerón <ceronman(a)gmail.com> wrote: > > After some frustration with bpo, I decided to file some issues into the > meta tracker, just to find out that the link [1] provided by the Python > Developer's Guide [2] is broken, giving a connection timeout. > > Sometime ago we've started experimenting moving the meta tracker to GitHub. https://github.com/python/bugs.python.org I don't know whether this is now the "official" place for it, but I've definitely been referring people to this repo if they need to file issue about bpo. Again I don't know if this is now official or not, and should we start updating all documentations accordingly? ᐧ

1 0

Loading modules from a folder
by Mani Sarkar 07 Mar '19

07 Mar '19

Hi, I have seen multiple ways to load modules from a folder. Say I have a folder src which contains a number of folders each one of them is a module. What are the conventional ways to load modules from such a folder? I have used >From src.[module] import xxxxx But I don't want to use the src prefix, any other ways to indicate that in python? Thanks Cheers, Mani -- @theNeomatrix369 <http://twitter.com/theNeomatrix369>* | **Blog <http://neomatrix369.wordpress.com/>** | *@adoptopenjdk <http://twitter.com/adoptopenjdk> @graalvm <http://twitter.com/graalvm> @graal <http://twitter.com/graal> @truffleruby <http://twitter.com/truffleruby> | Dev. communities | *Bitbucket <https://bitbucket.org/neomatrix369>* * | **Github <https://github.com/neomatrix369>* * | * *Slideshare <https://slideshare.net/neomatrix369>* * | **LinkedIn <http://uk.linkedin.com/pub/mani-sarkar/71/a77/39b>* *Come to Devoxx UK 2019:* http://www.devoxx.co.uk/ *Don't chase success, rather aim for "Excellence", and success will come chasing after you!*

2 2

Compact ordered set
by INADA Naoki 07 Mar '19

07 Mar '19

Hello, folks. I'm working on compact and ordered set implementation. It has internal data structure similar to new dict from Python 3.6. It is still work in progress. Comments, tests, and documents should be updated. But it passes existing tests excluding test_sys and test_gdb (both tests checks implementation detail) https://github.com/methane/cpython/pull/16 Before completing this work, I want to evaluate it. Following is my current thoughts about the compact ordered set. ## Preserving insertion order Order is not fundamental for set. There are no order in set in the math world. But it is convenient sometime in real world. For example, it makes doctest easy. When writing set to logs, we can use "grep" command if print order is stable. pyc is stable without PYTHONHASHSEED=0 hack. Additionally, consistency with dict is desirable. It removes one pitfall for new Python users. "Remove duplicated items from list" idiom become `list(set(duplicated))` from `list(dict.fromkeys(duplicated))`. ## Memory efficiency Hash table has dilemma. To reduce collision rate, hash table should be sparse. But it wastes memory. Since current set is optimized for both of hit and miss cases, it is more sparse than dict. (It is bit surprise that set typically uses more memory than same size dict!) New implementation partially solve this dilemma. It has sparse "index table" which items are small (1byte when table size <= 256, 2bytes when table size <= 65536), and dense entry table (each item has key and hash, which is 16bytes on 64bit system). I use 1/2 for capacity rate for now. So new implementation is memory efficient when len(s) <= 32768. But memory efficiency is roughly equal to current implementation when 32768 < len(s) <= 2**31, and worse than current implementation when len(s) > 2**31. Here is quick test about memory usage. https://gist.github.com/methane/98b7f43fc00a84964f66241695112e91 # Performance pyperformance result: $ ./python -m perf compare_to master.json oset2.json -G --min-speed=2 Slower (3): - unpickle_list: 8.48 us +- 0.09 us -> 12.8 us +- 0.5 us: 1.52x slower (+52%) - unpickle: 29.6 us +- 2.5 us -> 44.1 us +- 2.5 us: 1.49x slower (+49%) - regex_dna: 448 ms +- 3 ms -> 462 ms +- 2 ms: 1.03x slower (+3%) Faster (4): - meteor_contest: 189 ms +- 1 ms -> 165 ms +- 1 ms: 1.15x faster (-13%) - telco: 15.8 ms +- 0.2 ms -> 15.3 ms +- 0.2 ms: 1.03x faster (-3%) - django_template: 266 ms +- 6 ms -> 259 ms +- 3 ms: 1.03x faster (-3%) - unpickle_pure_python: 818 us +- 6 us -> 801 us +- 9 us: 1.02x faster (-2%) Benchmark hidden because not significant (49) unpickle and unpickle_list shows massive slowdown. I suspect this slowdown is not caused from set change. Linux perf shows many pagefault is happened in pymalloc_malloc. I think memory usage changes hit weak point of pymalloc accidentally. I will try to investigate it. On the other hand, meteor_contest shows 13% speedup. It uses set. Other doesn't show significant performance changes. I need to write more benchmarks for various set workload. I expect new set is faster on simple creation, iteration and destruction. Especially, sequential iteration and deletion will reduce cache misses. (e.g. https://bugs.python.org/issue32846 ) On the other hand, new implementation will be slow on complex (heavy random add & del) case. ----- Any comments are welcome. And any benchmark for set workloads are very welcome. Regards, -- INADA Naoki <songofacandy(a)gmail.com>

12 22

[RELEASE] Python 2.7.16
by Benjamin Peterson 06 Mar '19

06 Mar '19

Hello all, I'm pleased to announce the immediate availability of Python 2.7.16 for download at https://www.python.org/downloads/release/python-2716/. The only change since the release candidate was a fix for the IDLE icon on macOS. See https://bugs.python.org/issue32129. Refer to the changelog for a full list of changes: https://raw.githubusercontent.com/python/cpython/v2.7.16/Misc/NEWS.d/2.7.16… Please report any bugs to https://bugs.python.org/. Regards, Benjamin 2.7 release manager (on behalf of all Python 2.7's contributors)

6 6