Mailman 3 February 2021 - Python-Dev

bpo-42819 PR review request
by Dustin Rodrigues 08 Feb '21

08 Feb '21

Hello, I submitted https://bugs.python.org/issue42819 a little over a month ago with an accompanying PR: https://github.com/python/cpython/pull/24108. Would it be possible to get feedback on it? This addresses a bug that occurs when Python is compiled with the most recent release of GNU Readline and affects both Linux and macOS. Thanks, Dustin

1 0

Change windows installation program name
by Barry Scott 08 Feb '21

08 Feb '21

I raise https://bugs.python.org/issue43156 that suggests that new users on windows would be less confused if the name of the installation program did not seem to imply it was the python program. In a nutshell the proposal is to change from this: python-3.10.0a5.exe python-3.10.0a5-amd64.exe to this (thanks to Matthew Barnett for the improved names: python-3.10.0a5-win32-setup.exe python-3.10.0a5-win64-setup.exe Barry

3 2

Comments on PEP 558
by Mark Shannon 08 Feb '21

08 Feb '21

Hi Nick, Regarding `f_locals` PEP 558 states: """ Instead of being a direct reference to the internal dynamic snapshot used to populate the independent snapshots returned by locals(), frame.f_locals will be updated to instead return a dedicated proxy type (implemented as a private subclass of the existing types.MappingProxyType) that has two internal attributes not exposed as part of the Python runtime API: * mapping: an implicitly updated snapshot of the function local variables and closure references, as well as any arbitrary items that have been set via the mapping API, even if they don't have storage allocated for them on the underlying frame * frame: the underlying frame that the snapshot is for """ This seems rather complex, and consequently fragile. I fear that this is just going to result in different bugs, rather than fixing the bugs it proposes to fix. Why not just make `f_local` a direct view on the underlying frame? It would be simpler to understand, more robust, and should perform better. Cheers, Mark.

3 13

What's up with assignment expression and tuples?
by Paul Sokolovsky 08 Feb '21

08 Feb '21

Hello, Everyone knows how hard to find a compelling usecase for the assignment expression operator (":=", colloquially "walrus operator"). https://www.python.org/dev/peps/pep-0572/ examples never felt compelling and we all remember the split it caused. I finally found a usecase where *not* using assignment expression is *much* worse than using it. I'm working on SSA (Static Single Assignment, https://en.wikipedia.org/wiki/Static_single_assignment_form) conversion of Python programs, where there's a need to "join" dataflow of values from different control flow paths using a special function (Phi function). This "joining" itself creates a new variable, and of course, the original variable was used in an expression. We've got assignment in expression, assignment expression operator to the rescue! With it, a simple loop like: ---- a = 0 while a < 5: a += 1 ---- becomes: ---- a0 = 0 while (a1 := phi(a0, a2)) < 5: a2 = a1 + 1 ---- So far, so good. But semantics of Phi function is parallel assignment. No problem with Python either, "a, b = b, c" is exactly parallel assignment. So, let's try example with 2 variables: ---- a = 0 b = 10 while a < 5: a += 1 b += 1 ---- becomes: ---- a0 = 0 b0 = 10 while ((a1, b1) := phi([a0, a2], [b0, b2]))[0] < 5: a2 = a1 + 1 b2 = b1 + 1 ---- But oops: > SyntaxError: cannot use assignment expressions with tuple To reproduce in the REPL: ---- >>> ((a, b) := (1, 2)) File "<stdin>", line 1 SyntaxError: cannot use assignment expressions with tuple ---- Why this accidental syntactic gap? Why assignment statement can do parallel assignment with a tuple on LHS, and assignment operator suddenly can't? Why the adhoc naming and conceptual shift on the AST level, when PEP572 explicitly talks about *assignment operator*, but corresponding node on the AST level is called NamedExpr? Why look at assignment expression as "name of expression" instead of assignment expression per se? It's of course not a problem to recast: NamedExpr(expr target, expr value) to NamedExpr(expr* target, expr value) in the ASDL (and it works out of the box), the point is that it should have been ExprAssign from the start (folloing the AugAssign and AnnAssign tradition). -- Best regards, Paul mailto:pmiscml@gmail.com

5 10

Dusting off PEP 533?
by Paul Bryan 07 Feb '21

07 Feb '21

I've now experienced an issue I believe that PEP 533 was intended to address: When an asynchronous context manager is created within an asynchronous generator, if the generator is not iterated fully, the context manager will not exit until the event loop cancels the task by raising a CancelledError, long after the context manager is assumed to be out of scope. Per PEP 525, I can call aclose coroutine method to cleanup the generator, but it requires the code iterating to be aware that that closing the generator is necessary. Any appetite for putting PEP 533 back on the table to address this kind of issue?

4 4

Concerns about PEP 634
by Mark Shannon 07 Feb '21

07 Feb '21

Hi, Since a decision on PEP 634 is imminent, I'd like to reiterate some concerns that I voiced last year. I am worried about the semantics and implementation of PEP 634. I don't want to comment on the merits of pattern matching in general, or the proposed syntax in PEP 634 (or PEP 640 or PEP 642). Semantics --------- 1. PEP 634 eschews the object model, in favour of adhoc instance checks, length checks and attribute accesses. This is in contrast to almost all of the the rest of the language, which uses special (dunder) methods: All operators, subscripting, attribute lookup, iteration, calls, tests, object creation, conversions, and the with statement AFAICT, no justification is given for this. Why can't pattern matching use the object model? PEP 343 (the "with" statement) added the __enter__ and __exit__ methods to the object model, and that works very well. 2. PEP 634 deliberately introduces a large area of undefined behaviour into Python. https://www.python.org/dev/peps/pep-0634/#side-effects-and-undefined-behavi… Python has, in general, been strict about not having undefined behaviour. Having defined semantics means that you can reason about programs, even non-idiomatic ones. [This is not unique to Python, it is really only C and C++ that have areas of undefined behaviour] I can see no good reason for adding undefined behaviour. It doesn't help anyone. The lack of precise semantics makes programs harder to understand, and it makes the language harder to implement. If the semantics aren't specified, then the implementation becomes the specification. This bakes bugs into the language and makes it harder to maintain, as bug-for-bug compatibility must be maintained. 3. Performance PEP 634 says that each pattern must be checked in turn. That means that multiple redundant checks must be performed on (for example) a sequence if there are several mapping patterns. This is unnecessarily slow. Implementation -------------- My main concern with the implementation is that it does too much work into the interpreter. Much of that work can and should be done in the compiler. For example, deep in the implementation of the MATCH_CLASS instruction is the following comment: https://github.com/brandtbucher/cpython/blob/patma/Python/ceval.c#L981 Such complex control flow should be handled during compilation, rather than in the interpreter. Giant instructions like MATCH_CLASS are likely to have odd corner cases, and may well have a negative impact on the performance of the rest of the language. It is a lot easier to reason about a sequence of simple bytecodes, than one giant one with context-dependent behaviour. We have spent quite a lot of effort over the last few years streamlining the interpreter. Adding these extremely complex instructions would be a big backward step. Cheers, Mark.

12 29

PEP 637 - Support for indexing with keyword arguments: request for feedback for SC submission
by Stefano Borini 06 Feb '21

06 Feb '21

Hi all, I would like to request feedback by python-dev on the current implementation of PEP 637 - Support for indexing with keyword arguments. https://www.python.org/dev/peps/pep-0637/ The PEP is ready for SC submission and it has a prototype implementation ready, available here (note, not reviewed, but apparently fully functional) https://github.com/python/cpython/compare/master...stefanoborini:PEP-637-im… (note: not sure if there's a preference for the link to be to the diff or to the branch, let me know if you prefer I change the PEP link) Thank you for your help. -- Kind regards, Stefano Borini

8 14

PR review request: shutil permission errors
by Winson Luk 06 Feb '21

06 Feb '21

Hi folks, I've had an open PR for a few weeks that still needs a review: https://github.com/python/cpython/pull/24001 This PR fixes a shutil.move() bug when permission to a directory is denied. Thanks, Winson

1 1

Summary of Python tracker Issues
by Python tracker 05 Feb '21

05 Feb '21

ACTIVITY SUMMARY (2021-01-29 - 2021-02-05) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7379 (-209) closed 47495 (+284) total 54874 (+75) Open issues with patches: 2926 Issues opened (55) ================== #16202: sys.path[0] security issues https://bugs.python.org/issue16202 reopened by christian.heimes #24275: lookdict_* give up too soon https://bugs.python.org/issue24275 reopened by Jim.Jewett #42278: Remove usage of tempfile.mktemp in stdlib https://bugs.python.org/issue42278 reopened by epaine #42616: C Extensions on Darwin that link against libpython are likely https://bugs.python.org/issue42616 reopened by froody #43048: Printing RecursionError results in RecursionError https://bugs.python.org/issue43048 reopened by iritkatriel #43064: Cannot clear signal handler set with loop.add_signal_handler i https://bugs.python.org/issue43064 opened by selimb #43065: 'concurrent.futures' stucks in 3.9.1 https://bugs.python.org/issue43065 opened by doublex #43066: Zipfile with leading slashes https://bugs.python.org/issue43066 opened by garrison.taylor #43067: inspect.signature(bytes.hex) raises ValueError "builtin has in https://bugs.python.org/issue43067 opened by boris #43068: test_subprocess: test_specific_shell() fails on AMD64 FreeBSD https://bugs.python.org/issue43068 opened by vstinner #43069: Python fails to read a script whose path is `/dev/fd/X` https://bugs.python.org/issue43069 opened by ciprian.craciun #43071: IDLE: Windows 7 - Trackpad two-finger vertical scrolling is no https://bugs.python.org/issue43071 opened by Kritzy #43072: Memory Leak when using Thread https://bugs.python.org/issue43072 opened by goyalakshat14 #43073: Adding a ctypes.Union to a ctypes.BigEndianStructure results i https://bugs.python.org/issue43073 opened by MrSurly #43075: ReDoS in request https://bugs.python.org/issue43075 opened by yetingli #43077: Update bundled pip to 21.0.1 and setuptools to 52.0.0 https://bugs.python.org/issue43077 opened by paul.moore #43079: wrong IP address range given for ipv4_mapped https://bugs.python.org/issue43079 opened by alwhaley #43080: pprint for dataclass instances https://bugs.python.org/issue43080 opened by LewisGaul #43081: Recursive call crash module multiprocessing https://bugs.python.org/issue43081 opened by xxm #43084: curses.window.enclose() should return bool https://bugs.python.org/issue43084 opened by serhiy.storchaka #43085: Loosening | and |= operator type checking restriction https://bugs.python.org/issue43085 opened by Siy_V #43086: Excess data in not handled properly in binascii.a2b_base64() https://bugs.python.org/issue43086 opened by idan22moral #43087: ctypes documentation error https://bugs.python.org/issue43087 opened by metolone #43092: Python installation fails to install pip3 if pip, setuptools i https://bugs.python.org/issue43092 opened by quotemstr #43093: Make modules picklable https://bugs.python.org/issue43093 opened by Arusekk #43094: sqlite3.create_function takes parameter named narg, not num_pa https://bugs.python.org/issue43094 opened by nchammas #43095: Windows does not raise IsADirectoryError https://bugs.python.org/issue43095 opened by Jairo Llopis #43096: Adding `read_into` method to `asyncio.StreamReader` https://bugs.python.org/issue43096 opened by jakirkham #43098: tarfile list() method does not show file type https://bugs.python.org/issue43098 opened by val.shkolnikov #43101: Add deprecation of float limits for resource to documentation https://bugs.python.org/issue43101 opened by nafur #43103: Add configure --without-static-libpython to not build libpytho https://bugs.python.org/issue43103 opened by vstinner #43105: Can't import extension modules resolved via relative paths in https://bugs.python.org/issue43105 opened by smunday #43109: When using Apple Clang, --with-lto builds should not check for https://bugs.python.org/issue43109 opened by acmorrow #43111: webbrowser.py triggers unwanted XQuartz startup https://bugs.python.org/issue43111 opened by hans-meine #43112: SOABI on Linux does not distinguish between GNU libc and musl https://bugs.python.org/issue43112 opened by ncopa #43115: locale.getlocale fails if locale is set https://bugs.python.org/issue43115 opened by AndersMunch #43118: inspect.signature() raises RuntimeError on failed to resolve t https://bugs.python.org/issue43118 opened by tkomiya #43119: asyncio.Queue.put never yields if the queue is unbounded https://bugs.python.org/issue43119 opened by spenczar #43120: Some OSX syslog facilities are missing from syslog module https://bugs.python.org/issue43120 opened by ronaldoussoren #43122: Python Launcher doesn't open a terminal window https://bugs.python.org/issue43122 opened by kjw #43123: email MIME splitting https://bugs.python.org/issue43123 opened by martin.ortner #43124: [security] smtplib multiple CRLF injection https://bugs.python.org/issue43124 opened by martin.ortner #43125: Trying To Concatenate Bytes and String in SMTPLIB https://bugs.python.org/issue43125 opened by boom0192 #43126: IOBase.readlines(0) behaviour is inconsistent with documentati https://bugs.python.org/issue43126 opened by althonos2 #43127: Unable to install Python 3.9.1 - Windows 10 https://bugs.python.org/issue43127 opened by adrian.e.d.lloyd #43128: make test fails Lib/test/test_httpservers.py on 3.9.1 https://bugs.python.org/issue43128 opened by ostermana #43129: Number of object on stack required by opcode https://bugs.python.org/issue43129 opened by mdartiailh #43131: MMAP duplicate reads and writes on ARM64 platform https://bugs.python.org/issue43131 opened by psswirhun #43132: Incorrect handling of PyObject_RichCompareBool() in the _zonei https://bugs.python.org/issue43132 opened by ZackerySpytz #43133: Accept file mode "rw" to open-or-create a file and seek to the https://bugs.python.org/issue43133 opened by ncoghlan #43134: (list have item) instate (item in list) https://bugs.python.org/issue43134 opened by mablue #43135: Can't input Japanese on idle (MacOS). https://bugs.python.org/issue43135 opened by Marusoftware #43136: multiprocessing.Queue.get(block=False) can raise queue.Empty o https://bugs.python.org/issue43136 opened by sbraz #43137: webbrowser to support "gio open <URL>" https://bugs.python.org/issue43137 opened by doko #43138: [doc] Inconsistent / confusing documentation regarding importi https://bugs.python.org/issue43138 opened by Dirk Herrmann Most recent 15 issues with no replies (15) ========================================== #43138: [doc] Inconsistent / confusing documentation regarding importi https://bugs.python.org/issue43138 #43137: webbrowser to support "gio open <URL>" https://bugs.python.org/issue43137 #43128: make test fails Lib/test/test_httpservers.py on 3.9.1 https://bugs.python.org/issue43128 #43127: Unable to install Python 3.9.1 - Windows 10 https://bugs.python.org/issue43127 #43126: IOBase.readlines(0) behaviour is inconsistent with documentati https://bugs.python.org/issue43126 #43125: Trying To Concatenate Bytes and String in SMTPLIB https://bugs.python.org/issue43125 #43124: [security] smtplib multiple CRLF injection https://bugs.python.org/issue43124 #43123: email MIME splitting https://bugs.python.org/issue43123 #43120: Some OSX syslog facilities are missing from syslog module https://bugs.python.org/issue43120 #43115: locale.getlocale fails if locale is set https://bugs.python.org/issue43115 #43111: webbrowser.py triggers unwanted XQuartz startup https://bugs.python.org/issue43111 #43109: When using Apple Clang, --with-lto builds should not check for https://bugs.python.org/issue43109 #43105: Can't import extension modules resolved via relative paths in https://bugs.python.org/issue43105 #43101: Add deprecation of float limits for resource to documentation https://bugs.python.org/issue43101 #43098: tarfile list() method does not show file type https://bugs.python.org/issue43098 Most recent 15 issues waiting for review (15) ============================================= #43132: Incorrect handling of PyObject_RichCompareBool() in the _zonei https://bugs.python.org/issue43132 #43120: Some OSX syslog facilities are missing from syslog module https://bugs.python.org/issue43120 #43119: asyncio.Queue.put never yields if the queue is unbounded https://bugs.python.org/issue43119 #43118: inspect.signature() raises RuntimeError on failed to resolve t https://bugs.python.org/issue43118 #43103: Add configure --without-static-libpython to not build libpytho https://bugs.python.org/issue43103 #43087: ctypes documentation error https://bugs.python.org/issue43087 #43086: Excess data in not handled properly in binascii.a2b_base64() https://bugs.python.org/issue43086 #43085: Loosening | and |= operator type checking restriction https://bugs.python.org/issue43085 #43084: curses.window.enclose() should return bool https://bugs.python.org/issue43084 #43080: pprint for dataclass instances https://bugs.python.org/issue43080 #43077: Update bundled pip to 21.0.1 and setuptools to 52.0.0 https://bugs.python.org/issue43077 #43075: ReDoS in request https://bugs.python.org/issue43075 #43049: Use io.IncrementalNewlineDecoder for doctest newline conversio https://bugs.python.org/issue43049 #43047: logging.config formatters documentation is out of sync with co https://bugs.python.org/issue43047 #43046: argparse: capturing actions https://bugs.python.org/issue43046 Top 10 most discussed issues (10) ================================= #43048: Printing RecursionError results in RecursionError https://bugs.python.org/issue43048 8 msgs #43080: pprint for dataclass instances https://bugs.python.org/issue43080 8 msgs #42899: Is it legal to eliminate tests of a value, when that test has https://bugs.python.org/issue42899 7 msgs #39951: Ignore specific errors when closing ssl connections https://bugs.python.org/issue39951 6 msgs #43085: Loosening | and |= operator type checking restriction https://bugs.python.org/issue43085 6 msgs #43112: SOABI on Linux does not distinguish between GNU libc and musl https://bugs.python.org/issue43112 6 msgs #24275: lookdict_* give up too soon https://bugs.python.org/issue24275 5 msgs #43103: Add configure --without-static-libpython to not build libpytho https://bugs.python.org/issue43103 4 msgs #42115: Caching infrastructure for the evaluation loop: specialised op https://bugs.python.org/issue42115 3 msgs #42825: Build libraries with "/OPT:REF" linker optimization on Windows https://bugs.python.org/issue42825 3 msgs Issues closed (262) =================== #1887: Document that distutils doesn't support out-of-source builds https://bugs.python.org/issue1887 closed by steve.dower #2200: find_executable fails to find .bat files on win32 https://bugs.python.org/issue2200 closed by steve.dower #2437: Distutils runtime_library_dirs broken on Windows https://bugs.python.org/issue2437 closed by steve.dower #2445: Use The CygwinCCompiler Under Cygwin https://bugs.python.org/issue2445 closed by steve.dower #2624: swig support in distutils should use the build and temp dirs https://bugs.python.org/issue2624 closed by steve.dower #2943: Distutils should generate a better error message when the SDK https://bugs.python.org/issue2943 closed by steve.dower #4010: configure options don't trickle down to distutils https://bugs.python.org/issue4010 closed by steve.dower #4032: distutils doesn't search ".dll.a" as library on cygwin https://bugs.python.org/issue4032 closed by steve.dower #4508: distutils compiler not handling spaces in path to output/src f https://bugs.python.org/issue4508 closed by steve.dower #5243: Missing dependency in distutils build https://bugs.python.org/issue5243 closed by steve.dower #5300: distutils should preserve +x bit on data files https://bugs.python.org/issue5300 closed by steve.dower #5936: Add MSI suport for uninstalling individual versions https://bugs.python.org/issue5936 closed by steve.dower #6040: bdist_msi does not deal with pre-release version https://bugs.python.org/issue6040 closed by steve.dower #6087: distutils.sysconfig.get_python_lib gives surprising result whe https://bugs.python.org/issue6087 closed by steve.dower #6114: distutils build_ext path comparison only based on strings https://bugs.python.org/issue6114 closed by steve.dower #6145: distutils.extension.read_setup_file misinterprets -C switch https://bugs.python.org/issue6145 closed by steve.dower #6792: Distutils-based installer does not detect 64bit versions of Py https://bugs.python.org/issue6792 closed by steve.dower #6863: Wrong linker command if CXX set to "ccache g++" https://bugs.python.org/issue6863 closed by steve.dower #7182: For non-debug builds, the cygwinccompiler.py should define NDE https://bugs.python.org/issue7182 closed by steve.dower #7202: "python setup.py cmd --verbose" does not set verbosity https://bugs.python.org/issue7202 closed by steve.dower #7219: Unhelpful error message when a distutils package install fails https://bugs.python.org/issue7219 closed by steve.dower #7412: distutils install race condition https://bugs.python.org/issue7412 closed by steve.dower #7546: msvc9compiler.py: add .asm extension https://bugs.python.org/issue7546 closed by steve.dower #7562: Custom order for the subcommands of build https://bugs.python.org/issue7562 closed by steve.dower #7677: upload: improve display for error messages from gpg https://bugs.python.org/issue7677 closed by steve.dower #8027: distutils fail to determine c++ linker with unixcompiler if us https://bugs.python.org/issue8027 closed by steve.dower #8170: Wrong Paths for distutils build --plat-name=win-amd64 https://bugs.python.org/issue8170 closed by steve.dower #8384: Better error message for executables not found https://bugs.python.org/issue8384 closed by steve.dower #8501: setup.py install --dry-run option doesn't work https://bugs.python.org/issue8501 closed by steve.dower #8763: py3K bdist_msi wrongly installs itself in ALL python versions https://bugs.python.org/issue8763 closed by steve.dower #8908: friendly errors for UAC misbehavior in windows installers https://bugs.python.org/issue8908 closed by steve.dower #8954: wininst regression: errors when building on linux https://bugs.python.org/issue8954 closed by steve.dower #8987: Distutils doesn't quote Windows command lines properly https://bugs.python.org/issue8987 closed by steve.dower #9023: distutils relative path errors https://bugs.python.org/issue9023 closed by steve.dower #9322: Don’t fail silently if ext_modules use absolute paths https://bugs.python.org/issue9322 closed by steve.dower #9912: Fail when vsvarsall.bat produces stderr https://bugs.python.org/issue9912 closed by steve.dower #9995: "setup.py register sdist upload" requires pass to be saved https://bugs.python.org/issue9995 closed by steve.dower #10374: distutils[2] should recreate scripts in the build tree https://bugs.python.org/issue10374 closed by steve.dower #10932: distutils.core.setup - data_files misbehaviour ? https://bugs.python.org/issue10932 closed by steve.dower #11376: Solaris/GCC/shared lib problem https://bugs.python.org/issue11376 closed by steve.dower #11472: upload command fails to read auth information from .pypirc https://bugs.python.org/issue11472 closed by steve.dower #11600: PY_CFLAGS and PY_CPPFLAGS inconsistent https://bugs.python.org/issue11600 closed by steve.dower #11601: UnixCCompiler always uses compiler_so, not compiler https://bugs.python.org/issue11601 closed by steve.dower #11723: Add support for mingw64 compiler https://bugs.python.org/issue11723 closed by steve.dower #11928: fail on filename with space at the end https://bugs.python.org/issue11928 closed by steve.dower #12259: Document which compilers can be created on which platform https://bugs.python.org/issue12259 closed by steve.dower #12585: distutils dereferences symlinks for zip but not for bztar/gzta https://bugs.python.org/issue12585 closed by steve.dower #12669: test_curses skipped on buildbots https://bugs.python.org/issue12669 closed by serhiy.storchaka #13055: Distutils tries to handle null versions but fails https://bugs.python.org/issue13055 closed by steve.dower #13317: building with 2to3 generates wrong import paths because build_ https://bugs.python.org/issue13317 closed by steve.dower #13420: newer() function in dep_util.py discard changes in the same se https://bugs.python.org/issue13420 closed by steve.dower #13486: msvc9compiler.py doesn't properly generate manifest files. https://bugs.python.org/issue13486 closed by steve.dower #13943: distutils’ build_py fails when package string is unicode https://bugs.python.org/issue13943 closed by steve.dower #13962: multiple lib and include directories on Linux https://bugs.python.org/issue13962 closed by steve.dower #14030: Be more careful about selecting the compiler in distutils https://bugs.python.org/issue14030 closed by steve.dower #14106: Distutils manifest: recursive-(include|exclude) matches suffix https://bugs.python.org/issue14106 closed by steve.dower #14529: distutils's build_msi command ignores the data_files argument https://bugs.python.org/issue14529 closed by steve.dower #14530: distutils's build_wininst command fails to correctly interpret https://bugs.python.org/issue14530 closed by steve.dower #14894: distutils.LooseVersion fails to compare number and a word https://bugs.python.org/issue14894 closed by steve.dower #15018: Incomplete Python LDFLAGS and CPPFLAGS used for extension modu https://bugs.python.org/issue15018 closed by steve.dower #15182: find_library_file() should try to link https://bugs.python.org/issue15182 closed by steve.dower #15205: distutils dereferences symlinks on Mac OS X but not on Linux https://bugs.python.org/issue15205 closed by steve.dower #15266: Perform the same checks as PyPI for Description field https://bugs.python.org/issue15266 closed by steve.dower #15797: bdist_msi does not pass -install/remove flags to install_scrip https://bugs.python.org/issue15797 closed by steve.dower #16108: Include maintainer information in register/upload https://bugs.python.org/issue16108 closed by steve.dower #16466: register command forgets password if no config file is created https://bugs.python.org/issue16466 closed by steve.dower #16472: Distutils+mingw links agains msvcr90, while python27.dll is li https://bugs.python.org/issue16472 closed by steve.dower #16879: distutils.command.config uses fragile constant temporary file https://bugs.python.org/issue16879 closed by steve.dower #16926: setup.py register does not always respect --repository https://bugs.python.org/issue16926 closed by steve.dower #17144: Distutils: "sdist register upload" ignores -r argument https://bugs.python.org/issue17144 closed by steve.dower #17274: distutils silently omits relative symlinks https://bugs.python.org/issue17274 closed by steve.dower #18454: distutils crashes when uploading to PyPI having only the usern https://bugs.python.org/issue18454 closed by steve.dower #18639: mingw: avoid circular dependency from time module during nativ https://bugs.python.org/issue18639 closed by steve.dower #18640: mingw: generalization of posix build in distutils/sysconfig.py https://bugs.python.org/issue18640 closed by steve.dower #18654: modernize mingw&cygwin compiler classes https://bugs.python.org/issue18654 closed by steve.dower #18970: run_setup() behavior differs from cli invocation of setup.py https://bugs.python.org/issue18970 closed by steve.dower #18987: distutils.utils.get_platform() for 32-bit Python on a 64-bit m https://bugs.python.org/issue18987 closed by steve.dower #19549: PKG-INFO is created with CRLF on Windows https://bugs.python.org/issue19549 closed by steve.dower #20106: warn_dir is always true for install_data, even if an install_d https://bugs.python.org/issue20106 closed by steve.dower #20212: distutils: fix build_ext check to find whether we're building https://bugs.python.org/issue20212 closed by steve.dower #20337: bdist_rpm should support %config(noreplace) https://bugs.python.org/issue20337 closed by steve.dower #20397: distutils --record option does not validate existence of byte- https://bugs.python.org/issue20397 closed by steve.dower #20464: Update distutils sample config file in Doc/install/index.rst https://bugs.python.org/issue20464 closed by steve.dower #20705: distutils.extension.Extension with empty 'sources' list https://bugs.python.org/issue20705 closed by steve.dower #20788: distutils.msvccompiler - flags are hidden inside initialize() https://bugs.python.org/issue20788 closed by steve.dower #20819: reinitialize_command doesn't clear install_lib on install and https://bugs.python.org/issue20819 closed by steve.dower #21409: setup.py check - should fail and retrun a non 0 exit code https://bugs.python.org/issue21409 closed by steve.dower #21410: setup.py check --restructuredtext -- appears to pass if docut https://bugs.python.org/issue21410 closed by steve.dower #21453: Support of RPM subpackages in distutils https://bugs.python.org/issue21453 closed by steve.dower #21710: --install-base option ignored? https://bugs.python.org/issue21710 closed by steve.dower #21749: pkgutil ImpLoader does not support frozen modules https://bugs.python.org/issue21749 closed by steve.dower #21821: The function cygwinccompiler.is_cygwingcc leads to FileNotFoun https://bugs.python.org/issue21821 closed by steve.dower #22269: Resolve distutils option conflicts with priorities https://bugs.python.org/issue22269 closed by steve.dower #22309: distutils/spawn.py handle fork() not implemented. https://bugs.python.org/issue22309 closed by steve.dower #22888: ensurepip and distutils' build_scripts fails on Windows when p https://bugs.python.org/issue22888 closed by steve.dower #22990: bdist installation dialog https://bugs.python.org/issue22990 closed by steve.dower #23023: ./Modules/ld_so_aix not found on AIX during test_distutils https://bugs.python.org/issue23023 closed by steve.dower #23102: distutils: isinstance checks fail with setuptools-monkeypatche https://bugs.python.org/issue23102 closed by steve.dower #24239: Allow to configure which gpg to use in distutils upload https://bugs.python.org/issue24239 closed by steve.dower #24591: offer option to suppress "clean --all" output relating to none https://bugs.python.org/issue24591 closed by steve.dower #24673: distutils/_msvccompiler does not remove /DLL during link(CComp https://bugs.python.org/issue24673 closed by steve.dower #24691: out of memory in distutils.upload with large files https://bugs.python.org/issue24691 closed by steve.dower #24908: sysconfig.py and distutils.sysconfig.py disagree on directory https://bugs.python.org/issue24908 closed by steve.dower #24935: LDSHARED is not set according when CC is set. https://bugs.python.org/issue24935 closed by steve.dower #25100: Test_distutils fails instead of skipping if no VS2015 https://bugs.python.org/issue25100 closed by steve.dower #25187: bdist_rpm fails due to wrong hardcoded assumption about RPM fi https://bugs.python.org/issue25187 closed by steve.dower #25229: distutils doesn't add "-Wl," prefix to "-R" on Linux if the C https://bugs.python.org/issue25229 closed by steve.dower #25385: Spurious warning when compiling extension module https://bugs.python.org/issue25385 closed by steve.dower #25392: setup.py --quiet doesn't silence "no previously-included direc https://bugs.python.org/issue25392 closed by steve.dower #25544: cleanup temporary files in distutils.has_function https://bugs.python.org/issue25544 closed by steve.dower #25627: distutils : file "bdist_rpm.py" does not quote filenames when https://bugs.python.org/issue25627 closed by steve.dower #26425: 'TypeError: object of type 'NoneType' has no len()' in 'splitd https://bugs.python.org/issue26425 closed by steve.dower #26786: bdist_msi duplicates directories with names in ALL CAPS to a b https://bugs.python.org/issue26786 closed by steve.dower #26790: bdist_msi package duplicates everything to a bogus location wh https://bugs.python.org/issue26790 closed by steve.dower #26872: Default ConfigParser in python is not able to load values habi https://bugs.python.org/issue26872 closed by steve.dower #26876: Extend MSVCCompiler class to respect environment variables https://bugs.python.org/issue26876 closed by steve.dower #27004: Handle script shbang options https://bugs.python.org/issue27004 closed by steve.dower #27123: Allow `install_headers` command to follow specific directory s https://bugs.python.org/issue27123 closed by steve.dower #27226: distutils: unable to compile both .opt-1.pyc and .opt2.pyc sim https://bugs.python.org/issue27226 closed by steve.dower #27317: Handling data_files: too much is removed in uninstall https://bugs.python.org/issue27317 closed by steve.dower #27790: test_distutils spews linker messages on Windows https://bugs.python.org/issue27790 closed by steve.dower #27919: Deprecate and remove extra_path distribution kwarg https://bugs.python.org/issue27919 closed by steve.dower #28209: Exe or MSI unable to find Py3.5 https://bugs.python.org/issue28209 closed by steve.dower #28981: distutils/check.py overzealous catch block hides errors https://bugs.python.org/issue28981 closed by steve.dower #29225: distutils.command.install_lib.get_outputs() wrong with extensi https://bugs.python.org/issue29225 closed by steve.dower #29313: msi by bdist_msi will fail execute install-scripts if space in https://bugs.python.org/issue29313 closed by steve.dower #29320: bdist_msi install_script fail to execute if alt python locatio https://bugs.python.org/issue29320 closed by steve.dower #29408: Add timestamp-based dependency system to build_clib https://bugs.python.org/issue29408 closed by steve.dower #29895: Distutils blows up with an incorrect pypirc, should be caught https://bugs.python.org/issue29895 closed by steve.dower #30613: Distutils register command creates non-standard multipart data https://bugs.python.org/issue30613 closed by steve.dower #30925: RPM build lacks ability to include other files similar to doc_ https://bugs.python.org/issue30925 closed by steve.dower #31332: Building modules by Clang with Microsoft CodeGen https://bugs.python.org/issue31332 closed by steve.dower #31551: test_distutils fails if current directory contains spaces https://bugs.python.org/issue31551 closed by steve.dower #32054: Creating RPM on Python 2 works, but Python 3 fails because of https://bugs.python.org/issue32054 closed by steve.dower #32223: distutils doesn't correctly read UTF-8 content from config fil https://bugs.python.org/issue32223 closed by steve.dower #32597: Bad detection of clang https://bugs.python.org/issue32597 closed by steve.dower #32716: setup.py register --repository is broken https://bugs.python.org/issue32716 closed by steve.dower #32773: distutils should NOT preserve timestamps https://bugs.python.org/issue32773 closed by steve.dower #32957: distutils.command.install checks truthiness of .ext_modules in https://bugs.python.org/issue32957 closed by steve.dower #33168: distutils build/build_ext and --debug https://bugs.python.org/issue33168 closed by steve.dower #33457: python-config ldflags, PEP 513 and explicit linking to libpyth https://bugs.python.org/issue33457 closed by steve.dower #33466: Distutils does not support the compilation of Objective-C++ (� https://bugs.python.org/issue33466 closed by steve.dower #33637: pip cannot build extensions for debug Python https://bugs.python.org/issue33637 closed by steve.dower #33875: Allow dynamic password evaluation in pypirc configuration file https://bugs.python.org/issue33875 closed by steve.dower #33969: "copytree" refuses to copy to a mount point https://bugs.python.org/issue33969 closed by steve.dower #33980: distutils upload: SSL Error when uploading package to your own https://bugs.python.org/issue33980 closed by steve.dower #33990: CPPFLAGS during ./configure are not passed-through in sysconfi https://bugs.python.org/issue33990 closed by steve.dower #34012: No option to include system headers in distutils.core.Extensio https://bugs.python.org/issue34012 closed by steve.dower #34322: modification to Lib/distutils/ccompiler.py to simplify handlin https://bugs.python.org/issue34322 closed by steve.dower #35172: Add support for other MSVC compiler versions to distutils. dis https://bugs.python.org/issue35172 closed by steve.dower #35244: Allow to setup Clang as default compiler for modules build https://bugs.python.org/issue35244 closed by steve.dower #35295: Please clarify whether PyUnicode_AsUTF8AndSize() or PyUnicode_ https://bugs.python.org/issue35295 closed by methane #35718: Cannot initialize the distutils "force" command-option https://bugs.python.org/issue35718 closed by steve.dower #35882: distutils fails with UnicodeEncodeError with strange filename https://bugs.python.org/issue35882 closed by steve.dower #35893: distutils fails to build extension on windows when it is a pac https://bugs.python.org/issue35893 closed by steve.dower #36238: distutils complains "package init file 'xxx/__init__.py' not f https://bugs.python.org/issue36238 closed by steve.dower #36353: rpath incorrectly handled on OSX by build_ext https://bugs.python.org/issue36353 closed by steve.dower #36383: Virtual environment sysconfig.get_path() and distutils.sysconf https://bugs.python.org/issue36383 closed by steve.dower #36640: distutils.sysconfig.get_python_lib behaves incorrectly when pi https://bugs.python.org/issue36640 closed by steve.dower #36768: distutils.util.convert_path mangles windows paths with forward https://bugs.python.org/issue36768 closed by steve.dower #36998: distutils sdist command fails to create MANIFEST if any filena https://bugs.python.org/issue36998 closed by steve.dower #37247: swap distutils build_ext and build_py commands to allow proper https://bugs.python.org/issue37247 closed by steve.dower #37541: get_python_lib() returns incorrect path for site-packages https://bugs.python.org/issue37541 closed by steve.dower #37570: [Windows] distutils.util.byte_compile() fails indirect byte co https://bugs.python.org/issue37570 closed by steve.dower #37680: distutils appends LDFLAGS environment variable before object f https://bugs.python.org/issue37680 closed by steve.dower #37794: Replace /Ox with /O2 https://bugs.python.org/issue37794 closed by steve.dower #37916: distutils: allow overriding of the RANLIB command on macOS (da https://bugs.python.org/issue37916 closed by steve.dower #38078: IDLE: Don't run internal code in user namespace. https://bugs.python.org/issue38078 closed by terry.reedy #38569: Unknown distribution option: 'license_files' https://bugs.python.org/issue38569 closed by steve.dower #38597: C Extension import limit https://bugs.python.org/issue38597 closed by steve.dower #38632: setup.py sdist should honor SOURCE_DATE_EPOCH https://bugs.python.org/issue38632 closed by steve.dower #38709: distutils - setuptools - alias command removes comments from s https://bugs.python.org/issue38709 closed by steve.dower #38711: setup parameter 'distclass' ignored for configuration files https://bugs.python.org/issue38711 closed by steve.dower #38714: setup command alias erroneous for names with hyphens https://bugs.python.org/issue38714 closed by steve.dower #38718: query of global metadata options delivers error messages even https://bugs.python.org/issue38718 closed by steve.dower #38727: setup.py sdist --format=gztar should use (equivalent of) `gzip https://bugs.python.org/issue38727 closed by steve.dower #39260: distutils.spawn: find_executable() Fails To Find Many Executab https://bugs.python.org/issue39260 closed by steve.dower #39286: Configure includes LIBS but does not pass it to distutils https://bugs.python.org/issue39286 closed by steve.dower #39917: new_compiler() called 2nd time causes error https://bugs.python.org/issue39917 closed by steve.dower #40042: Enum Flag: psuedo-members have None for name attribute https://bugs.python.org/issue40042 closed by ethan.furman #40070: GCC crashed on AMD64 RHEL7 LTO + PGO 3.7 (compiler bug) https://bugs.python.org/issue40070 closed by vstinner #40455: GCC 10 compiler warnings https://bugs.python.org/issue40455 closed by corona10 #40478: allow finding nmake.exe executable in MSVCCompiler https://bugs.python.org/issue40478 closed by steve.dower #40530: distutils/cygwinccompiler.py doesn't support recent msvc versi https://bugs.python.org/issue40530 closed by steve.dower #40963: distutils make_zipfile uses random order https://bugs.python.org/issue40963 closed by steve.dower #41013: test_os.test_memfd_create() fails on AMD64 FreeBSD Shared 3.x https://bugs.python.org/issue41013 closed by vstinner #41027: get_version() fails to return gcc version for gcc-7 https://bugs.python.org/issue41027 closed by steve.dower #41134: distutils.dir_util.copy_tree FileExistsError when updating sym https://bugs.python.org/issue41134 closed by steve.dower #41143: distutils uses the locale encoding for the .pypirc file https://bugs.python.org/issue41143 closed by steve.dower #41149: Threads can fail to start https://bugs.python.org/issue41149 closed by pablogsal #41154: test_pkgutil:test_name_resolution fails on macOS HFS+ file sys https://bugs.python.org/issue41154 closed by steve.dower #41490: Update bundled pip to 20.2.1 and setuptools to 49.2.1 https://bugs.python.org/issue41490 closed by steve.dower #41601: Performance issue using isspace() in extension module on Windo https://bugs.python.org/issue41601 closed by steve.dower #41604: [curses.panel] Failed panel.set_userptr will decrement refcou https://bugs.python.org/issue41604 closed by serhiy.storchaka #41748: HTMLParser: comma in attribute values with/without space https://bugs.python.org/issue41748 closed by ezio.melotti #41880: Get Python include directories from sysconfigdata https://bugs.python.org/issue41880 closed by steve.dower #41882: CCompiler.has_function does not delete temporary files https://bugs.python.org/issue41882 closed by steve.dower #42009: Unable to compile with message compiler due to source order https://bugs.python.org/issue42009 closed by steve.dower #42204: "import setuptools" Results in "ModuleNotFoundError: No module https://bugs.python.org/issue42204 closed by steve.dower #42323: [AIX] test_math: test_nextafter(float('nan'), 1.0) does not re https://bugs.python.org/issue42323 closed by vstinner #42554: distutils.util.get_platform() depends on minor version for mac https://bugs.python.org/issue42554 closed by steve.dower #42596: aarch64 Fedora Rawhide LTO + PGO 3.8 fails to build Python: /u https://bugs.python.org/issue42596 closed by vstinner #42605: dir_util.copy_tree crashes if folder it previously created is https://bugs.python.org/issue42605 closed by steve.dower #42628: binascii doesn't work on some base64 https://bugs.python.org/issue42628 closed by serhiy.storchaka #42727: [Enum] EnumMeta.__prepare__ needs to accept **kwds https://bugs.python.org/issue42727 closed by ethan.furman #42773: build.yml workflow not testing on pushes https://bugs.python.org/issue42773 closed by ammar2 #42834: [subinterpreters] Convert "global" static variable caches in _ https://bugs.python.org/issue42834 closed by vstinner #42865: sysconfig appends CFLAGS to LD https://bugs.python.org/issue42865 closed by steve.dower #42911: Addition chains for pow saves 5-20% time for pow(int,int) https://bugs.python.org/issue42911 closed by mark.dickinson #42927: Inline cache for slots https://bugs.python.org/issue42927 closed by gvanrossum #42986: pegen parser: Crash on SyntaxError with f-string on Windows https://bugs.python.org/issue42986 closed by pablogsal #42990: Improve the C code for calling Python code: _PyEval_EvalCode() https://bugs.python.org/issue42990 closed by Mark.Shannon #42997: Improve error message for missing : before suites https://bugs.python.org/issue42997 closed by pablogsal #43016: Improve tests for curses https://bugs.python.org/issue43016 closed by serhiy.storchaka #43017: Improve error message in the parser when using un-parenthesise https://bugs.python.org/issue43017 closed by pablogsal #43029: unittest: Add assertUniqeIn https://bugs.python.org/issue43029 closed by terry.reedy #43030: signed/unsigned mismatch in Py_UNICODE_ISSPACE macro https://bugs.python.org/issue43030 closed by serhiy.storchaka #43035: FileNotFoundError in distutils\file_util.py copy_tree https://bugs.python.org/issue43035 closed by steve.dower #43039: tempfile.TemporaryDirectory() name string is incorrect https://bugs.python.org/issue43039 closed by eric.smith #43053: Speed up math.isqrt, again https://bugs.python.org/issue43053 closed by mark.dickinson #43056: Use of dicts in 4.2 and 4.7 of Python tutorial a bit confusing https://bugs.python.org/issue43056 closed by rhettinger #43059: sqlite3: Externally developed? https://bugs.python.org/issue43059 closed by berker.peksag #43062: Non-integer values in collections.Counter https://bugs.python.org/issue43062 closed by rhettinger #43070: Control keys stop working after pressing Ctrl-C Escape Enter https://bugs.python.org/issue43070 closed by trey #43074: Unable to use system proxy with HTTPS connection on Windows https://bugs.python.org/issue43074 closed by admin3 #43076: str.split() indexing issue https://bugs.python.org/issue43076 closed by steven.daprano #43078: Equality Errors when Unpickling and Pickling a Dictionary with https://bugs.python.org/issue43078 closed by rp4fx12 #43082: descriptor howto: typo in "Definition and introduction" https://bugs.python.org/issue43082 closed by rhettinger #43083: Minor errors related to error handling in _sqlite3 module https://bugs.python.org/issue43083 closed by serhiy.storchaka #43088: Regression in math.hypot https://bugs.python.org/issue43088 closed by rhettinger #43089: The optional argument to random.Random() is called "x" not "se https://bugs.python.org/issue43089 closed by rhettinger #43090: parseaddr (from email.utils) returns invalid input string inst https://bugs.python.org/issue43090 closed by r.david.murray #43091: console encode is not utf-8!! https://bugs.python.org/issue43091 closed by christian.heimes #43097: IndexError msg of random.choice() not helpful https://bugs.python.org/issue43097 closed by rhettinger #43099: Deleted positions lists https://bugs.python.org/issue43099 closed by eric.smith #43100: Python fails to execute shiv pyz files greater than 2 GB https://bugs.python.org/issue43100 closed by mark.dickinson #43102: namedtuple's __new__.__globals__['__builtins__'] is None https://bugs.python.org/issue43102 closed by rhettinger #43104: csv package in class https://bugs.python.org/issue43104 closed by eric.smith #43106: Some macOS open flags are missing from posixmodule.c https://bugs.python.org/issue43106 closed by corona10 #43107: RotatingFileHandler with multi processes creates too small bac https://bugs.python.org/issue43107 closed by vinay.sajip #43108: test_curses is leaking references https://bugs.python.org/issue43108 closed by pablogsal #43110: import aiohttp crash closes Python from Windows Installer https://bugs.python.org/issue43110 closed by giovanniwijaya #43113: os.posix_spawn errors with wrong information when shebang poin https://bugs.python.org/issue43113 closed by gregory.p.smith #43114: Python 3.6 MSI Installer for Windows https://bugs.python.org/issue43114 closed by steve.dower #43116: Integer division numerical error https://bugs.python.org/issue43116 closed by mark.dickinson #43117: Translation Mistakes https://bugs.python.org/issue43117 closed by Mariatta #43121: Incorrect SyntaxError message for missing comma (3.10.a5) https://bugs.python.org/issue43121 closed by pablogsal #43130: Should this construct throw an exception? https://bugs.python.org/issue43130 closed by eric.smith #809163: Can't add files with spaces under bdist_rpm https://bugs.python.org/issue809163 closed by steve.dower #1109658: distutils dry-run breaks when attempting to bytecompile https://bugs.python.org/issue1109658 closed by steve.dower #1371826: distutils is silent about multiple -I/-L/-R https://bugs.python.org/issue1371826 closed by steve.dower #1112955: move_file()'s return value when dry_run=1 unclear https://bugs.python.org/issue1112955 closed by steve.dower #1016626: distutils support for swig is under par https://bugs.python.org/issue1016626 closed by steve.dower #1382562: --install-base not honored on win32 https://bugs.python.org/issue1382562 closed by steve.dower #1254718: GCC detection for runtime_library_dirs when ccache is used https://bugs.python.org/issue1254718 closed by steve.dower #993766: bdist_dumb and --relative on Windows fails https://bugs.python.org/issue993766 closed by steve.dower #919238: Recursive variable definition causes sysconfig infinite loop https://bugs.python.org/issue919238 closed by steve.dower #1052827: filelist.findall should not fail on dangling symlinks https://bugs.python.org/issue1052827 closed by steve.dower #832159: Bad value for Mingw32Compiler.compiler_cxx https://bugs.python.org/issue832159 closed by steve.dower #763043: unable to specify another compiler https://bugs.python.org/issue763043 closed by steve.dower #644744: bdist_rpm fails when installing man pages https://bugs.python.org/issue644744 closed by steve.dower #1479255: Fix building with SWIG's -c++ option set in setup.py https://bugs.python.org/issue1479255 closed by steve.dower

1 0

PEP 624: Remove Py_UNICODE encoder APIs
by Inada Naoki 04 Feb '21

04 Feb '21

Hi, folks. Since the previous discussion was suspended without consensus, I wrote a new PEP for it. (Thank you Victor for reviewing it!) This PEP looks very similar to PEP 623 "Remove wstr from Unicode", but for encoder APIs, not for Unicode object APIs. URL (not available yet): https://www.python.org/dev/peps/pep-0624/ --- PEP: 624 Title: Remove Py_UNICODE encoder APIs Author: Inada Naoki <songofacandy(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 06-Jul-2020 Python-Version: 3.11 Abstract ======== This PEP proposes to remove deprecated ``Py_UNICODE`` encoder APIs in Python 3.11: * ``PyUnicode_Encode()`` * ``PyUnicode_EncodeASCII()`` * ``PyUnicode_EncodeLatin1()`` * ``PyUnicode_EncodeUTF7()`` * ``PyUnicode_EncodeUTF8()`` * ``PyUnicode_EncodeUTF16()`` * ``PyUnicode_EncodeUTF32()`` * ``PyUnicode_EncodeUnicodeEscape()`` * ``PyUnicode_EncodeRawUnicodeEscape()`` * ``PyUnicode_EncodeCharmap()`` * ``PyUnicode_TranslateCharmap()`` * ``PyUnicode_EncodeDecimal()`` * ``PyUnicode_TransformDecimalToASCII()`` .. note:: `PEP 623 <https://www.python.org/dev/peps/pep-0623/>`_ propose to remove Unicode object APIs relating to ``Py_UNICODE``. On the other hand, this PEP is not relating to Unicode object. These PEPs are split because they have different motivation and need different discussion. Motivation ========== In general, reducing the number of APIs that have been deprecated for a long time and have few users is a good idea for not only it improves the maintainability of CPython, but it also helps API users and other Python implementations. Rationale ========= Deprecated since Python 3.3 --------------------------- ``Py_UNICODE`` and APIs using it are deprecated since Python 3.3. Inefficient ----------- All of these APIs are implemented using ``PyUnicode_FromWideChar``. So these APIs are inefficient when user want to encode Unicode object. Not used widely --------------- When searching from top 4000 PyPI packages [1]_, only pyodbc use these APIs. * ``PyUnicode_EncodeUTF8()`` * ``PyUnicode_EncodeUTF16()`` pyodbc uses these APIs to encode Unicode object into bytes object. So it is easy to fix it. [2]_ Alternative APIs ================ There are alternative APIs to accept ``PyObject *unicode`` instead of ``Py_UNICODE *``. Users can migrate to them. ========================================= ========================================== Deprecated API Alternative APIs ========================================= ========================================== ``PyUnicode_Encode()`` ``PyUnicode_AsEncodedString()`` ``PyUnicode_EncodeASCII()`` ``PyUnicode_AsASCIIString()`` \(1) ``PyUnicode_EncodeLatin1()`` ``PyUnicode_AsLatin1String()`` \(1) ``PyUnicode_EncodeUTF7()`` \(2) ``PyUnicode_EncodeUTF8()`` ``PyUnicode_AsUTF8String()`` \(1) ``PyUnicode_EncodeUTF16()`` ``PyUnicode_AsUTF16String()`` \(3) ``PyUnicode_EncodeUTF32()`` ``PyUnicode_AsUTF32String()`` \(3) ``PyUnicode_EncodeUnicodeEscape()`` ``PyUnicode_AsUnicodeEscapeString()`` ``PyUnicode_EncodeRawUnicodeEscape()`` ``PyUnicode_AsRawUnicodeEscapeString()`` ``PyUnicode_EncodeCharmap()`` ``PyUnicode_AsCharmapString()`` \(1) ``PyUnicode_TranslateCharmap()`` ``PyUnicode_Translate()`` ``PyUnicode_EncodeDecimal()`` \(4) ``PyUnicode_TransformDecimalToASCII()`` \(4) ========================================= ========================================== Notes: (1) ``const char *errors`` parameter is missing. (2) There is no public alternative API. But user can use generic ``PyUnicode_AsEncodedString()`` instead. (3) ``const char *errors, int byteorder`` parameters are missing. (4) There is no direct replacement. But ``Py_UNICODE_TODECIMAL`` can be used instead. CPython uses ``_PyUnicode_TransformDecimalAndSpaceToASCII`` for converting from Unicode to numbers instead. Plan ==== Python 3.9 ---------- Add ``Py_DEPRECATED(3.3)`` to following APIs. This change is committed already [3]_. All other APIs have been marked ``Py_DEPRECATED(3.3)`` already. * ``PyUnicode_EncodeDecimal()`` * ``PyUnicode_TransformDecimalToASCII()``. Document all APIs as "will be removed in version 3.11". Python 3.11 ----------- These APIs are removed. * ``PyUnicode_Encode()`` * ``PyUnicode_EncodeASCII()`` * ``PyUnicode_EncodeLatin1()`` * ``PyUnicode_EncodeUTF7()`` * ``PyUnicode_EncodeUTF8()`` * ``PyUnicode_EncodeUTF16()`` * ``PyUnicode_EncodeUTF32()`` * ``PyUnicode_EncodeUnicodeEscape()`` * ``PyUnicode_EncodeRawUnicodeEscape()`` * ``PyUnicode_EncodeCharmap()`` * ``PyUnicode_TranslateCharmap()`` * ``PyUnicode_EncodeDecimal()`` * ``PyUnicode_TransformDecimalToASCII()`` Alternative ideas ================= Instead of just removing deprecated APIs, we may be able to use thier names with different signature. Make some private APIs public ------------------------------ ``PyUnicode_EncodeUTF7()`` doesn't have public alternative APIs. Some APIs have alternative public APIs. But they are missing ``const char *errors`` or ``int byteorder`` parameters. We can rename some private APIs and make them public to cover missing APIs and parameters. ============================= ================================ Rename to Rename from ============================= ================================ ``PyUnicode_EncodeASCII()`` ``_PyUnicode_AsASCIIString()`` ``PyUnicode_EncodeLatin1()`` ``_PyUnicode_AsLatin1String()`` ``PyUnicode_EncodeUTF7()`` ``_PyUnicode_EncodeUTF7()`` ``PyUnicode_EncodeUTF8()`` ``_PyUnicode_AsUTF8String()`` ``PyUnicode_EncodeUTF16()`` ``_PyUnicode_EncodeUTF16()`` ``PyUnicode_EncodeUTF32()`` ``_PyUnicode_EncodeUTF32()`` ============================= ================================ Pros: * We have more consistent API set. Cons: * We have more public APIs to maintain. * Existing public APIs are enough for most use cases, and ``PyUnicode_AsEncodedString()`` can be used in other cases. Replace ``Py_UNICODE*`` with ``Py_UCS4*`` ----------------------------------------- We can replace ``Py_UNICODE`` (typedef of ``wchar_t``) with ``Py_UCS4``. Since builtin codecs support UCS-4, we don't need to convert ``Py_UCS4*`` string to Unicode object. Pros: * We have more consistent API set. * User can encode UCS-4 string in C without creating Unicode object. Cons: * We have more public APIs to maintain. * Applications which uses UTF-8 or UTF-32 can not use these APIs anyway. * Other Python implementations may not have builtin codec for UCS-4. * If we change the Unicode internal representation to UTF-8, we need to keep UCS-4 support only for these APIs. Replace ``Py_UNICODE*`` with ``wchar_t*`` ----------------------------------------- We can replace ``Py_UNICODE`` to ``wchar_t``. Pros: * We have more consistent API set. * Backward compatible. Cons: * We have more public APIs to maintain. * They are inefficient on platforms ``wchar_t*`` is UTF-16. It is because built-in codecs supports only UCS-1, UCS-2, and UCS-4 input. Rejected ideas ============== Using runtime warning --------------------- These APIs doesn't release GIL for now. Emitting a warning from such APIs is not safe. See this example. .. code-block:: PyObject *u = PyList_GET_ITEM(list, i); // u is borrowed reference. PyObject *b = PyUnicode_EncodeUTF8(PyUnicode_AS_UNICODE(u), PyUnicode_GET_SIZE(u), NULL); // Assumes u is still living reference. PyObject *t = PyTuple_Pack(2, u, b); Py_DECREF(b); return t; If we emit Python warning from ``PyUnicode_EncodeUTF8()``, warning filters and other threads may change the ``list`` and ``u`` can be a dangling reference after ``PyUnicode_EncodeUTF8()`` returned. Additionally, since we are not changing behavior but removing C APIs, runtime ``DeprecationWarning`` might not helpful for Python developers. We should warn to extension developers instead. Discussions =========== * `Plan to remove Py_UNICODE APis except PEP 623 <https://mail.python.org/archives/list/python-dev@python.org/thread/S7KW2U6I…>`_ * `bpo-41123: Remove Py_UNICODE APIs except PEP 623: <https://bugs.python.org/issue41123>`_ References ========== .. [1] Source package list chosen from top 4000 PyPI packages. (https://github.com/methane/notes/blob/master/2020/wchar-cache/package_list.…) .. [2] pyodbc -- Don't use PyUnicode_Encode API #792 (https://github.com/mkleehammer/pyodbc/pull/792) .. [3] Uncomment Py_DEPRECATED for Py_UNICODE APIs (GH-21318) (https://github.com/python/cpython/commit/9c3840870814493fed62e140cfa43c2883…) Copyright ========= This document has been placed in the public domain. -- Inada Naoki <songofacandy(a)gmail.com>

9 27