Mailman 3 March 2018 - Python-Dev

Re: [Python-Dev] Move ensurepip blobs to external place
by Ned Deily 25 Mar '18

25 Mar '18

On Mar 24, 2018, at 16:13, Steve Dower <steve.dower(a)python.org> wrote: > Or we could just pull the right version directly from PyPI? (Note that updating the version should be an explicit step, as it is today, but the file should be identical to what’s on PyPI, right? And a urlretrieve is easier than pulling from a git repo.) I think the primary original rationale for having the pip wheel and its dependencies checked into the cpython repo was so that users would be able to install pip even if they did not have an Internet connection. But perhaps that requirement can be relaxed a bit if we say that the necessary wheels are vendored into all of our downloadable release items, that is, included in the packaging of source release files (the various tarballs) and the Windows and macOS binary installers. The main change would likely be making ensurepip a bit smarter to download if the bundled wheels are not present in the source directory. Assuming that people building from a cpython repo need to have a network connection if they want to run ensurepip, at least for the first time, is probably not an onerous requirement. -- Ned Deily nad(a)python.org -- []

3 2

Better support for consuming vendored packages
by Gregory Szorc 24 Mar '18

24 Mar '18

I'd like to start a discussion around practices for vendoring package dependencies. I'm not sure python-dev is the appropriate venue for this discussion. If not, please point me to one and I'll gladly take it there. I'll start with a problem statement. Not all consumers of Python packages wish to consume Python packages in the common `pip install <package>` + `import <package>` manner. Some Python applications may wish to vendor Python package dependencies such that known compatible versions are always available. For example, a Python application targeting a general audience may not wish to expose the existence of Python nor want its users to be concerned about Python packaging. This is good for the application because it reduces complexity and the surface area of things that can go wrong. But at the same time, Python applications need to be aware that the Python environment may contain more than just the Python standard library and whatever Python packages are provided by that application. If using the system Python executable, other system packages may have installed Python packages in the system site-packages and those packages would be visible to your application. A user could `pip install` a package and that would be in the Python environment used by your application. In short, unless your application distributes its own copy of Python, all bets are off with regards to what packages are installed. (And even then advanced users could muck with the bundled Python, but let's ignore that edge case.) In short, `import X` is often the wild west. For applications that want to "just work" without requiring end users to manage Python packages, `import X` is dangerous because `X` could come from anywhere and be anything - possibly even a separate code base providing the same package name! Since Python applications may not want to burden users with Python packaging, they may vendor Python package dependencies such that a known compatible version is always available. In most cases, a Python application can insert itself into `sys.path` to ensure its copies of packages are picked up first. This works a lot of the time. But the strategy can fall apart. Some Python applications support loading plugins or extensions. When user-provided code can be executed, that code could have dependencies on additional Python packages. Or that custom code could perform `sys.path` modifications to provide its own package dependencies. What this means is that `import X` from the perspective of the main application becomes dangerous again. You want to pick up the packages that you provided. But you just aren't sure that those packages will actually be picked up. And to complicate matters even more, an extension may wish to use a *different* version of a package from what you distribute. e.g. they may want to adopt the latest version that you haven't ported to you or they may want to use an old versions because they haven't ported yet. So now you have the requirements that multiple versions of packages be available. In Python's shared module namespace, that means having separate package names. A partial solution to this quagmire is using relative - not absolute - imports. e.g. say you have a package named "knights." It has a dependency on a 3rd party package named "shrubbery." Let's assume you distribute your application with a copy of "shrubbery" which is installed at some packages root, alongside "knights:" / /knights/__init__.py /knights/ni.py /shrubbery/__init__.py If from `knights.ni` you `import shrubbery`, you /could/ get the copy of "shrubbery" distributed by your application. Or you could pick up some other random copy that is also installed somewhere in `sys.path`. Whereas if you vendor "shrubbery" into your package. e.g. / /knights/__init__.py /knights/ni.py /knights/vendored/__init__.py /knights/vendored/shrubbery/__init__.py Then from `knights.ni` you `from .vendored import shrubbery`, you are *guaranteed* to get your local copy of the "shrubbery" package. This reliable behavior is highly desired by Python applications. But there are problems. What we've done is effectively rename the "shrubbery" package to "knights.vendored.shrubbery." If a module inside that package attempts an `import shrubbery.x`, this could fail because "shrubbery" is no longer the package name. Or worse, it could pick up a separate copy of "shrubbery" somewhere else in `sys.path` and you could have a Frankenstein package pulling its code from multiple installs. So for this to work, all package-local imports must be using relative imports. e.g. `from . import x`. The takeaway is that packages using relative imports for their own modules are much more flexible and therefore friendly to downstream consumers that may wish to vendor them under different names. Packages using relative imports can be dropped in and used, often without source modifications. This is a big deal, as downstream consumers don't want to be modifying/forking packages they don't maintain. Because of the advantages of relative imports, *I've individually reached the conclusion that relative imports within packages should be considered a best practice.* I would encourage the Python community to discuss adopting that practice more formally (perhaps as a PEP or something). But package-local relative imports aren't a cure-all. There is a major problem with nested dependencies. e.g. if "shrubbery" depends on the "herring" package. There's no reasonable way of telling "shrubbery" that "herring" is actually provided by "knights.vendored." You might be tempted to convert non package-local imports to relative. e.g. `from .. import herring`. But the importer doesn't allow relative imports outside the current top-level package and this would break classic installs where "shrubbery" and "herring" are proper top-level packages and not sub-packages in e.g. a "vendored" sub-package. For cases where this occurs, the easiest recourse today is to rewrite imported source code to use relative imports. That's annoying, but it works. In summary, some Python applications may want to vendor and distribute Python package dependencies. Reliance on absolute imports is dangerous because the global Python environment is effectively undefined from the perspective of the application. The safest thing to do is use relative imports from within the application. But because many packages don't use relative imports themselves, vendoring a package can require rewriting source code so imports are relative. And even if relative imports are used within that package, relative imports can't be used for other top-level packages. So source code rewriting is required to handle these. If you vendor your Python package dependencies, your world often consists of a lot of pain. It's better to absorb that pain than inflict it on the end-users of your application (who shouldn't need to care about Python packaging). But this is a pain that Python application developers must deal with. And I feel that pain undermines the health of the Python ecosystem because it makes Python a less attractive platform for standalone applications. I would very much welcome a discussion and any ideas on improving the Python package dependency problem for standalone Python applications. I think encouraging the use of relative imports within packages is a solid first step. But it obviously isn't a complete solution. Gregory

6 9

Re: [Python-Dev] ttk.Treeview.insert() does not allow to insert item with iid=0
by Igor Yakovchenko 24 Mar '18

24 Mar '18

I had opened a tracker issue <https://bugs.python.org/issue33096>. -- Igor Yakovchenko <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campai…> Без вирусов. www.avast.ru <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campai…> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

1 0

IMPORTANT - 3.7.0b3 cutoff / 3.7.0 ABI freeze
by Ned Deily 23 Mar '18

23 Mar '18

Just a reminder that 3.7.0b3 is almost upon us. Please get your feature fixes, bug fixes, and documentation updates in before 2018-03-26 ~23:59 Anywhere on Earth (UTC-12:00). That's a little over 3.5 days from now. IMPORTANT: We are now entering the final phases of 3.7.0. After the tagging for 3.7.0b3, the intention is that the ABI for 3.7.0 is frozen. After next week's 3.7.0b3, there will only be two more opportunities planned for changes prior to 3.7.0 final: - 2018-04-30 3.7.0 beta 4 - 2018-05-31 3.7.0 release candidate As I've noted in previous communications, we need to start locking down 3.7.0 so that our downstream users, that is, third-party package developers, Python distributors, and end users, can test their code with confidence that the actual release of 3.7.0 will hold no unpleasant surprises. So after 3.7.0b3, you should treat the 3.7 branch as if it is already released and in maintenance mode. That means you should only push the kinds of changes that are appropriate for a maintenance release: non-ABI-changing bug and feature fixes and documentation updates. If you find a problem that requires an ABI-altering or other significant user-facing change (for example, something likely to introduce an incompatibility with existing users' code or require rebuilding of user extension modules), please make sure to set the b.p.o issue to "release blocker" priority and describe there why you feel the change is necessary. If you are reviewing PRs for 3.7 (and please do!), be on the lookout for and flag potential incompatibilities (we've all made them). Thanks again for all of your hard work towards making 3.7.0 yet another great release! --Ned -- Ned Deily nad(a)python.org -- []

1 0

Summary of Python tracker Issues
by Python tracker 23 Mar '18

23 Mar '18

ACTIVITY SUMMARY (2018-03-16 - 2018-03-23) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 6528 ( +3) closed 38349 (+37) total 44877 (+40) Open issues with patches: 2542 Issues opened (30) ================== #33087: No reliable clean shutdown method https://bugs.python.org/issue33087 opened by Void2258 #33088: Cannot pass a SyncManager proxy to a multiprocessing subproces https://bugs.python.org/issue33088 opened by jjdmon #33089: Add multi-dimensional Euclidean distance function to the math https://bugs.python.org/issue33089 opened by rhettinger #33090: race condition between send and recv in _ssl with non-zero tim https://bugs.python.org/issue33090 opened by nneonneo #33091: ssl.SSLError: Invalid error code (_ssl.c:2217) https://bugs.python.org/issue33091 opened by devkid #33092: The bytecode for f-string formatting is inefficient. https://bugs.python.org/issue33092 opened by Mark.Shannon #33093: Fatal error on SSL transport https://bugs.python.org/issue33093 opened by Eric Toombs #33095: Cross-reference isolated mode from relevant locations https://bugs.python.org/issue33095 opened by ncoghlan #33096: ttk.Treeview.insert() does not allow to insert item with "Fals https://bugs.python.org/issue33096 opened by igor.yakovchenko #33097: concurrent futures Executors accept tasks after interpreter sh https://bugs.python.org/issue33097 opened by mrknmc #33099: test_poplib hangs with the changes done in PR https://bugs.python.org/issue33099 opened by jayyin11043 #33102: get the nth folder of a given path https://bugs.python.org/issue33102 opened by amjad ben hedhili #33105: os.path.isfile returns false on Windows when file path is long https://bugs.python.org/issue33105 opened by ldconejo #33106: Deleting a key in a read-only gdbm results in KeyError, not gd https://bugs.python.org/issue33106 opened by sam-s #33109: argparse: make new 'required' argument to add_subparsers defau https://bugs.python.org/issue33109 opened by wolma #33110: Adding a done callback to a concurrent.futures Future once it https://bugs.python.org/issue33110 opened by samm #33111: Merely importing tkinter breaks parallel code (multiprocessing https://bugs.python.org/issue33111 opened by ezwelty #33113: Query performance is very low and can even lead to denial of s https://bugs.python.org/issue33113 opened by ghi5107 #33114: random.sample() behavior is unexpected/unclear from docs https://bugs.python.org/issue33114 opened by Scott Eilerman #33115: Asyncio loop blocks with a lot of parallel tasks https://bugs.python.org/issue33115 opened by decaz #33117: asyncio example uses non-existing/documented method https://bugs.python.org/issue33117 opened by hfingler #33118: No clean way to get notified when a Transport's write buffer e https://bugs.python.org/issue33118 opened by vitaly.krug #33119: python sys.argv argument parsing not clear https://bugs.python.org/issue33119 opened by Jonathan Huot #33120: infinite loop in inspect.unwrap(unittest.mock.call) https://bugs.python.org/issue33120 opened by peterdemin #33121: recv returning 0 on closed connection not documented https://bugs.python.org/issue33121 opened by joders #33122: ftplib: FTP_TLS seems to have problems with sites that close t https://bugs.python.org/issue33122 opened by jottbe #33123: Path.unlink should have a missing_ok parameter https://bugs.python.org/issue33123 opened by rbu #33124: Lazy execution of module bytecode https://bugs.python.org/issue33124 opened by nascheme #33125: Windows 10 ARM64 platform support https://bugs.python.org/issue33125 opened by Steven Noonan #33126: Some C buffer protocol APIs not documented https://bugs.python.org/issue33126 opened by pitrou Most recent 15 issues with no replies (15) ========================================== #33126: Some C buffer protocol APIs not documented https://bugs.python.org/issue33126 #33124: Lazy execution of module bytecode https://bugs.python.org/issue33124 #33123: Path.unlink should have a missing_ok parameter https://bugs.python.org/issue33123 #33122: ftplib: FTP_TLS seems to have problems with sites that close t https://bugs.python.org/issue33122 #33121: recv returning 0 on closed connection not documented https://bugs.python.org/issue33121 #33119: python sys.argv argument parsing not clear https://bugs.python.org/issue33119 #33117: asyncio example uses non-existing/documented method https://bugs.python.org/issue33117 #33113: Query performance is very low and can even lead to denial of s https://bugs.python.org/issue33113 #33110: Adding a done callback to a concurrent.futures Future once it https://bugs.python.org/issue33110 #33099: test_poplib hangs with the changes done in PR https://bugs.python.org/issue33099 #33096: ttk.Treeview.insert() does not allow to insert item with "Fals https://bugs.python.org/issue33096 #33095: Cross-reference isolated mode from relevant locations https://bugs.python.org/issue33095 #33090: race condition between send and recv in _ssl with non-zero tim https://bugs.python.org/issue33090 #33088: Cannot pass a SyncManager proxy to a multiprocessing subproces https://bugs.python.org/issue33088 #33085: *** Error in `python': double free or corruption (out): 0x0000 https://bugs.python.org/issue33085 Most recent 15 issues waiting for review (15) ============================================= #33124: Lazy execution of module bytecode https://bugs.python.org/issue33124 #33123: Path.unlink should have a missing_ok parameter https://bugs.python.org/issue33123 #33097: concurrent futures Executors accept tasks after interpreter sh https://bugs.python.org/issue33097 #33092: The bytecode for f-string formatting is inefficient. https://bugs.python.org/issue33092 #33083: math.factorial accepts non-integral Decimal instances https://bugs.python.org/issue33083 #33082: multiprocessing docs bury very important 'callback=' args https://bugs.python.org/issue33082 #33070: Add platform triplet for RISC-V https://bugs.python.org/issue33070 #33063: failed to build _ctypes: undefined reference to `ffi_closure_F https://bugs.python.org/issue33063 #33061: NoReturn missing from __all__ in typing.py https://bugs.python.org/issue33061 #33058: Enhance Python's Memory Instrumentation with COUNT_ALLOCS https://bugs.python.org/issue33058 #33057: logging.Manager.logRecordFactory is never used https://bugs.python.org/issue33057 #33051: IDLE: Create new tab for editor options in configdialog https://bugs.python.org/issue33051 #33048: macOS job broken on Travis CI https://bugs.python.org/issue33048 #33042: New 3.7 startup sequence crashes PyInstaller https://bugs.python.org/issue33042 #33038: GzipFile doesn't always ignore None as filename https://bugs.python.org/issue33038 Top 10 most discussed issues (10) ================================= #33053: Avoid adding an empty directory to sys.path when running a mod https://bugs.python.org/issue33053 18 msgs #33089: Add multi-dimensional Euclidean distance function to the math https://bugs.python.org/issue33089 12 msgs #33118: No clean way to get notified when a Transport's write buffer e https://bugs.python.org/issue33118 9 msgs #6083: Reference counting bug in PyArg_ParseTuple and PyArg_ParseTupl https://bugs.python.org/issue6083 7 msgs #33115: Asyncio loop blocks with a lot of parallel tasks https://bugs.python.org/issue33115 7 msgs #25345: Unable to install Python 3.5 on Windows 10 https://bugs.python.org/issue25345 5 msgs #32949: Simplify "with"-related opcodes https://bugs.python.org/issue32949 5 msgs #33109: argparse: make new 'required' argument to add_subparsers defau https://bugs.python.org/issue33109 5 msgs #33111: Merely importing tkinter breaks parallel code (multiprocessing https://bugs.python.org/issue33111 5 msgs #31550: Inconsistent error message for TypeError with subscripting https://bugs.python.org/issue31550 4 msgs Issues closed (36) ================== #18802: ipaddress documentation errors https://bugs.python.org/issue18802 closed by xiang.zhang #23203: Aliasing import of sub-{module,package} from the package raise https://bugs.python.org/issue23203 closed by ncoghlan #25836: Documentation of MAKE_FUNCTION/MAKE_CLOSURE_EXTENDED_ARG is mi https://bugs.python.org/issue25836 closed by serhiy.storchaka #27683: ipaddress subnet slicing iterator malfunction https://bugs.python.org/issue27683 closed by xiang.zhang #28247: Add an option to zipapp to produce a Windows executable https://bugs.python.org/issue28247 closed by csabella #31639: http.server and SimpleHTTPServer hang after a few requests https://bugs.python.org/issue31639 closed by mdk #32056: Improve exceptions in aifc, sunau and wave https://bugs.python.org/issue32056 closed by serhiy.storchaka #32421: Keeping an exception in cache can segfault the interpreter https://bugs.python.org/issue32421 closed by zunger #32489: Allow 'continue' in 'finally' clause https://bugs.python.org/issue32489 closed by serhiy.storchaka #32505: dataclasses: make field() with no annotation an error https://bugs.python.org/issue32505 closed by eric.smith #32506: dataclasses: no need for OrderedDict now that dict guarantees https://bugs.python.org/issue32506 closed by eric.smith #32829: Lib/ be more pythonic https://bugs.python.org/issue32829 closed by r.david.murray #32838: Fix Python versions in the table of magic numbers https://bugs.python.org/issue32838 closed by serhiy.storchaka #32896: Error when subclassing a dataclass with a field that uses a de https://bugs.python.org/issue32896 closed by eric.smith #32953: Dataclasses: frozen should not be inherited for non-dataclass https://bugs.python.org/issue32953 closed by eric.smith #32960: dataclasses: disallow inheritance between frozen and non-froze https://bugs.python.org/issue32960 closed by eric.smith #33018: Improve issubclass() error checking and message https://bugs.python.org/issue33018 closed by levkivskyi #33027: handling filename encoding in Content-Disposition by cgi.Field https://bugs.python.org/issue33027 closed by pawciobiel #33034: urllib.parse.urlparse and urlsplit not raising ValueError for https://bugs.python.org/issue33034 closed by berker.peksag #33041: Issues with "async for" https://bugs.python.org/issue33041 closed by serhiy.storchaka #33044: pdb from base class, get inside a method of derived class https://bugs.python.org/issue33044 closed by terry.reedy #33049: itertools.count() confusingly mentions zip() and sequence numb https://bugs.python.org/issue33049 closed by rhettinger #33069: Maintainer information discarded when writing PKG-INFO https://bugs.python.org/issue33069 closed by ncoghlan #33078: Queue with maxsize can lead to deadlocks https://bugs.python.org/issue33078 closed by pitrou #33081: multiprocessing Queue leaks a file descriptor associated with https://bugs.python.org/issue33081 closed by pitrou #33086: pip: IndexError https://bugs.python.org/issue33086 closed by eric.smith #33094: dataclasses: ClassVar attributes are not working properly https://bugs.python.org/issue33094 closed by eric.smith #33098: add implicit conversion for random.choice() on a dict https://bugs.python.org/issue33098 closed by tim.peters #33100: dataclasses and __slots__ - non-default argument (member_descr https://bugs.python.org/issue33100 closed by eric.smith #33101: Possible name inversion in heapq implementation https://bugs.python.org/issue33101 closed by rhettinger #33103: Syntax to get multiple arbitrary items from an sequence https://bugs.python.org/issue33103 closed by serhiy.storchaka #33104: Documentation for EXTENDED_ARG in dis module is incorrect for https://bugs.python.org/issue33104 closed by Eric Appelt #33107: Feature request: more typing.SupportsXXX https://bugs.python.org/issue33107 closed by gvanrossum #33108: Unicode char 304 in lowercase has len = 2 https://bugs.python.org/issue33108 closed by serhiy.storchaka #33112: SequenceMatcher bug https://bugs.python.org/issue33112 closed by tim.peters #33116: Field is not exposed in dataclasses.__all__ https://bugs.python.org/issue33116 closed by eric.smith

1 0

PEP 541 - Accepted
by Mark Mangoba 23 Mar '18

23 Mar '18

Hi All, As the BDFL-Delegate, I’m happy to announce PEP 541 has been accepted. PEP 541 has been voted by the packaging-wg (https://wiki.python.org/psf/ PackagingWG/Charter): - Donald Stufft - Dustin Ingram - Ernest W. Durbin III - Ewa Jodlowska - Kenneth Reitz - Mark Mangoba - Nathaniel J. Smith - Nick Coghlan - Nicole Harris - Sumana Harihareswara Thank you to the packaging-wg and to everyone that has contributed to PEP 541. Best regards, Mark -- Mark Mangoba | PSF IT Manager | Python Software Foundation | mmangoba(a)python.org | python.org | Infrastructure Staff: infrastructure-staff(a)python.org | GPG: 2DE4 D92B 739C 649B EBB8 CCF6 DC05 E024 5F4C A0D1

2 1

Deprecating float.is_integer()
by Serhiy Storchaka 23 Mar '18

23 Mar '18

I searched usages of is_integer() on GitHub and have found that it is used *only* in silly code like (x/5).is_integer(), (x**0.5).is_integer() (or even (x**(1/3)).is_integer()) and in loops like: i = 0 while i < 20: if i.is_integer(): print(i) i += 0.1 (x/5).is_integer() is an awful way of determining the divisibility by 5. It returns wrong result for large integers and some floats. (x % 5 == 0) is a more clear and reliable way (or PEP 8 compliant (not x % 5)). Does anybody know examples of the correct use of float.is_integer() in real programs? For now it looks just like a bug magnet. I suggest to deprecate it in 3.7 or 3.8 and remove in 3.9 or 3.10. If you even need to test if a float is an exact integer, you could use (not x % 1.0). It is even faster than x.is_integer().

11 25

Do we have vlookup function
by Rohit Adhikari 22 Mar '18

22 Mar '18

Do we have vlookup function which can be used in dataframe same as used in excel. Can you please provide the pointer for the same? Thanks!!!

2 1

ThreadedProcessPoolExecutor
by Roberto Martínez 21 Mar '18

21 Mar '18

Hi, I've made a custom concurrent.futures.Executor mixing the ProcessPoolExecutor and ThreadPoolExecutor. I've published it here: https://github.com/nilp0inter/threadedprocess This executor is very similar to a ProcessPoolExecutor, but each process in the pool have it's own ThreadPoolExecutor inside. The motivation for this executor is mitigate the problem we have in a project were we have a very large number of long running IO bounded tasks, that have to run concurrently. Those long running tasks have sparse CPU bounded operations. To resolve this problem I considered multiple solutions: 1. Use asyncio to run the IO part as tasks and use a ProcessPoolExecutor to run the CPU bounded operations with "run_in_executor". Unfortunately the CPU operations depends on a large memory context, and using a ProcessPoolExecutor this way force the parent process to picklelize all the context to send it to the task, and because the context is so large, this operation is itself very CPU demanding. So it doesn't work. 2. Executing the IO/CPU bounded operations in different processes with multiprocessing.Process. This actually works, but the number of idle processes in the system is too large, resulting in a bad memory footprint. 3. Executing the IO/CPU bounded operations in threads. This doesn't work because the sum of all CPU operations saturate the core where the Python process is running and the other cores are wasted doing nothing. So I coded the ThreadedProcessPoolExecutor that helped me maintaining the number of processes under control (I just have one process per CPU core) allowing me to have a very high concurrency (hundreds of threads per process). I have a couple of questions: The first one is about the license. Given that I copied the majority of the code from the concurrent.futures library, I understand that I have to publish the code under the PSF LICENSE. Is this correct? My second question is about the package namespace. Given that this is an concurrent.futures.Executor subclass I understand that more intuitive place to locate it is under concurrent.futures. Is this a suitable use case for namespace packages? Is this a good idea? Best regards, Roberto

2 2

What is the purpose of NEXT_BLOCK()?
by Serhiy Storchaka 21 Mar '18

21 Mar '18

There is the NEXT_BLOCK() macro in compile.c. It creates a new block, creates an implicit jump from the current block to the new block, and sets it as the current block. But why it is used? All seems working if remove NEXT_BLOCK(). If there was a need of NEXT_BLOCK() (if it reduces the computational complexity of compilation or allows some optimizations), it should be documented, and we should analyze the code and add missed NEXT_BLOCK() where they are needed, and perhaps add new tests. Otherwise it can be removed.

2 2