From ncoghlan at gmail.com Wed Nov 1 00:55:00 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 1 Nov 2017 14:55:00 +1000 Subject: [Python-ideas] Defining an easily installable "Recommended baseline package set" In-Reply-To: References: <13e5b1bc-7f93-c530-35a8-cbe4651c5975@mrabarnett.plus.com> <5d0d0c90-0919-f248-60eb-2be208514d99@mrabarnett.plus.com> Message-ID: On 1 November 2017 at 05:56, Guido van Rossum wrote: > On Tue, Oct 31, 2017 at 12:24 PM, MRAB wrote: > >> At least Python 3.6 is only 1 year/release behind, which is fine! >> > > OK, so presumably that argument doesn't preclude inclusion in the 3.7 (or > later) stdlib. I'm beginning to warm up to the idea again... Maybe we > should just bite the bullet. Nick, what do you think? Is it worth a small > PEP? > I'm personally still in favour of swapping out the current _sre based implementation for a _regex based implementation (such that 3.7+ still only contained a single regex engine, and the stdlib re module and a PyPI regex backport module could still happily coexist), and a PEP + draft patch would be the way to do that. The framing of a PEP for that approach would be "Replace the regex engine backing the re module" rather than "Add regex to the standard library". The existing engine could then potentially be spun out as a new "sre" project on PyPI, such that folks that *were* depending on _sre internals had access to an upgrade path that didn't necessarily require them porting their code to a new regex engine (the PEP author wouldn't have to commit to doing that work - we'd just ask the PyPI admins to reserve the name in case there proved to be demand for such a project). However, I'm also not one of the folks that would be on the hook for handling any compatibility regressions that were subsequently reported against the 3.7 re module, so I'd also take my +1 with a rather large grain of salt - it's easy to be positive about a plan when the potential downsides don't affect me personally :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Wed Nov 1 01:01:09 2017 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 31 Oct 2017 22:01:09 -0700 Subject: [Python-ideas] Defining an easily installable "Recommended baseline package set" In-Reply-To: References: Message-ID: On Oct 31, 2017 4:42 AM, "Nick Coghlan" wrote: On 31 October 2017 at 02:29, Guido van Rossum wrote: > What's your proposed process to arrive at the list of recommended packages? > I'm thinking it makes the most sense to treat inclusion in the recommended packages list as a possible outcome of proposals for standard library inclusion, rather than being something we'd provide a way to propose specifically. We'd only use it in cases where a proposal would otherwise meet the criteria for stdlib inclusion, but the logistics of actually doing so don't work for some reason. Running the initial 5 proposals through that filter: * six: a cross-version compatibility layer clearly needs to be outside the standard library * setuptools: we want to update this in line with the PyPA interop specs, not the Python language version * cffi: updates may be needed for PyPA interop specs, Python implementation updates or C language definition updates * requests: updates are more likely to be driven by changes in network protocols and client platform APIs than Python language changes * regex: we don't want two regex engines in the stdlib, transparently replacing _sre would be difficult, and _sre is still good enough for most purposes Some other packages that might meet these criteria, or at least be useful for honing them: - lxml - numpy - cryptography - idna -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Nov 1 01:25:38 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 1 Nov 2017 15:25:38 +1000 Subject: [Python-ideas] install pip packages from Python prompt In-Reply-To: References: <01f701d350eb$c4ee6210$4ecb2630$@sdamon.com> <021601d350ee$88f236d0$9ad6a470$@sdamon.com> <045101d351b9$54a25240$fde6f6c0$@sdamon.com> <4409775187778125080@unknownmsgid> Message-ID: On 1 November 2017 at 08:50, Terry Reedy wrote: > In April 2016, after posting the idea to pydev list and getting 'go > ahead's from Nick Coughlin and someone else, with no negatives, I approved > Upendra Kumar's GSOC proposal to write a pip gui. This was > https://bugs.python.org/issue27051. On June 20, Ned Deily and Nick > Coughlin vetoed adding a pip gui anywhere in the stdlib since it depended > on something not in the stdlib, and perhaps for other reasons I don't fully > understand. > Clarifying the objection here (since the linked issue is a fairly long one): what I'm against is tightly coupling the pip-gui development & release process to the CPython development & release process when we don't have any compelling reason to do so. While PEP 434 provides a workaround that helps to keep IDLE consistent across different support branches, it still isn't as robust a development model as folks being able to install an application and try it out on existing Python versions *before* we bundle it with a CPython release. https://packaging.python.org/tutorials/distributing-packages/ provides a general guide on how to publish new packages, and the combination of tox and Travis CI makes it reasonably straightforward to run pre-merge CI testing across multiple Python versions. Given an independently released pip-gui on PyPI (with its own version numbering and release cadence), then I'd be +1 on bundling that as an optional IDLE addon, ensurepip style. Such a project could also lay the foundation for switching IDLE itself to a similar bundling model, which would allow the need for PEP 434 to be eliminated as well (since bundled applications are already permitted to add new features in maintenance releases). Cheers, Nick. P.S. I'll also note that a useful feature Travis CI offers is the ability to automate PyPI releases: https://docs.travis-ci.com/user/deployment/pypi/ -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Wed Nov 1 01:46:25 2017 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 1 Nov 2017 01:46:25 -0400 Subject: [Python-ideas] install pip packages from Python prompt In-Reply-To: References: <01f701d350eb$c4ee6210$4ecb2630$@sdamon.com> <021601d350ee$88f236d0$9ad6a470$@sdamon.com> <045101d351b9$54a25240$fde6f6c0$@sdamon.com> Message-ID: On 10/31/2017 10:41 AM, Nick Coghlan wrote: > On 31 October 2017 at 05:57, Alex Walters > > wrote: > > > While I completely agree with this in principle, I think you > > overestimate the average beginner. The average Python beginner may have a decade or more experience interacting with computers by voice, gestures, fingers on a screen, mouse clicks and movements, joystick controller clicks and movements, and single key presses. This includes using a word processor. They may or may not have consciously learned (or remember) anything about natural language grammar. Python may well be their first experience with a formal language grammar and a picky interpreter. > Nope.? I totally get that they don?t know what a shell or command > prompt is. A shell is a program in which users issue instructions one at a time as sentences in a formal language and wait for the response before issuing another. >? THEY. NEED. TO. LEARN. Yes. But what shell should people who want to learn Python learn first? Interactive python, which comes with a 'using' manual besides a tutorial and two reference manuals? Or, on Windows, Command Prompt, which runs a sparsely documented unnamed language, has no tutorial, and has mostly been replaced by File Explorer? One can just as well or better learn the idea of a REPL with formal language input from Python first. Once one has learned interactive Python, learning to use other shells should be straightforward. It should not be too hard to convey that a different language with a different syntax and different built-in commands requires a different interpreter. > Hiding it is not a good idea for anyone. Tell that to Microsoft, which in Win 10 removed command prompt and other 'expert' tools from the regular start menu. > We're not in the business of making judgements about who should and > shouldn't become Python programmers - we're in the business of making > sure that Python is accessible to as many people as possible by removing > irrelevant barriers to adoption, whether that's translating > documentation so folks can start learning with instructions in their > native language, or making it possible for them to defer learning the > idiosyncrasies of the Windows, Linux, and Mac OS X command line > environments. My daughter learned Python and IDLE (from me) about 5 years before she had to learn about Command Prompt. (In between, she had courses that used the Racket and Jave/Eclipse IDEs.) The notion that she should have been forced to learn an inferior and useless (to her) shell first is to me absurd. > On the latter front, the details of the development interfaces offered > by traditional desktop operating systems may *never* be relevant to the > new generation of folks coming through that are learning to program by > manipulating remote coding environments on tablets and other app-centric > devices, just as most developers nowadays are able to get by without > even learning C, let alone any flavour of assembly language. Our role in > this process isn't to create future generations that think and work in > exactly the same ways we do, it's to enable them to discover new ways of > working that build on top of whatever we create. Amen. At least 99+% of my interaction with the hg repository and my clones thereof was through TortoiseHg and its Workbench gui. > That means I now see a few potential RFEs from this thread: > > 1. An import system feature that allows a running Python program to > report a timestamp (with the same granularity as pyc file timestamp > headers) for *when* the currently loaded modules were last modified. > This could be as simple as a new `__mtime__`? attribute in each module > to store that number. > 2. A new importlib.util API to check for potentially out of date modules > in sys.modules (those where a freshly calculated module mtime doesn't > match the stored __mtime__ attribute) > 3. Support in IDLE for Jupyter-style "!" commands I believe that this was suggested on idle-dev and rejected by Guido as an expert feature, or perhaps as abbreviating and thereby encouraging an expert usage. I will let him say what he thinks now. I have considered adding 'Run OS Command' to the Run menu, but I mostly do not see the point for a Python shell or REPL. If one knows what to type after '!', one can just as well click the appropriate icon and enter the command there. Or use os.system or subprocess. What am I missing? > 4. Having IDLE call that importlib API and warn about any stale modules > after each command line operation This seems heavy handed, usually unnecessary, but also inadequate. One can now manipulate site-packages while running a Python shell in ways a shell will not know about. One should then either start the shell manually or take the consequences. As it is, IDLE restarts the user process each time one runs code from an editor. Or one can explicitly restart the user process, without exiting IDLE, after installing or updating modules. One objection to the latter has been that restarting, even on IDLE, clears more than one might want. I am working on a separate proposal that would make it easier to regenerate a workspace. ***** Hardly mentioned in this thread is that command line apps in general and pip in particular are not easily usable by many people. One needs to use a particular app regularly or have an unusually good memory for fiddy details or a high tolerance for reading and ability to understand reading -h output. That is why I think a pip gui would greatly help. If nothing that directly goes in the stdlib, other than ensurepip, can depend on pip and its api, a helper module could be installed from pypi along with pip (by ensurepip!). A pip wrapper, gui or not, could alleviate the following problem using pip in Windows. There are up to date Windows binaries on Christopher Gohlke's site, https://www.lfd.uci.edu/~gohlke/pythonlibs/. It currently has 32- and 64-bit builds for 2.7, 3.4, 3.5, and 3.6, as appropriate, for perhaps 450 packages. At least some are source-only on pypi. The site is not a pip package index, so the wheels must be downloaded first before being installed by pip. As for the subject of this thread: I intended that the pip gui that you and Ned Deily vetoed should, unlike pip itself, be runnable from python with "import pipgui; pip.main()". An import module can detect whether is it being run directly with Python or IDLE and, if sys.modules is modified, adjust its exit message or action accordingly. The main problem with pip that I have seen on Stackoverflow is that people run pip on a command line with a binary different from the one that they run IDLE with, and then post 'X imports Y, IDLE does not'. The easiest way to ensure that something is installed for the binary running IDLE is for IDLE to run something with its sys.executable. -- Terry Jan Reedy From rosuav at gmail.com Wed Nov 1 02:24:31 2017 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 1 Nov 2017 17:24:31 +1100 Subject: [Python-ideas] install pip packages from Python prompt In-Reply-To: References: <01f701d350eb$c4ee6210$4ecb2630$@sdamon.com> <021601d350ee$88f236d0$9ad6a470$@sdamon.com> <045101d351b9$54a25240$fde6f6c0$@sdamon.com> Message-ID: On Wed, Nov 1, 2017 at 4:46 PM, Terry Reedy wrote: >> Hiding it is not a good idea for anyone. > > Tell that to Microsoft, which in Win 10 removed command prompt and other > 'expert' tools from the regular start menu. ... but also added the Linux subsystem, making it rather more viable to recommend that people learn bash on all platforms. Not that bash is exactly a paragon of virtue and beauty, but it does at least have practicality and proven usefulness. I'd be all in favour of Windows users being pushed to IDLE rather than the vanilla REPL, and then taught that IDLE is the gateway to a bunch of other tools (including pip). But the shell is an important tool to learn too; I agree with Terry that it shouldn't be a prerequisite to learning Python, but I wouldn't be against it being the only way to do the more complicated tasks. For instance, maybe you can install packages using pip-gui, and you can list installed packages, but searching, uninstalling, etc are left to the command-line tool. Or whereever you draw the line. There's no point trying to put *everything* into the GUI. ChrisA From wes.turner at gmail.com Wed Nov 1 02:54:27 2017 From: wes.turner at gmail.com (Wes Turner) Date: Wed, 1 Nov 2017 02:54:27 -0400 Subject: [Python-ideas] install pip packages from Python prompt In-Reply-To: References: <01f701d350eb$c4ee6210$4ecb2630$@sdamon.com> <021601d350ee$88f236d0$9ad6a470$@sdamon.com> <045101d351b9$54a25240$fde6f6c0$@sdamon.com> <4409775187778125080@unknownmsgid> Message-ID: Suggestions to help to minimize unnecessary logged bandwidth use and even work with a closed loop LAN: This reads from the filesystem: import requests This would read from the PyPi service over the network bandwidth: #!pip install -U requests #%run pip install -U requests #pip('install -U requests') This doesn't work because you SHOULD restart the interpreter after running pip (because imports are cached): import requests !pip install -U requests import requests Some tips on running educational environments for beginners (optionally in a lab): - IPython - requests? # docstring - requests?? # source - !pydoc ipython - %run pydoc ipython - Spyder includes an IPython terminal (``conda install spyder`` even installs Qt) - Just use conda (and docker and binder) - Aggressively cache network accesses (and/or just block internet access entirely) - PIP_INDEX="https://pip.local/simple/" - DevPi can transparently proxy cache packages from pypi (and then work without internet access) - a local httpbin is easy to host within the LAN - Binder can provision Docker containers with conda and everything already installed (With Kubernetes and JupyterHub (and OAuth)) - Each person gets their own container - Jupyter has a web terminal built in - You can also just run the Docker container locally #!pip install ipython scipy devpi certbot # Some back of the napkin calculations: # 30 users run a script which runs 2 pip install requests at least 5 times. # (These should be at the top) These Docker containers are ready to go with everything you need for a lesson: https://github.com/jupyter/docker-stacks You don't need this bash shell lesson if all you need to do is ``!pip install`` with IPython: https://swcarpentry.github.io/shell-novice/ A Tk GUI for pip would need to frustratingly duplicate ~/.bash_history and up-arrow to get the previous command. (Qt and GTK are not consistently installed; though ``conda install spyder`` does install Qt (which is very accessible)) IDLE sucks. https://devpi.net/docs/devpi/devpi/stable/%2Bd/index.html https://github.com/kennethreitz/httpbin/blob/master/Dockerfile These should pretty much get you started: https://github.com/jrjohansson/scientific-python-lectures https://github.com/jrjohansson/scientific-python-lectures/blob/master/Lecture-0-Scientific-Computing-with-Python.ipynb https://github.com/jrjohansson/scientific-python-lectures/blob/master/Lecture-1-Introduction-to-Python-Programming.ipynb https://github.com/jrjohansson/scientific-python-lectures/blob/master/Lecture-7-Revision-Control-Software.ipynb GitLab and Mattermost work with an offline LAN: https://docs.gitlab.com/omnibus/README.html https://docs.gitlab.com/omnibus/docker/README.html#run-the-image Binder can pull images from a GitLab Docker Container Registry: https://docs.gitlab.com/ce/user/project/container_registry.html These suggestions should help to minimize unnecessary logged bandwidth use and even work with a closed loop LAN. See: "#!pip install ipython scipy devpi certbot" in the middle of this email. On Tuesday, October 31, 2017, Wes Turner wrote: > You could teach them subprocess and os command injection safety from the > start: > > ```python > import subprocess > import sys > cmd = [sys.executable, -m', 'pip', 'install', '-r', > 'psfblessed-requirements.txt']) > retcode = subprocess.check_call(cmd) > assert retcode == 0 > ``` > > (Because shell=True is dangerous and str.split is dangerous): > > ```python > filename = "'/etc/ passwd' ; shutdown -r now" > cmd = ("cat '%s'" % filename) > cmd > # "cat ''/etc/ passwd'' ; shutdown -r now" > > cmd.split() > # ["'", '/etc', 'passwd', ';', 'shutdown', '-r', 'now'] > > shlex.split(cmd) > # ['cmd', '', '/etc', 'passwd', ';', 'shutdown', '-r', 'now'] > ``` > > (Sarge solves for a number of additional cases beyond shlex.split (empty > string should be '', 'already-quoted commands') > > https://sarge.readthedocs.io/en/latest/overview.html#why- > not-just-use-subprocess > > https://en.wikipedia.org/wiki/Code_injection#Shell_injection > > https://sarge.readthedocs.io/en/latest/internals.html#how- > shell-quoting-works > > Of course, we're programmers and our input is not untrusted, so shell=True > without any string operations is not as dangerous. > > > On Tuesday, October 31, 2017, Wes Turner > wrote: > >> You could teach them subprocess and os command injection safety from the >> start: >> >> ```python >> import subprocess >> import sys >> cmd = [sys.executable, -m', 'pip', 'install', '-r', >> 'psfblessed-requirements.txt']) >> retcode = subprocess.check_call(cmd) >> assert retcode == 0 >> ``` >> >> (Because shell=True is dangerous) >> >> On Tuesday, October 31, 2017, Terry Reedy wrote: >> >>> On 10/31/2017 12:21 PM, Ivan Levkivskyi wrote: >>> >>>> I think it was proposed several times before, but I just wanted to >>>> revive the idea that we could add >>>> a GUI interface to install/update packages from IDLE (maybe even with >>>> some package browser). >>>> >>> >>> https://bugs.python.org/issue23551. I agreed with and still agree with >>> Raymond's opening message in Feb 2015: >>> "In teaching Python, I find that many Windows users are command-line >>> challenged and have difficulties using and accessing PIP. ... I would love >>> to be able to start a class with a fresh Python download from python.org >>> and effortlessly install requests and other tools without users having to >>> fire-up a terminal window and wrestle with the various parts." >>> >>> The one change I made in Raymond's proposal is that instead of having >>> multiple IDLE menu entries tied to multiple IDLE functions invoking >>> multiple pip functions, there would be one IDLE menu entry, perhaps 'Help >>> => Install packages' (plural intentional), that would invoke a standalone >>> tkinter based gui front-end to pip. 'Standalone' means no dependency on >>> IDLE code. I don't think every IDE or app should *have to* write its own >>> gui. Plus, a standalone tkinter module could be invoked from a command >>> line with 'python -m pipgui' or invoked from interactive python with >>> 'import pipgui; pipgui.main()'. >>> >>> In April 2016, after posting the idea to pydev list and getting 'go >>> ahead's from Nick Coughlin and someone else, with no negatives, I approved >>> Upendra Kumar's GSOC proposal to write a pip gui. This was >>> https://bugs.python.org/issue27051. On June 20, Ned Deily and Nick >>> Coughlin vetoed adding a pip gui anywhere in the stdlib since it depended >>> on something not in the stdlib, and perhaps for other reasons I don't fully >>> understand. >>> >>> Looking back, I can see that I made two mistakes. >>> >>> The first was proposing to use the public-looking pip.main after >>> importing pip. It is actually intended to be private (and should have been >>> named '_main' to make that clearer). As it turns out, the extra work of >>> accessing pip through the intended command line interface (via >>> subprocess) is necessary anyway since running pip makes changes to the >>> in-memory modules that are not reset when .main is called again. So it >>> might as well be used for every access. >>> >>> The second was not requiring an approved PEP before proceeding to actual >>> coding. >>> >>> -- >>> Terry Jan Reedy >>> >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Wed Nov 1 03:06:15 2017 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 1 Nov 2017 03:06:15 -0400 Subject: [Python-ideas] install pip packages from Python prompt In-Reply-To: References: <01f701d350eb$c4ee6210$4ecb2630$@sdamon.com> <021601d350ee$88f236d0$9ad6a470$@sdamon.com> <045101d351b9$54a25240$fde6f6c0$@sdamon.com> <4409775187778125080@unknownmsgid> Message-ID: On 11/1/2017 1:25 AM, Nick Coghlan wrote: > On 1 November 2017 at 08:50, Terry Reedy > > wrote: > > In April 2016, after posting the idea to pydev list and getting 'go > ahead's from Nick Coughlin and someone else, with no negatives, I > approved Upendra Kumar's GSOC proposal to write a pip gui.? This was > https://bugs.python.org/issue27051 > .? On June 20, Ned Deily and > Nick Coughlin vetoed adding a pip gui anywhere in the stdlib since > it depended on something not in the stdlib, and perhaps for other > reasons I don't fully understand. > > > Clarifying the objection here (since the linked issue is a fairly long > one): what I'm against is tightly coupling the pip-gui development & > release process to the CPython development & release process when we > don't have any compelling reason to do so. Thank you for clarifying. This is a reason that I did not understand ;-). > Given an independently released pip-gui on PyPI (with its own version > numbering and release cadence), then I'd be +1 on bundling that as an > optional IDLE addon, ensurepip style. I already agree in my second response on this thread that pip gui should be on pypi. A pip gui would be a pip add-on, not an IDLE add-on, just as turtledemo is a turtle add-on even though one can invoke it from the IDLE menu -- as long as turtle and turtledemo are installed. Issue 27051, write pip gui, is properly a separate issue from https://bugs.python.org/issue23551, start a pip gui, if and when available, from IDLE menu. I am against 'tightly coupling' something to IDLE when there is 'no compelling reason to do so'. -- Terry Jan Reedy From tjreedy at udel.edu Wed Nov 1 03:25:37 2017 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 1 Nov 2017 03:25:37 -0400 Subject: [Python-ideas] install pip packages from Python prompt In-Reply-To: References: <01f701d350eb$c4ee6210$4ecb2630$@sdamon.com> <021601d350ee$88f236d0$9ad6a470$@sdamon.com> <045101d351b9$54a25240$fde6f6c0$@sdamon.com> <4409775187778125080@unknownmsgid> Message-ID: On 11/1/2017 2:54 AM, Wes Turner wrote: > A Tk GUI for pip would need to frustratingly duplicate ~/.bash_history > and up-arrow to get the previous command. No it wouldn't, as the user would not be directly issuing pip commands. In any case, it would be trivial to keep a list of the pip commands submitted behind the gui facade and even to have an optional window to display them. > IDLE sucks. Why pollute your post with stupid flame bait? In many respects, IDLE's Shell is unequivocally better than interactive python in a line-oriented shell like Command Prompt. Ironically, given your comment above, one of its improvements is keeping a history of *statements* rather than a history of lines. This makes it possible to retrieve and edit a whole command (statement) at once, rather than in pieces (lines). -- Terry Jan Reedy From gadgetsteve at live.co.uk Wed Nov 1 04:17:20 2017 From: gadgetsteve at live.co.uk (Steve Barnes) Date: Wed, 1 Nov 2017 08:17:20 +0000 Subject: [Python-ideas] install pip packages from Python prompt In-Reply-To: References: <01f701d350eb$c4ee6210$4ecb2630$@sdamon.com> <021601d350ee$88f236d0$9ad6a470$@sdamon.com> <045101d351b9$54a25240$fde6f6c0$@sdamon.com> <4409775187778125080@unknownmsgid> Message-ID: On 01/11/2017 06:54, Wes Turner wrote: > Suggestions to help to minimize unnecessary logged bandwidth use and > even work with a closed loop LAN: > > This reads from the filesystem: > > ? ? import requests > > This would read from the PyPi service over the network bandwidth: > > ? ?#!pip install -U requests > ? ?#%run pip install -U requests > > ? ?#pip('install -U requests') > > This doesn't work because you SHOULD restart the interpreter after > running pip (because imports are cached): > > ? ? import requests > ? ? !pip install -U requests > ? ? import requests > > Some tips on running educational environments for beginners (optionally > in a lab): > One tip that I have used when teaching python in a closed, (sometimes internet free environment), was to pre-prepare by, in an on-line environment: 1. Create a virtual environment with the version of Python that I am going to be teaching on the target platform 2. Activate that environment 3. Ensure that I am On-line 4. Download the pip install packages that I know I will need by using `pip download` to download but not install the packages, ideally using the -r requirements.txt syntax, (plus any windows specific builds from Christoph Gohlke's site). 5. Go Off-line and run pip install with the downloaded package - if I hit any errors due to packages having unspecified dependencies add those to the requirements list and repeat from 3. (While I am at it I often log an issue with the package maintainer). 6. A fast, personal, run through my lesson plan to ensure that I haven't missed anything. I normally also download a few goodies that might not be essential to the lesson but that can act as a teaser for the more interested students. At the start of the first lesson I give the students the downloaded packages directory, usually on a USB key, and get them to pip install them while explaining the difference between local and on-line installation. I know that I could save having to get the students to run pip by packaging up the virtual environment as a portable, or using by pyInstaller, but having them run pip on the local downloads gives me a chance to explain how to do it in the wild. BTW while Docker is great for this it is a whole other learning experience, (plus getting it running with some corporate security & anti-virus can be quite a challenge). These USB keys are often re-used by other co-workers as a getting started or after my computer got changed/re-imaged starting point. -- Steve (Gadget) Barnes Any opinions in this message are my personal opinions and do not reflect those of my employer. --- This email has been checked for viruses by AVG. http://www.avg.com From wes.turner at gmail.com Wed Nov 1 09:12:08 2017 From: wes.turner at gmail.com (Wes Turner) Date: Wed, 1 Nov 2017 09:12:08 -0400 Subject: [Python-ideas] install pip packages from Python prompt In-Reply-To: References: <01f701d350eb$c4ee6210$4ecb2630$@sdamon.com> <021601d350ee$88f236d0$9ad6a470$@sdamon.com> <045101d351b9$54a25240$fde6f6c0$@sdamon.com> <4409775187778125080@unknownmsgid> Message-ID: On Wednesday, November 1, 2017, Terry Reedy wrote: > On 11/1/2017 2:54 AM, Wes Turner wrote: > > A Tk GUI for pip would need to frustratingly duplicate ~/.bash_history and >> up-arrow to get the previous command. >> > > No it wouldn't, as the user would not be directly issuing pip commands. In > any case, it would be trivial to keep a list of the pip commands submitted > behind the gui facade and even to have an optional window to display them. Now that there's a ``pip freeze`` and a ``conda env export``, it seems like it's much easier to determine what one just did to their {system, --user, virtualenv, conda root, or condaenv} without having a list of commands to review (ala .bash_history) or filesystem diff's. Would such a GUI handle these cases and also virtualenv create-and-restart? Bash can be installed in Windows. > > IDLE sucks. >> > > Why pollute your post with stupid flame bait? In many respects, IDLE's > Shell is unequivocally better than interactive python in a line-oriented > shell like Command Prompt. Ironically, given your comment above, one of > its improvements is keeping a history of *statements* rather than a history > of lines. This makes it possible to retrieve and edit a whole command > (statement) at once, rather than in pieces (lines). Spyder is an IDE with a command prompt (shell window) and a python prompt (python window, ipython window). > > -- > Terry Jan Reedy > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Wed Nov 1 09:41:11 2017 From: wes.turner at gmail.com (Wes Turner) Date: Wed, 1 Nov 2017 09:41:11 -0400 Subject: [Python-ideas] install pip packages from Python prompt In-Reply-To: References: <01f701d350eb$c4ee6210$4ecb2630$@sdamon.com> <021601d350ee$88f236d0$9ad6a470$@sdamon.com> <045101d351b9$54a25240$fde6f6c0$@sdamon.com> <4409775187778125080@unknownmsgid> Message-ID: On Wednesday, November 1, 2017, Steve Barnes wrote: > > > On 01/11/2017 06:54, Wes Turner wrote: > > Suggestions to help to minimize unnecessary logged bandwidth use and > > even work with a closed loop LAN: > > > > This reads from the filesystem: > > > > import requests > > > > This would read from the PyPi service over the network bandwidth: > > > > #!pip install -U requests > > #%run pip install -U requests > > > > #pip('install -U requests') > > > > This doesn't work because you SHOULD restart the interpreter after > > running pip (because imports are cached): > > > > import requests > > !pip install -U requests > > import requests > > > > Some tips on running educational environments for beginners (optionally > > in a lab): > > > > > One tip that I have used when teaching python in a closed, (sometimes > internet free environment), was to pre-prepare by, in an on-line > environment: > > 1. Create a virtual environment with the version of Python that I am > going to be teaching on the target platform > 2. Activate that environment > 3. Ensure that I am On-line > 4. Download the pip install packages that I know I will need by using > `pip download` to download but not install the packages, ideally using > the -r requirements.txt syntax, (plus any windows specific builds from > Christoph Gohlke's site). > 5. Go Off-line and run pip install with the downloaded package - if I > hit any errors due to packages having unspecified dependencies add those > to the requirements list and repeat from 3. (While I am at it I often > log an issue with the package maintainer). > 6. A fast, personal, run through my lesson plan to ensure that I > haven't missed anything. > > I normally also download a few goodies that might not be essential to > the lesson but that can act as a teaser for the more interested students. > > At the start of the first lesson I give the students the downloaded > packages directory, usually on a USB key, and get them to pip install > them while explaining the difference between local and on-line > installation. That works. You could also host the packages with devpi or just a static HTTP server. As a transparent cache, if you test your ``requirements.txt`` with each OS/CPU_architecture combination, DevPi will store and serve each package to everyone at once. So do you do (in a virtualenv): pip install --download ./dir -r requirements.txt And then pip install --no-index --find-links ./dir -r requirements.text Like in the pip docs? https://pip.pypa.io/en/stable/user_guide/#installing-from-local-packages DevPi can host package documentation. https://devpi.net/docs/devpi/devpi/stable/+doc/userman/devpi_packages.html#uploading-documentation * GitLab Pages can host package documentation and any static HTML pages (e.g. the CPython docs) alongside **the source** https://docs.gitlab.com/ce/user/project/pages/index.html https://github.com/python/cpython/tree/3.6/Doc ReadTheDocs can host Sphinx docs and can be run in a Docker container for an internet-free LAN: https://github.com/rtfd/readthedocs-docker-images > > I know that I could save having to get the students to run pip by > packaging up the virtual environment as a portable, or using by > pyInstaller, but having them run pip on the local downloads gives me a > chance to explain how to do it in the wild. AFAIU, YMMV with ``virtualenv --relocatable``; and it certainly doesn't do all combinations of OS and processor architecture: https://virtualenv.pypa.io/en/stable/userguide/#making-environments-relocatable itertools.combinations( ['win', 'mac', 'lin', '', '686', 'x86_64', 'ARM'], 2) > > BTW while Docker is great for this it is a whole other learning > experience, (plus getting it running with some corporate security & > anti-virus can be quite a challenge). Yeah, hosting Docker containers with Kubernetes across a couple extra workstations is initially more work than creating a few USB keys and waiting for everyone's virtualenvs to converge (with ``$ pip install``, ``!PIP_INDEX='' pip install``, and now a new pip GUI button press sequence). > > These USB keys are often re-used by other co-workers as a getting > started or after my computer got changed/re-imaged starting point. Wouldn't it be create if, just like oldschool floppies, there was a way to break off the sliding write protect tab of a USB stick? https://en.wikipedia.org/wiki/USB_flash_drive_security#Malware_infections CD-ROM and DVD-ROM drives seem to be more and more scare these days. > > -- > Steve (Gadget) Barnes > Any opinions in this message are my personal opinions and do not reflect > those of my employer. > > --- > This email has been checked for viruses by AVG. > http://www.avg.com > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Wed Nov 1 10:41:22 2017 From: guido at python.org (Guido van Rossum) Date: Wed, 1 Nov 2017 07:41:22 -0700 Subject: [Python-ideas] Defining an easily installable "Recommended baseline package set" In-Reply-To: References: Message-ID: Can you write 1-2 paragraphs with the argument for each? On Tue, Oct 31, 2017 at 10:01 PM, Nathaniel Smith wrote: > On Oct 31, 2017 4:42 AM, "Nick Coghlan" wrote: > > On 31 October 2017 at 02:29, Guido van Rossum wrote: > >> What's your proposed process to arrive at the list of recommended >> packages? >> > > I'm thinking it makes the most sense to treat inclusion in the recommended > packages list as a possible outcome of proposals for standard library > inclusion, rather than being something we'd provide a way to propose > specifically. > > We'd only use it in cases where a proposal would otherwise meet the > criteria for stdlib inclusion, but the logistics of actually doing so don't > work for some reason. > > Running the initial 5 proposals through that filter: > > * six: a cross-version compatibility layer clearly needs to be outside the > standard library > * setuptools: we want to update this in line with the PyPA interop specs, > not the Python language version > * cffi: updates may be needed for PyPA interop specs, Python > implementation updates or C language definition updates > * requests: updates are more likely to be driven by changes in network > protocols and client platform APIs than Python language changes > * regex: we don't want two regex engines in the stdlib, transparently > replacing _sre would be difficult, and _sre is still good enough for most > purposes > > > Some other packages that might meet these criteria, or at least be useful > for honing them: > > - lxml > - numpy > - cryptography > - idna > > -n > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From brent.bejot at gmail.com Wed Nov 1 13:49:57 2017 From: brent.bejot at gmail.com (brent bejot) Date: Wed, 1 Nov 2017 13:49:57 -0400 Subject: [Python-ideas] Hello from a new lurker Message-ID: Hello everyone, The python-dev website said I should introduce myself before lurking around for a while. So hi! I've been using python for 7 or so years now, live on the east coast of the U.S., grew up in Nebraska, and occasionally play the tuba and table-top games (through rarely together). I'm looking forward to hearing where the language is going and hope I can be of use after I lurk in the shadows for a while. Cheers, Brent -------------- next part -------------- An HTML attachment was scrubbed... URL: From k7hoven at gmail.com Wed Nov 1 14:29:56 2017 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Wed, 1 Nov 2017 20:29:56 +0200 Subject: [Python-ideas] Membership of infinite iterators In-Reply-To: References: <59E7D6EC.6090708@canterbury.ac.nz> Message-ID: Let me add something to the discussion that we had last month about code that cannot be interrupted with Ctrl-C or with the Jupyter Notebook interrupt command etc. I wasn't completely sure if anyone cared much about being able to interrupt long-running code and about things like atomic operations etc., but maybe I'll just write this one down anyway while it's still in my head: As a reminder of what we discussed, for KeyboardInterrupt to happen, C code needs to manually call PyErr_CheckSignals() regularly to handle signals, but it might be too much overhead to do that in tight loops. But where does the overhead come from and what could perhaps be done? Read below: On Thu, Oct 19, 2017 at 10:54 AM, Koos Zevenhoven wrote: > On Thu, Oct 19, 2017 at 3:42 AM, Nick Coghlan wrote: > >> On 19 October 2017 at 08:34, Greg Ewing >> wrote: >> >>> Nick Coghlan wrote: >>> >>>> since breaking up the current single level loops as nested loops would >>>> be a pre-requisite for allowing these APIs to check for signals while >>>> they're running while keeping the per-iteration overhead low >>>> >>> >>> Is there really much overhead? Isn't it just checking a flag? >>> >> >> It's checking an atomically updated flag, so it forces CPU cache >> synchronisation, which means you don't want to be doing it on every >> iteration of a low level loop. >> >> > Even just that it's a C function call makes me not want to recommend doing > it in a lot of tight loops. Who knows what the function does anyway, let > alone what it might or might not do in the future. > ?While what Nick states above about CPU cache synchronization doesn't sound quite **right**, it may take a core developer a whole day to roughly figure out if the statement is even **true**. In particular, it is not obvious if it may **occasionally** be true, or if it is **always** true in **some cases**, or if it's *sometimes* true in *all cases*. The difficulties are partly because pyatomic.h [1] is "arranged" in layers of branched C preprocessor directives [2], and because the whole of pyatomic.h is in fact "deep magic", as is confirmed by a comment in the beginning of the file. Some of the complexity has emerged after that comment was already there. SPOILER ALERT! At the moment, Nick's statement is in fact **always** true in **all** cases (at least when ignoring untypical cases and some inaccuracies in phrasing). ?Another question is whether the statement **should** be true at all.? ? PyErr_CheckSignals(), the function that checks for pending signals, now **implicitly** uses the strictest possible memory-order requirement (SEQ_CST) for checking the `is_tripped` flag, a value which can be used to peek whether there are any pending signals. This means that two threads that call PyErr_CheckSignals can't even **check** the value of that flag at the same time, and they'll have to wait for each other and whatever the CPU hardware implementation does for cache synchronzation. ?From a correctness point of view, that is absolutely great: if PyErr_CheckSignals() is called, it is guaranteed to notice a new signal regardles of how small the number of picoseconds after the `is_tripped` flag has been set. But is that really important? Do we really need things to be slowed down by this? And do we want people to "optimize" code by avoiding signal checking? The signals won't be caught until checked anyway, and as we've seen, one solution is to use a counter to determine if enough time has passed that another check for potential signals should happen. That does, however, raise the question of who should own the counter, and if it's OK for multiple threads to use the same counter. If yes, then would we be able to avoid slow atomic decrements (or increments)? But another solution might be to make a less strict but almost equivalent functionality with much less overhead. Something like this in a header file: static inline int PyErr_PROBE_SIGNALS() { static int volatile *flag = (int volatile *) &is_tripped; if (*flag) { return PyErr_CheckSignals(); } else { return 0; } } ?Here, the flag is casted to volatile to force a check to happen each time from memory/cache. However, to make it more standard and to make sure it works with all compilers/platforms, it might be better to, instead of *flag, use an atomic load with "MEMORY_ORDER_RELAXED".? Another thing is that `is_tripped`, which is currently declared as static inside signalmodule.c [4], should somehow be exposed in the API to make it available for the inline function in headers. ?This solution might be fast enough for a lot of cases, although still slightly slower than the counter approach, at least if the counter approach would completely avoid per-iteration memory access by requiring each function to own the counter that is used for this. So you read the whole email? Cool! ???Koos? ??[1] https://github.com/python/cpython/blob/master/Include/pyatomic.h [2] C preprocessor code is kind of like commented-out Python code, except that not even the Spanish Inquisition [3] expects you to indent ?the blocks that they define. [3] https://www.youtube.com/watch?v=Nf_Y4MbUCLY? [4] https://github.com/python/cpython/blob/master/Modules/signalmodule.c > >> However, reviewing Serhiy's PR reminded me that PyErr_CheckSignals() >> already encapsulates the "Should this thread even be checking for signals >> in the first place?" logic, which means the code change to make the >> itertools iterators inherently interruptible with Ctrl-C is much smaller >> than I thought it would be. >> > > ?And if it didn't encapsulate that, you would probably have written a > wrapper that does.? Good thing it's the wrapper that's exposed in the API. > > > >> That approach is also clearly safe from an exception handling point of >> view, since all consumer loops already need to cope with the fact that >> itr.__next__() may raise arbitrary exceptions (including KeyboardInterrupt). >> >> > So that change alone already offers a notable improvement, and combining >> it with a __length_hint__() implementation that keeps container >> constructors from even starting to iterate would go even further towards >> making the infinite iterators more user friendly. >> >> Similar signal checking changes to the consumer loops would also be >> possible, but I don't think that's an either/or decision: changing the >> iterators means they'll be interruptible for any consumer, while changing >> the consumers would make them interruptible for any iterator, and having >> checks in both the producer & the consumer merely means that you'll be >> checking for signals twice every 65k iterations, rather than once. >> >> > ?Indeed it's not strictly an either/or decision, but more about where we > might spend time executing C code. But I'm leaning a bit towards doing it > on the consumer side, because there it's more obvious that ?the code might > take some time to run. > > If the consumer ends up iterating over pure-Python objects, there are no > concerns about the overhead. But if it *does* call a C-implemented > __next__, then that's the case where we actully need the whole thing. > Adding the check in both places would double the (small) overhead. And > nested (wrapped) iterators are also a thing. > > ???Koos > > > > -- > + Koos Zevenhoven + http://twitter.com/k7hoven + > -- + Koos Zevenhoven + http://twitter.com/k7hoven + -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Wed Nov 1 15:08:19 2017 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 01 Nov 2017 12:08:19 -0700 Subject: [Python-ideas] Hello from a new lurker In-Reply-To: References: Message-ID: <59FA1BA3.1030200@stoneleaf.us> On 11/01/2017 10:49 AM, brent bejot wrote: > The python-dev website said I should introduce myself before lurking around for a while. So hi! I've been using python > for 7 or so years now, live on the east coast of the U.S., grew up in Nebraska, and occasionally play the tuba and > table-top games (through rarely together). I'm looking forward to hearing where the language is going and hope I can be > of use after I lurk in the shadows for a while. Welcome! Enjoy your time lurking. When new ideas come up, feel free to comment if that idea will help or hinder you. -- ~Ethan~ From tjreedy at udel.edu Wed Nov 1 17:47:52 2017 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 1 Nov 2017 17:47:52 -0400 Subject: [Python-ideas] install pip packages from Python prompt In-Reply-To: References: <01f701d350eb$c4ee6210$4ecb2630$@sdamon.com> <021601d350ee$88f236d0$9ad6a470$@sdamon.com> <045101d351b9$54a25240$fde6f6c0$@sdamon.com> <4409775187778125080@unknownmsgid> Message-ID: On 11/1/2017 3:06 AM, Terry Reedy wrote: > On 11/1/2017 1:25 AM, Nick Coghlan wrote: >> On 1 November 2017 at 08:50, Terry Reedy > > wrote: >> >> ??? In April 2016, after posting the idea to pydev list and getting 'go >> ??? ahead's from Nick Coughlin and someone else, with no negatives, I >> ??? approved Upendra Kumar's GSOC proposal to write a pip gui.? This was >> ??? https://bugs.python.org/issue27051 >> ??? .? On June 20, Ned Deily and >> ??? Nick Coughlin vetoed adding a pip gui anywhere in the stdlib since >> ??? it depended on something not in the stdlib, and perhaps for other >> ??? reasons I don't fully understand. >> >> >> Clarifying the objection here (since the linked issue is a fairly long >> one): what I'm against is tightly coupling the pip-gui development & >> release process to the CPython development & release process when we >> don't have any compelling reason to do so. > > Thank you for clarifying. This is a reason that I did not understand ;-). > >> Given an independently released pip-gui on PyPI (with its own version >> numbering and release cadence), then I'd be +1 on bundling that as an >> optional IDLE addon, ensurepip style. > > I already agree in my second response on this thread that pip gui should > be on pypi. > > A pip gui would be a pip add-on, not an IDLE add-on, just as turtledemo > is a turtle add-on even though one can invoke it from the IDLE menu -- > as long as turtle and turtledemo are installed. > > Issue 27051, write pip gui, is properly a separate issue from > https://bugs.python.org/issue23551, start a pip gui, if and when > available, from IDLE menu.? I am against 'tightly coupling' something to > IDLE when there is 'no compelling reason to do so'. When pip installs a package into site_packages, does it at any point run package-specific installation code? setup.py? More specifically, can pip install an IDLE extension. If so, I think installing pipgui should add 'x_pipgui.py' to idlelib, if it exists, and add a section to idlelib/config-extension.def. Using the existing extension mechanism would be an alternative to patching IDLE to conditionally add pipgui to some menu. -- Terry Jan Reedy From tjreedy at udel.edu Wed Nov 1 18:08:28 2017 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 1 Nov 2017 18:08:28 -0400 Subject: [Python-ideas] install pip packages from Python prompt In-Reply-To: References: <01f701d350eb$c4ee6210$4ecb2630$@sdamon.com> <021601d350ee$88f236d0$9ad6a470$@sdamon.com> <045101d351b9$54a25240$fde6f6c0$@sdamon.com> <4409775187778125080@unknownmsgid> Message-ID: On 11/1/2017 5:47 PM, Terry Reedy wrote: > When pip installs a package into site_packages, does it at any point run > ?package-specific installation code?? setup.py?? More specifically, can > pip install an IDLE extension. If so, I think installing pipgui should > add 'x_pipgui.py' to idlelib, if it exists, and add a section to > idlelib/config-extension.def.? Using the existing extension mechanism > would be an alternative to patching IDLE to conditionally add pipgui to > some menu. I meant to add that using the existing extension mechanism would allow pipgui to be run from 3.5 IDLE or even earlier, depending on how it is coded and the tcl/tk installed on the machine. Pipgui really needs ttk, which requires tcl/tk >= 8.5, but that has been true on Python Windows installs at least since 2.7. -- Terry Jan Reedy From ncoghlan at gmail.com Wed Nov 1 21:39:03 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 2 Nov 2017 11:39:03 +1000 Subject: [Python-ideas] Membership of infinite iterators In-Reply-To: References: <59E7D6EC.6090708@canterbury.ac.nz> Message-ID: On 2 November 2017 at 04:29, Koos Zevenhoven wrote: > SPOILER ALERT! At the moment, Nick's statement is in fact **always** true > in **all** cases (at least when ignoring untypical cases and some > inaccuracies in phrasing). ?Another question is whether the statement > **should** be true at all.? > ? > > PyErr_CheckSignals(), the function that checks for pending signals, now > **implicitly** uses the strictest possible memory-order requirement > (SEQ_CST) for checking the `is_tripped` flag, a value which can be used to > peek whether there are any pending signals. This means that two threads > that call PyErr_CheckSignals can't even **check** the value of that flag > at the same time, and they'll have to wait for each other and whatever the > CPU hardware implementation does for cache synchronzation. > Nice, that's deeper than I went - I just assumed there was an interlocked_increment in there somewhere, without checking whether or not there were any optimised code paths that managed to avoid that call :) > ?From a correctness point of view, that is absolutely great: if > PyErr_CheckSignals() is called, it is guaranteed to notice a new signal > regardles of how small the number of picoseconds after the `is_tripped` > flag has been set. But is that really important? Do we really need things > to be slowed down by this? And do we want people to "optimize" code by > avoiding signal checking? > It isn't signal handling latency that's the concern here, it's potentially missing signals. Consider the case where we have 3 threads: A, B, C. A is the main thread that will actually handle signals, B and C both happened to receive them. We'll also assume the two threads receive *different* signals (otherwise they'll potentially get collapsed into one notification regardless). The scenario we want to avoid is having one or both of the signals set, but is_tripped cleared. With an interlocked check (where both 0->1 and 1->0 are memory barriers), that clearly can't happen. I suspect you're right that this could also be achieved with a less strict memory sync requirement on the 0->1 check, but that's much harder to reason about than simply limiting the number of iterations we make through an iterator consumption loop before checking for signals. > The signals won't be caught until checked anyway, and as we've seen, one > solution is to use a counter to determine if enough time has passed that > another check for potential signals should happen. That does, however, > raise the question of who should own the counter, and if it's OK for > multiple threads to use the same counter. If yes, then would we be able to > avoid slow atomic decrements (or increments)? > > But another solution might be to make a less strict but almost equivalent > functionality with much less overhead. Something like this in a header file: > > static inline int PyErr_PROBE_SIGNALS() { > static int volatile *flag = (int volatile *) &is_tripped; > if (*flag) { > return PyErr_CheckSignals(); > } > else { > return 0; > } > } > > ?Here, the flag is casted to volatile to force a check to happen each time > from memory/cache. However, to make it more standard and to make sure it > works with all compilers/platforms, it might be better to, instead of > *flag, use an atomic load with "MEMORY_ORDER_RELAXED".? Another thing is > that `is_tripped`, which is currently declared as static inside > signalmodule.c [4], should somehow be exposed in the API to make it > available for the inline function in headers. > > ?This solution might be fast enough for a lot of cases, although still > slightly slower than the counter approach, at least if the counter approach > would completely avoid per-iteration memory access by requiring each > function to own the counter that is used for this. > One thing I like about a nested inner loop is that it's easy to understand the rationale for it *without* knowing any of the details of how memory synchronisation works, as it's just: - we want to check for signals regularly so the latency in interrupt handling isn't too high - PyErr_CheckSignals() is too expensive to call on every iteration - so we only check every N iterations Memory synchronisation then only comes up if someone asks why "PyErr_CheckSignals" is expensive to call. And while it wouldn't surprise at all if you're right and there are ways to make that call cheaper, they're still never going to be cheaper than explicitly reducing the frequency with which it is called. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Wed Nov 1 22:36:42 2017 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 1 Nov 2017 19:36:42 -0700 Subject: [Python-ideas] Defining an easily installable "Recommended baseline package set" In-Reply-To: References: Message-ID: On Wed, Nov 1, 2017 at 7:41 AM, Guido van Rossum wrote: > Can you write 1-2 paragraphs with the argument for each? > > On Tue, Oct 31, 2017 at 10:01 PM, Nathaniel Smith wrote: >> - lxml My impression (probably others are more knowledgeable) is that lxml has more or less replaced the stdlib 'xml' package as the de facto standard -- sort of similar to the urllib2/requests situation. AFAIK lxml has never been proposed for stdlib inclusion and I believe the fact that it's all in Cython would be a barrier even if the maintainers were amenable. But it might be helpful to our users to put a box at the top of the 'xml' docs suggesting people check out 'lxml', similar to the one on the urllib2 docs. >> - numpy Numpy's arrays are a foundational data structure and de facto standard, and would probably fit naturally in the stdlib semantically, but for a number of logistical/implementational reasons it doesn't make sense to merge. Probably doesn't make much difference whether python-dev "blesses" it or not in practice, since there aren't any real competitors inside or outside the stdlib; it'd more just be an acknowledgement of the status quo. >> - cryptography Conceptually, core cryptographic operations are the kind of functionality that you might expect to see in the stdlib, but the unique sensitivity of crypto code makes this a bad idea. Historically there have been a variety of basic crypto packages for Python, but at this point IIUC the other ones are all considered obsolete-and-potentially-dangerous and the consensus is everyone should move to 'cryptography', so documenting that in this PEP might help send people in the right direction. >> - idna This is a bit of a funny one. IDNA functionality is pretty fundamental -- you need it to do unicode<->bytes conversions on hostnames, so basically anyone doing networking needs it. Python ships with some built-in IDNA functionality (as the "idna" codec), but it's using an obsolete standard (IDNA2003, versus the current IDNA2008, see bpo-17305), and IIRC Christian thinks the whole codec-based design is the wrong approach... basically what we have in the stdlib has been broken for most of a decade and there doesn't seem to be anyone interested in fixing it. So... in the long run the stdlib support should either be fixed or deprecated. I'm not sure which is better. (The argument for deprecating it would be that IIUC you need to update the tables whenever a new unicode standard comes out, and since it's a networking thing you want to keep in sync with the rest of the world, which is easier with a standalone library. I don't know how much this matters in practice.) But right now, this library is just better than the stdlib functionality, and it wouldn't hurt to document that. -n -- Nathaniel J. Smith -- https://vorpus.org From guido at python.org Wed Nov 1 22:52:30 2017 From: guido at python.org (Guido van Rossum) Date: Wed, 1 Nov 2017 19:52:30 -0700 Subject: [Python-ideas] Defining an easily installable "Recommended baseline package set" In-Reply-To: References: Message-ID: Good write-ups. Though it sounds like idna should just be moved into the stdlib. (Agreed that for things like this codecs are not a great idea.) We have other things that must be updated whenever the Unicode standard changes and it seems that doing this in feature releases is typically fine, and treating it as a bugfix (so doing it in bugfix releases is acceptable) sounds fine to me too. Unless it were so security-critical that we'd want to enable users to do such updates to less-supported Pythons or on a tighter schedule (which is one of the reasons requests can't go into the stdlib -- they have their own cert store). On Wed, Nov 1, 2017 at 7:36 PM, Nathaniel Smith wrote: > On Wed, Nov 1, 2017 at 7:41 AM, Guido van Rossum wrote: > > Can you write 1-2 paragraphs with the argument for each? > > > > On Tue, Oct 31, 2017 at 10:01 PM, Nathaniel Smith wrote: > >> - lxml > > My impression (probably others are more knowledgeable) is that lxml > has more or less replaced the stdlib 'xml' package as the de facto > standard -- sort of similar to the urllib2/requests situation. AFAIK > lxml has never been proposed for stdlib inclusion and I believe the > fact that it's all in Cython would be a barrier even if the > maintainers were amenable. But it might be helpful to our users to put > a box at the top of the 'xml' docs suggesting people check out 'lxml', > similar to the one on the urllib2 docs. > > >> - numpy > > Numpy's arrays are a foundational data structure and de facto > standard, and would probably fit naturally in the stdlib semantically, > but for a number of logistical/implementational reasons it doesn't > make sense to merge. Probably doesn't make much difference whether > python-dev "blesses" it or not in practice, since there aren't any > real competitors inside or outside the stdlib; it'd more just be an > acknowledgement of the status quo. > > >> - cryptography > > Conceptually, core cryptographic operations are the kind of > functionality that you might expect to see in the stdlib, but the > unique sensitivity of crypto code makes this a bad idea. Historically > there have been a variety of basic crypto packages for Python, but at > this point IIUC the other ones are all considered > obsolete-and-potentially-dangerous and the consensus is everyone > should move to 'cryptography', so documenting that in this PEP might > help send people in the right direction. > > >> - idna > > This is a bit of a funny one. IDNA functionality is pretty fundamental > -- you need it to do unicode<->bytes conversions on hostnames, so > basically anyone doing networking needs it. Python ships with some > built-in IDNA functionality (as the "idna" codec), but it's using an > obsolete standard (IDNA2003, versus the current IDNA2008, see > bpo-17305), and IIRC Christian thinks the whole codec-based design is > the wrong approach... basically what we have in the stdlib has been > broken for most of a decade and there doesn't seem to be anyone > interested in fixing it. So... in the long run the stdlib support > should either be fixed or deprecated. I'm not sure which is better. > (The argument for deprecating it would be that IIUC you need to update > the tables whenever a new unicode standard comes out, and since it's a > networking thing you want to keep in sync with the rest of the world, > which is easier with a standalone library. I don't know how much this > matters in practice.) But right now, this library is just better than > the stdlib functionality, and it wouldn't hurt to document that. > > -n > > -- > Nathaniel J. Smith -- https://vorpus.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Thu Nov 2 04:46:23 2017 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 2 Nov 2017 01:46:23 -0700 Subject: [Python-ideas] install pip packages from Python prompt In-Reply-To: References: <01f701d350eb$c4ee6210$4ecb2630$@sdamon.com> <021601d350ee$88f236d0$9ad6a470$@sdamon.com> <045101d351b9$54a25240$fde6f6c0$@sdamon.com> <4409775187778125080@unknownmsgid> Message-ID: On Wed, Nov 1, 2017 at 2:47 PM, Terry Reedy wrote: > When pip installs a package into site_packages, does it at any point run > package-specific installation code? setup.py? Nope. That's a can of worms that we've so far avoided opening. > More specifically, can pip > install an IDLE extension. If so, I think installing pipgui should add > 'x_pipgui.py' to idlelib, if it exists, and add a section to > idlelib/config-extension.def. Using the existing extension mechanism would > be an alternative to patching IDLE to conditionally add pipgui to some menu. There is a de facto standard way to do this, which is to advertise a setuptools entrypoint: https://setuptools.readthedocs.io/en/latest/setuptools.html#dynamic-discovery-of-services-and-plugins This is some static metadata that a plugin can include in their package in a well known place, and that tools like pkg_resources can then look up. But unfortuately this hasn't been standardized, and there's currently no way to do the lookup from the stdlib, so maybe this is not so helpful for IDLE... -n -- Nathaniel J. Smith -- https://vorpus.org From ncoghlan at gmail.com Thu Nov 2 06:53:02 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 2 Nov 2017 20:53:02 +1000 Subject: [Python-ideas] install pip packages from Python prompt In-Reply-To: References: <01f701d350eb$c4ee6210$4ecb2630$@sdamon.com> <021601d350ee$88f236d0$9ad6a470$@sdamon.com> <045101d351b9$54a25240$fde6f6c0$@sdamon.com> <4409775187778125080@unknownmsgid> Message-ID: On 2 November 2017 at 18:46, Nathaniel Smith wrote: > But unfortuately this hasn't been standardized, and there's currently > no way to do the lookup from the stdlib, so maybe this is not so > helpful for IDLE... > The entry point file format was recently promoted to a PyPA interoperability spec (without a PEP, as we documented it as-is, rather than changing anything): https://packaging.python.org/specifications/entry-points/ While the point about the standard library lacking the ability to read the metadata for installed packages still stands, it's also not too hard to implement a rudimentary version that just iterates over sys.path looking for `entry_points.txt` files in `*.dist-info` subdirectories. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From k7hoven at gmail.com Thu Nov 2 07:27:17 2017 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Thu, 2 Nov 2017 13:27:17 +0200 Subject: [Python-ideas] Membership of infinite iterators In-Reply-To: References: <59E7D6EC.6090708@canterbury.ac.nz> Message-ID: On Thu, Nov 2, 2017 at 3:39 AM, Nick Coghlan wrote: > On 2 November 2017 at 04:29, Koos Zevenhoven wrote: > >> SPOILER ALERT! At the moment, Nick's statement is in fact **always** >> true in **all** cases (at least when ignoring untypical cases and some >> inaccuracies in phrasing). ?Another question is whether the statement >> **should** be true at all.? >> ? >> >> PyErr_CheckSignals(), the function that checks for pending signals, now >> **implicitly** uses the strictest possible memory-order requirement >> (SEQ_CST) for checking the `is_tripped` flag, a value which can be used to >> peek whether there are any pending signals. This means that two threads >> that call PyErr_CheckSignals can't even **check** the value of that flag >> at the same time, and they'll have to wait for each other and whatever the >> CPU hardware implementation does for cache synchronzation. >> > > Nice, that's deeper than I went - I just assumed there was an > interlocked_increment in there somewhere, without checking whether or not > there were any optimised code paths that managed to avoid that call :) > > >> ?From a correctness point of view, that is absolutely great: if >> PyErr_CheckSignals() is called, it is guaranteed to notice a new signal >> regardles of how small the number of picoseconds after the `is_tripped` >> flag has been set. But is that really important? Do we really need things >> to be slowed down by this? And do we want people to "optimize" code by >> avoiding signal checking? >> > > It isn't signal handling latency that's the concern here, it's potentially > missing signals. Consider the case where we have 3 threads: A, B, C. A is > the main thread that will actually handle signals, B and C both happened to > receive them. We'll also assume the two threads receive *different* > signals (otherwise they'll potentially get collapsed into one notification > regardless). > > The scenario we want to avoid is having one or both of the signals set, > but is_tripped cleared. With an interlocked check (where both 0->1 and 1->0 > are memory barriers), that clearly can't happen. I suspect you're right > that this could also be achieved with a less strict memory sync requirement > on the 0->1 check, but that's much harder to reason about than simply > limiting the number of iterations we make through an iterator consumption > loop before checking for signals. > ??Missing signals is a concern, and there's another concern that one received signal might be interpreted as two separate occurrences. But it doesn't prevent us from *checking* is_tripped in a more relaxed manner, as long as the 0->1 check is *verified* using something like SEQ_CST and the associated signal handling is synchronized properly. And that's why my PyErr_PROBE_SIGNALS below should work, and wouldn't prevent multiple threads from handling signals either. ? > > >> The signals won't be caught until checked anyway, and as we've seen, one >> solution is to use a counter to determine if enough time has passed that >> another check for potential signals should happen. That does, however, >> raise the question of who should own the counter, and if it's OK for >> multiple threads to use the same counter. If yes, then would we be able to >> avoid slow atomic decrements (or increments)? >> >> But another solution might be to make a less strict but almost equivalent >> functionality with much less overhead. Something like this in a header file: >> >> static inline int PyErr_PROBE_SIGNALS() { >> static int volatile *flag = (int volatile *) &is_tripped; >> if (*flag) { >> return PyErr_CheckSignals(); >> } >> else { >> return 0; >> } >> } >> >> ?Here, the flag is casted to volatile to force a check to happen each >> time from memory/cache. However, to make it more standard and to make sure >> it works with all compilers/platforms, it might be better to, instead of >> *flag, use an atomic load with "MEMORY_ORDER_RELAXED".? Another thing is >> that `is_tripped`, which is currently declared as static inside >> signalmodule.c [4], should somehow be exposed in the API to make it >> available for the inline function in headers. >> >> ?This solution might be fast enough for a lot of cases, although still >> slightly slower than the counter approach, at least if the counter approach >> would completely avoid per-iteration memory access by requiring each >> function to own the counter that is used for this. >> > > One thing I like about a nested inner loop is that it's easy to understand > the rationale for it *without* knowing any of the details of how memory > synchronisation works, as it's just: > > - we want to check for signals regularly so the latency in interrupt > handling isn't too high > - PyErr_CheckSignals() is too expensive to call on every iteration > - so we only check every N iterations > > My initial "negative" reaction to that potential solution ("I don't believe nesting loops is strictly a requirement." ?K) was because going for manually splitting the loops into smaller inner loops sounds like preparing a nightmare for all C programmers. And I don't think it's possible to write a preprocessor macro that does that for you. Maybe things like Cython could in principle do this automatically, though. Using a counter can have a very small overhead, because things can happen completely in processor registers. > Memory synchronisation then only comes up if someone asks why > "PyErr_CheckSignals" is expensive to call. And while it wouldn't surprise > at all if you're right and there are ways to make that call cheaper, > they're still never going to be cheaper than explicitly reducing the > frequency with which it is called. > > ?There's a limit to how cheap the call to PyErr_CheckSignals() can be. As I mentioned earlier, even just the fact that it's a C function call? can be too much. That's why, in the above, I used a new name PyErr_PROBE_SIGNALS() instead of optimizing the existing PyErr_CheckSignals() ?? turning PyErr_CheckSignals() into a static inline function would change the ABI. I also didn't call it PyErr_CHECK_SIGNALS() because the functionality is not strictly equivalent to the existing function. ???Koos -- + Koos Zevenhoven + http://twitter.com/k7hoven + -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Thu Nov 2 07:41:58 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 2 Nov 2017 12:41:58 +0100 Subject: [Python-ideas] Membership of infinite iterators References: <59E7D6EC.6090708@canterbury.ac.nz> Message-ID: <20171102124158.73a5bcbb@fsol> On Thu, 2 Nov 2017 13:27:17 +0200 Koos Zevenhoven wrote: > ?There's a limit to how cheap the call to PyErr_CheckSignals() can be. As I > mentioned earlier, even just the fact that it's a C function call? can be > too much. > > That's why, in the above, I used a new name PyErr_PROBE_SIGNALS() instead > of optimizing the existing PyErr_CheckSignals() ?? turning > PyErr_CheckSignals() into a static inline function would change the ABI. I > also didn't call it PyErr_CHECK_SIGNALS() because the functionality is not > strictly equivalent to the existing function. Please. If you want to replace PyErr_CheckSignals() with something faster, the first thing to do is to prove that PyErr_CheckSignals() *is* too expensive. Regards Antoine. From k7hoven at gmail.com Thu Nov 2 08:00:26 2017 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Thu, 2 Nov 2017 14:00:26 +0200 Subject: [Python-ideas] Membership of infinite iterators In-Reply-To: <20171102124158.73a5bcbb@fsol> References: <59E7D6EC.6090708@canterbury.ac.nz> <20171102124158.73a5bcbb@fsol> Message-ID: On Thu, Nov 2, 2017 at 1:41 PM, Antoine Pitrou wrote: > On Thu, 2 Nov 2017 13:27:17 +0200 > Koos Zevenhoven wrote: > > ?There's a limit to how cheap the call to PyErr_CheckSignals() can be. > As I > > mentioned earlier, even just the fact that it's a C function call? can be > > too much. > > > > That's why, in the above, I used a new name PyErr_PROBE_SIGNALS() instead > > of optimizing the existing PyErr_CheckSignals() ?? turning > > PyErr_CheckSignals() into a static inline function would change the ABI. > I > > also didn't call it PyErr_CHECK_SIGNALS() because the functionality is > not > > strictly equivalent to the existing function. > > Please. If you want to replace PyErr_CheckSignals() with something > faster, the first thing to do is to prove that PyErr_CheckSignals() > *is* too expensive. > I believe Serhiy proved that by showing that his first patch did not have negligible overhead for all cases. One might also write a C-implemented fibonacci calculator or similar to prove the overhead is significant. While I agree that there's a hint of premature optimization involved, my justification for it is that I want people to be confident that when they add the check, it's not going to slow down the loop significantly. Maybe the fear of overhead is one of reasons for people to write uninterruptible code. IMO, it should be made as easy as possible for the C programmer. ??Koos ? -- + Koos Zevenhoven + http://twitter.com/k7hoven + -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Thu Nov 2 08:22:29 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 2 Nov 2017 13:22:29 +0100 Subject: [Python-ideas] Importance of noticing new signals References: <59E7D6EC.6090708@canterbury.ac.nz> Message-ID: <20171102132229.6e50913d@fsol> On Wed, 1 Nov 2017 20:29:56 +0200 Koos Zevenhoven wrote: > > ?From a correctness point of view, that is absolutely great: if > PyErr_CheckSignals() is called, it is guaranteed to notice a new signal > regardles of how small the number of picoseconds after the `is_tripped` > flag has been set. But is that really important? I was going to answer "no"... but actually yes. The important case is the event loop type case: while (1) { select([some file descriptors]); if (errno == EINTR) { PyErr_CheckSignals(); if (PyErr_Occurred()) break; } /* continue select()ing... */ } Now say at a given point in time, no fds are actually active (or even waited for), but some signal arrives (SIGINT perhaps). select() is woken up and returns with errno EINTR. Two things then can happen: - if PyErr_CheckSignals() notices the signal, it will run the relevant signal handler, which may raise an exception and trigger the select() loop to exit (e.g. SIGINT would raise KeyboardInterrupt) - if PyErr_CheckSignals() misses the signal, the loop will enter again, and select() may sleep for an infinite amount of time Of course, what we're doing with select() above can already apply for read() or other interruptible syscalls waiting for outside data... and that pattern is present a lot internally, especially since PEP 475 ("Retry system calls failing with EINTR"). Now, is the "sequentially consistent" ordering on is_tripped sufficient to guarantee that signals won't be missed on a weak-ordering platform? I *think* so, but an expert would need to check that code (or we cross our fingers and wait for a hypothetical bug report). Regards Antoine. From mistersheik at gmail.com Thu Nov 2 08:28:20 2017 From: mistersheik at gmail.com (Neil Girdhar) Date: Thu, 2 Nov 2017 05:28:20 -0700 (PDT) Subject: [Python-ideas] Add single() to itertools In-Reply-To: References: <2e42329e-294c-0143-de80-43617d658866@mail.mipt.ru> Message-ID: <5c2c8076-e95d-48b8-8f23-8f049843e8be@googlegroups.com> This request is called "one" in more-itertools: http://more-itertools.readthedocs.io/en/latest/api.html It raises ValueError as Steve suggested. On Monday, October 30, 2017 at 8:34:26 AM UTC-6, Guido van Rossum wrote: > > This is a key example of a case where code speaks. Can you write an > implementation of how you would want single() to work in Python code? > > On Mon, Oct 30, 2017 at 2:49 AM, Ivan Pozdeev via Python-ideas < > python... at python.org > wrote: > >> >> >> On 30.10.2017 9:29, python-ide... at python.org wrote: >> >>> If I have understood your use-case, you have a function that returns a >>> list of results (or possibly an iterator, or a tuple, or some other >>> sequence): >>> >>> print(search(haystack, needle)) >>> # prints ['bronze needle', 'gold needle', 'silver needle'] >>> >>> There are times you expect there to be a single result, and if there are >>> multiple results, that is considered an error. Am I correct so far? >>> >> Correct. >> >>> If so, then sequence unpacking is your friend: >>> >>> result, = search(haystack, needle) >>> >>> <...> >>> >>> I *think* this will solve your problem. >>> >>> If not, can you please explain what "single()" is supposed to do, why it >>> belongs in itertools, and show an example of how it will work. >>> >> That works. Too arcane in my book though (and others' too according to >> https://stackoverflow.com/a/473337/648265), and the error messages are >> cryptic in this use case. >> It also cannot be a part of an expression, unlike next(). >> >> The initial post on the above link summarizes the suggested >> implementation pretty well. >> >> -- >> Regards, >> Ivan >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python... at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > > -- > --Guido van Rossum (python.org/~guido) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From k7hoven at gmail.com Thu Nov 2 08:57:12 2017 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Thu, 2 Nov 2017 14:57:12 +0200 Subject: [Python-ideas] Importance of noticing new signals In-Reply-To: <20171102132229.6e50913d@fsol> References: <59E7D6EC.6090708@canterbury.ac.nz> <20171102132229.6e50913d@fsol> Message-ID: On Thu, Nov 2, 2017 at 2:22 PM, Antoine Pitrou wrote: > On Wed, 1 Nov 2017 20:29:56 +0200 > Koos Zevenhoven wrote: > > > > ?From a correctness point of view, that is absolutely great: if > > PyErr_CheckSignals() is called, it is guaranteed to notice a new signal > > regardles of how small the number of picoseconds after the `is_tripped` > > flag has been set. But is that really important? > > I was going to answer "no"... but actually yes. The important case is > the event loop type case: > > while (1) { > select([some file descriptors]); > if (errno == EINTR) { > PyErr_CheckSignals(); > if (PyErr_Occurred()) break; > } > /* continue select()ing... */ > } > > Now say at a given point in time, no fds are actually active (or even > waited for), but some signal arrives (SIGINT perhaps). > select() is woken up and returns with errno EINTR. Two things then can > happen: > > - if PyErr_CheckSignals() notices the signal, it will run the relevant > signal handler, which may raise an exception and trigger the select() > loop to exit (e.g. SIGINT would raise KeyboardInterrupt) > > - if PyErr_CheckSignals() misses the signal, the loop will enter again, > and select() may sleep for an infinite amount of time > > Oh! So that would provide a proper reason for my just-in-case decision to name the faster near-equivalent functionality PyErr_PROBE_SIGNALS instead of PyErr_CHECK_SIGNALS. Cross-referencing to that (thread about making Ctrl-C "always" work): https://mail.python.org/pipermail/python-ideas/2017-November/047631.html Of course, what we're doing with select() above can already apply for > read() or other interruptible syscalls waiting for outside data... and > that pattern is present a lot internally, especially since > PEP 475 ("Retry system calls failing with EINTR"). > > Now, is the "sequentially consistent" ordering on is_tripped sufficient > to guarantee that signals won't be missed on a weak-ordering platform? > I *think* so, but an expert would need to check that code (or we > cross our fingers and wait for a hypothetical bug report). > > I think the question is: Do we know for sure that is_trippe?d has been stored using sequentially consistent ordering prior to the call to PyErr_CheckSignals(), even if an interruptible syscall is involved? I suppose so? (But this is a separate question from the problem I was solving, of course. I'm not proposing to remove PyErr_CheckSignals()) ??Koos > Regards > > Antoine. > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- + Koos Zevenhoven + http://twitter.com/k7hoven + -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Thu Nov 2 09:08:26 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 2 Nov 2017 14:08:26 +0100 Subject: [Python-ideas] Importance of noticing new signals References: <59E7D6EC.6090708@canterbury.ac.nz> <20171102132229.6e50913d@fsol> Message-ID: <20171102140826.413bbf85@fsol> On Thu, 2 Nov 2017 14:57:12 +0200 Koos Zevenhoven wrote: > > Now, is the "sequentially consistent" ordering on is_tripped sufficient > > to guarantee that signals won't be missed on a weak-ordering platform? > > I *think* so, but an expert would need to check that code (or we > > cross our fingers and wait for a hypothetical bug report). > > > > > I think the question is: Do we know for sure that is_trippe?d has been > stored using sequentially consistent ordering prior to the call to > PyErr_CheckSignals(), even if an interruptible syscall is involved? Yes, it is. See trip_signal() in Modules/signalmodule.c Regards Antoine. From levkivskyi at gmail.com Thu Nov 2 12:22:46 2017 From: levkivskyi at gmail.com (Ivan Levkivskyi) Date: Thu, 2 Nov 2017 17:22:46 +0100 Subject: [Python-ideas] install pip packages from Python prompt In-Reply-To: References: <01f701d350eb$c4ee6210$4ecb2630$@sdamon.com> <021601d350ee$88f236d0$9ad6a470$@sdamon.com> <045101d351b9$54a25240$fde6f6c0$@sdamon.com> <4409775187778125080@unknownmsgid> Message-ID: Just another random idea: What about simply having two menu items in IDLE: * Install/update package manager * Open package manager The first one will install the pipgui from PyPI (and pip if not already installed). The second one will run the GUI. This way it looks like pipgui can be simply published on PyPI without special-casing at all, or am I missing something? -- Ivan On 2 November 2017 at 11:53, Nick Coghlan wrote: > On 2 November 2017 at 18:46, Nathaniel Smith wrote: > >> But unfortuately this hasn't been standardized, and there's currently >> no way to do the lookup from the stdlib, so maybe this is not so >> helpful for IDLE... >> > > The entry point file format was recently promoted to a PyPA > interoperability spec (without a PEP, as we documented it as-is, rather > than changing anything): https://packaging.python.org/ > specifications/entry-points/ > > While the point about the standard library lacking the ability to read the > metadata for installed packages still stands, it's also not too hard to > implement a rudimentary version that just iterates over sys.path looking > for `entry_points.txt` files in `*.dist-info` subdirectories. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From erik.m.bray at gmail.com Thu Nov 2 14:56:39 2017 From: erik.m.bray at gmail.com (Erik Bray) Date: Thu, 2 Nov 2017 19:56:39 +0100 Subject: [Python-ideas] install pip packages from Python prompt In-Reply-To: <045101d351b9$54a25240$fde6f6c0$@sdamon.com> References: <01f701d350eb$c4ee6210$4ecb2630$@sdamon.com> <021601d350ee$88f236d0$9ad6a470$@sdamon.com> <045101d351b9$54a25240$fde6f6c0$@sdamon.com> Message-ID: On Oct 30, 2017 8:57 PM, "Alex Walters" wrote: > -----Original Message----- > From: Python-ideas [mailto:python-ideas-bounces+tritium- > list=sdamon.com at python.org] On Behalf Of Erik Bray > Sent: Monday, October 30, 2017 6:28 AM > To: Python-Ideas > Subject: Re: [Python-ideas] install pip packages from Python prompt > > On Sun, Oct 29, 2017 at 8:45 PM, Alex Walters > wrote: > > Then those users have more fundamental problems. There is a minimum > level > > of computer knowledge needed to be successful in programming. > Insulating > > users from the reality of the situation is not preparing them to be > > successful. Pretending that there is no system command prompt, or shell, > or > > whatever platform specific term applies, only hurts new programmers. > Give > > users an error message they can google, and they will be better off in the > > long run than they would be if we just ran pip for them. > > While I completely agree with this in principle, I think you > overestimate the average beginner. Nope. I totally get that they don?t know what a shell or command prompt is. THEY. NEED. TO. LEARN. Hiding it is not a good idea for anyone. If this is an insurmountable problem for the newbie, maybe they really shouldn?t be attempting to program. This field is not for everyone. Reading this I get the impression, and correct me if I'm wrong, that you've never taught beginners programming. Of course long term (heck in fact fairly early on) they need to learn these nitty-gritty and sometimes frustrating lessons, but not in a 2 hour intro to programming for total beginners. And I beg to differ--this field is for everyone, and increasingly moreso every day. Doesn't mean it's easy, but it is and can be for everyone. Whether this specific proposal is technically feasible in a cross-platform manner with the state of the Python interpreter and import system is another question. But that's a discussion worth having. "Some people aren't cut out for programming" isn't. > Many beginners I've taught or > helped, even if they can manage to get to the correct command prompt, > often don't even know how to run the correct Python. They might often > have multiple Pythons installed on their system--maybe they have > Anaconda, maybe Python installed by homebrew, or a Python that came > with an IDE like Spyder. If they're on OSX often running "python" > from the command prompt gives the system's crippled Python 2.6 and > they don't know the difference. > > One thing that has been a step in the right direction is moving more > documentation toward preferring running `python -m pip` over just > `pip`, since this often has a better guarantee of running `pip` in the > Python interpreter you intended. But that still requires one to know > how to run the correct Python interpreter from the command-line (which > the newbie double-clicking on IDLE may not even have a concept of...). > > While I agree this is something that is important for beginners to > learn (e.g. print(sys.executable) if in doubt), it *is* a high bar for > many newbies just to install one or two packages from pip, which they > often might need/want to do for whatever educational pursuit they're > following (heck, it's pretty common even just to want to install the > `requests` module, as I would never throw `urllib` at a beginner). > > So while I don't think anything proposed here will work technically, I > am in favor of an in-interpreter pip install functionality. Perhaps > it could work something like this: > > a) Allow it *only* in interactive mode: running `pip(...)` (or > whatever this looks like) outside of interactive mode raises a > `RuntimeError` with the appropriate documentation > b) When running `pip(...)` the user is supplied with an interactive > prompt explaining that since installing packages with `pip()` can > result in changes to the interpreter, it is necessary to restart the > interpreter after installation--give them an opportunity to cancel the > action in case they have any work they need to save. If they proceed, > install the new package then restart the interpreter for them. This > avoids any ambiguity as to states of loaded modules before/after pip > install. > > From: Stephan Houben [mailto:stephanh42 at gmail.com] > > Sent: Sunday, October 29, 2017 3:43 PM > > To: Alex Walters > > Cc: Python-Ideas > > Subject: Re: [Python-ideas] install pip packages from Python prompt > > > > > > > > Hi Alex, > > > > > > > > 2017-10-29 20:26 GMT+01:00 Alex Walters : > > > > return ?Please run pip from your system command prompt? > > > > > > > > > > > > The target audience for my proposal are people who do not know > > > > which part of the sheep the "system command prompt" is. > > > > Stephan > > > > > > > > > > > > From: Python-ideas > > [mailto:python-ideas-bounces+tritium-list=sdamon.com at python.org] On > Behalf > > Of Stephan Houben > > Sent: Sunday, October 29, 2017 3:19 PM > > To: Python-Ideas > > Subject: [Python-ideas] install pip packages from Python prompt > > > > > > > > Hi all, > > > > Here is in somewhat more detail my earlier proposal for > > > > having in the interactive Python interpreter a `pip` function to > > > > install packages from Pypi. > > > > Motivation: it appears to me that there is a category of newbies > > > > for which "open a shell and do `pip whatever`" is a bit too much. > > > > It would, in my opinion, simplify things a bit if they could just > > > > copy-and-paste some text into the Python interpreter and have > > > > some packages from pip installed. > > > > That would simplify instructions on how to install package xyz, > > > > without going into the vagaries of how to open a shell on various > > > > platforms, and how to get to the right pip executable. > > > > I think this could be as simple as: > > > > def pip(args): > > import sys > > import subprocess > > subprocess.check_call([sys.executable, "-m", "pip"] + args.split()) > > > > print("Please re-start Python now to use installed or upgraded > > packages.") > > > > Note that I added the final message about restarting the interpreter > > > > as a low-tech solution to the problem of packages being already > > > > imported in the current Python session. > > > > I would imagine that the author of package xyz would then put on > > > > their webpage something like: > > > > To use, enter in your Python interpreter: > > > > pip("install xyz --user") > > > > As another example, consider prof. Baldwin from Woolamaloo university > > > > who teaches a course "Introductory Python programming for Sheep > Shavers". > > > > In his course material, he instructs his students to execute the > > > > following line in their Python interpreter. > > > > pip("install woolamaloo-sheepshavers-goodies --user") > > > > which will install a package which will in turn, as dependencies, > > > > pull in a number of packages which are relevant for sheep shaving but > > > > which have nevertheless irresponsibly been left outside the stdlib. > > > > Stephan > > > > > > > > > > > > > > > > > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Nov 3 02:50:36 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 3 Nov 2017 16:50:36 +1000 Subject: [Python-ideas] install pip packages from Python prompt In-Reply-To: References: <01f701d350eb$c4ee6210$4ecb2630$@sdamon.com> <021601d350ee$88f236d0$9ad6a470$@sdamon.com> <045101d351b9$54a25240$fde6f6c0$@sdamon.com> <4409775187778125080@unknownmsgid> Message-ID: On 3 November 2017 at 02:22, Ivan Levkivskyi wrote: > Just another random idea: What about simply having two menu items in IDLE: > > * Install/update package manager > * Open package manager > > The first one will install the pipgui from PyPI (and pip if not already > installed). > The second one will run the GUI. > > This way it looks like pipgui can be simply published on PyPI without > special-casing at all, or am I missing something? This would also deal with the case where "ensurepip" hadn't been run at install time for some reason. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From gadgetsteve at live.co.uk Fri Nov 3 04:11:57 2017 From: gadgetsteve at live.co.uk (Steve Barnes) Date: Fri, 3 Nov 2017 08:11:57 +0000 Subject: [Python-ideas] install pip packages from Python prompt In-Reply-To: References: <01f701d350eb$c4ee6210$4ecb2630$@sdamon.com> <021601d350ee$88f236d0$9ad6a470$@sdamon.com> <045101d351b9$54a25240$fde6f6c0$@sdamon.com> Message-ID: > > Reading this I get the impression, and correct me if I'm wrong, that > you've never taught beginners programming. Of course long term (heck in > fact fairly early on) they need to learn these nitty-gritty and > sometimes frustrating lessons, but not in a 2 hour intro to programming > for total beginners. > > And I beg to differ--this field is for everyone, and increasingly moreso > every day. Doesn't mean it's easy, but it is and can be for everyone. > > Whether this specific proposal is technically feasible in a > cross-platform manner with the state of the Python interpreter and > import system is another question. But that's a discussion worth having. > "Some people aren't cut out for programming" isn't. > > When teaching an absolute beginners course, (which I personally have done many times often for people with little or no experience of computers), on of the best approaches that I have found is to run JupyterHub on a local (WiFi) LAN - this has several advantages: - the students can get started on some python programming without having to install anything on their own machines - they just need a browser. - No internet connection needed, (often the case in a corporate environment). - I have complete control over the packages present. - Once they have a taste they are ready, usually keen, to learn how to install python and it's libraries, including Jupyter. -- Steve (Gadget) Barnes Any opinions in this message are my personal opinions and do not reflect those of my employer. --- This email has been checked for viruses by AVG. http://www.avg.com From guido at python.org Fri Nov 3 10:35:29 2017 From: guido at python.org (Guido van Rossum) Date: Fri, 3 Nov 2017 07:35:29 -0700 Subject: [Python-ideas] Moving typing out of the stdlib in Python 3.7? Message-ID: [A copy from https://github.com/python/typing/issues/495 to get more people's attention to this issue.] I'm wondering if we should remove typing from the stdlib. Now's the time to think about this, as the feature freeze for 3.7 is about 12 weeks away. Cons: - People have to depend on a PyPI package to use typing (but they do anyway for typing_extensions) - It's a backward incompatibility for users of Python 3.5 and 3.6 (but the typing module was always provisional) Pros: - The typing module can evolve much faster outside the stdlib - We could get rid of typing_extensions (and maybe even mypy_extensions) If we don't do this I worry that we're entering a period where many new typesystem features end up in typing_extensions and users will be confused about which items are in typing and which in typing_extensions (not to mention mypy_extensions). Anything new to be added to typing (e.g. Const, Final, Literal, or changing ABCs to Protocols) would have to be added to typing_extensions instead, and users would be confused about which features exist in which module. Moving typing out of the stdlib can make things potentially simpler, at the cost of an extra pip install (but they'll need one anyway for mypy). Thoughts? -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri Nov 3 10:45:11 2017 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 4 Nov 2017 01:45:11 +1100 Subject: [Python-ideas] Moving typing out of the stdlib in Python 3.7? In-Reply-To: References: Message-ID: On Sat, Nov 4, 2017 at 1:35 AM, Guido van Rossum wrote: > [A copy from https://github.com/python/typing/issues/495 to get more > people's attention to this issue.] > > I'm wondering if we should remove typing from the stdlib. Now's the time to > think about this, as the feature freeze for 3.7 is about 12 weeks away. > > Cons: > > People have to depend on a PyPI package to use typing (but they do anyway > for typing_extensions) If the lazy evaluation of annotations (PEP 563) also lands in 3.7, then this would be a very minor downside. You'd need to pip-install typing as well as mypy *for the actual type checking*, but at run time, you could ignore both (all those List[...] annotations would be stored unevaluated). Otherwise, it'd mean that any project that makes use of type hints would require typing as a run-time dependency. ChrisA From guido at python.org Fri Nov 3 10:49:13 2017 From: guido at python.org (Guido van Rossum) Date: Fri, 3 Nov 2017 07:49:13 -0700 Subject: [Python-ideas] Moving typing out of the stdlib in Python 3.7? In-Reply-To: References: Message-ID: On Fri, Nov 3, 2017 at 7:45 AM, Chris Angelico wrote: > On Sat, Nov 4, 2017 at 1:35 AM, Guido van Rossum wrote: > > [A copy from https://github.com/python/typing/issues/495 to get more > > people's attention to this issue.] > > > > I'm wondering if we should remove typing from the stdlib. Now's the time > to > > think about this, as the feature freeze for 3.7 is about 12 weeks away. > > > > Cons: > > > > People have to depend on a PyPI package to use typing (but they do anyway > > for typing_extensions) > > If the lazy evaluation of annotations (PEP 563) also lands in 3.7, > then this would be a very minor downside. You'd need to pip-install > typing as well as mypy *for the actual type checking*, but at run > time, you could ignore both (all those List[...] annotations would be > stored unevaluated). Otherwise, it'd mean that any project that makes > use of type hints would require typing as a run-time dependency. > This would not work if you use TypeVar, NewType, or any kind of type alias involving things imported from typing (e.g. Union or TypedDict). Also cast(). -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri Nov 3 10:51:32 2017 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 4 Nov 2017 01:51:32 +1100 Subject: [Python-ideas] Moving typing out of the stdlib in Python 3.7? In-Reply-To: References: Message-ID: On Sat, Nov 4, 2017 at 1:49 AM, Guido van Rossum wrote: > On Fri, Nov 3, 2017 at 7:45 AM, Chris Angelico wrote: >> >> On Sat, Nov 4, 2017 at 1:35 AM, Guido van Rossum wrote: >> > [A copy from https://github.com/python/typing/issues/495 to get more >> > people's attention to this issue.] >> > >> > I'm wondering if we should remove typing from the stdlib. Now's the time >> > to >> > think about this, as the feature freeze for 3.7 is about 12 weeks away. >> > >> > Cons: >> > >> > People have to depend on a PyPI package to use typing (but they do >> > anyway >> > for typing_extensions) >> >> If the lazy evaluation of annotations (PEP 563) also lands in 3.7, >> then this would be a very minor downside. You'd need to pip-install >> typing as well as mypy *for the actual type checking*, but at run >> time, you could ignore both (all those List[...] annotations would be >> stored unevaluated). Otherwise, it'd mean that any project that makes >> use of type hints would require typing as a run-time dependency. > > > This would not work if you use TypeVar, NewType, or any kind of type alias > involving things imported from typing (e.g. Union or TypedDict). Also > cast(). Ah, I forgot about those - they're not just used in the annotations. Oh well, was a nice idea. ChrisA From sf at fermigier.com Fri Nov 3 11:04:43 2017 From: sf at fermigier.com (=?UTF-8?Q?St=C3=A9fane_Fermigier?=) Date: Fri, 3 Nov 2017 16:04:43 +0100 Subject: [Python-ideas] Moving typing out of the stdlib in Python 3.7? In-Reply-To: References: Message-ID: I use typing quite systematically nowadays, even for projects that don't use mypy (for historical reasons: bringing an older code base to zero mypy issues can be quite time-consuming). For instance, adding typing annotation can help autocompletion under PyCharm (and hopefully other IDEs). With these annotations, PyCharm is also able to signal typing issues either directly in the editor, or when running a code check. I'm quite OK with removing the typing module from the stdlib as it can easily be added to my projects dependencies, and I can definitively understand the benefits of a faster release cycle, but I'm worried that this could hinder adoption of these practices by certain people. S. On Fri, Nov 3, 2017 at 3:35 PM, Guido van Rossum wrote: > [A copy from https://github.com/python/typing/issues/495 to get more > people's attention to this issue.] > > I'm wondering if we should remove typing from the stdlib. Now's the time > to think about this, as the feature freeze for 3.7 is about 12 weeks away. > > Cons: > > - People have to depend on a PyPI package to use typing (but they do > anyway for typing_extensions) > - It's a backward incompatibility for users of Python 3.5 and 3.6 (but > the typing module was always provisional) > > Pros: > > - The typing module can evolve much faster outside the stdlib > - We could get rid of typing_extensions (and maybe even mypy_extensions > ) > > If we don't do this I worry that we're entering a period where many new > typesystem features end up in typing_extensions and users will be > confused about which items are in typing and which in typing_extensions > (not to mention mypy_extensions). Anything new to be added to typing > (e.g. Const, Final, Literal, or changing ABCs to Protocols) would have to > be added to typing_extensions instead, and users would be confused about > which features exist in which module. Moving typing out of the stdlib can > make things potentially simpler, at the cost of an extra pip install (but > they'll need one anyway for mypy). > > Thoughts? > > -- > --Guido van Rossum (python.org/~guido) > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier - http://linkedin.com/in/sfermigier Founder & CEO, Abilian - Enterprise Social Software - http://www.abilian.com/ Chairman, Free&OSS Group / Systematic Cluster - http://www.gt-logiciel-libre.org/ Co-Chairman, National Council for Free & Open Source Software (CNLL) - http://cnll.fr/ Founder & Organiser, PyData Paris - http://pydata.fr/ --- ?You never change things by ?ghting the existing reality. To change something, build a new model that makes the existing model obsolete.? ? R. Buckminster Fuller -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Fri Nov 3 11:20:31 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 3 Nov 2017 16:20:31 +0100 Subject: [Python-ideas] Moving typing out of the stdlib in Python 3.7? References: Message-ID: <20171103162031.1573dcf5@fsol> On Fri, 3 Nov 2017 16:04:43 +0100 St?fane Fermigier wrote: > I use typing quite systematically nowadays, even for projects that don't > use mypy (for historical reasons: bringing an older code base to zero mypy > issues can be quite time-consuming). > > For instance, adding typing annotation can help autocompletion under > PyCharm (and hopefully other IDEs). > > With these annotations, PyCharm is also able to signal typing issues either > directly in the editor, or when running a code check. > > I'm quite OK with removing the typing module from the stdlib as it can > easily be added to my projects dependencies, and I can definitively > understand the benefits of a faster release cycle, but I'm worried that > this could hinder adoption of these practices by certain people. I don't think casual or beginner users of Python really have to worry about typing annotations. As I understand it, they become really useful on middle- to large-scale projects (disclaimer: I've never used them myself; the kind of typing the Numba project does -- which I don't participate in anymore -- is quite different). Regards Antoine. From k7hoven at gmail.com Fri Nov 3 11:22:07 2017 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Fri, 3 Nov 2017 17:22:07 +0200 Subject: [Python-ideas] Importance of noticing new signals In-Reply-To: References: <59E7D6EC.6090708@canterbury.ac.nz> <20171102132229.6e50913d@fsol> Message-ID: On Thu, Nov 2, 2017 at 2:57 PM, Koos Zevenhoven wrote: > On Thu, Nov 2, 2017 at 2:22 PM, Antoine Pitrou > wrote: > >> On Wed, 1 Nov 2017 20:29:56 +0200 >> Koos Zevenhoven wrote: >> > >> > ?From a correctness point of view, that is absolutely great: if >> > PyErr_CheckSignals() is called, it is guaranteed to notice a new signal >> > regardles of how small the number of picoseconds after the `is_tripped` >> > flag has been set. But is that really important? >> >> I was going to answer "no"... but actually yes. The important case is >> the event loop type case: >> >> while (1) { >> select([some file descriptors]); >> if (errno == EINTR) { >> PyErr_CheckSignals(); >> if (PyErr_Occurred()) break; >> } >> /* continue select()ing... */ >> } >> >> Now say at a given point in time, no fds are actually active (or even >> waited for), but some signal arrives (SIGINT perhaps). >> select() is woken up and returns with errno EINTR. Two things then can >> happen: >> >> - if PyErr_CheckSignals() notices the signal, it will run the relevant >> signal handler, which may raise an exception and trigger the select() >> loop to exit (e.g. SIGINT would raise KeyboardInterrupt) >> >> - if PyErr_CheckSignals() misses the signal, the loop will enter again, >> and select() may sleep for an infinite amount of time >> >> > Oh! So that would provide a proper reason for my just-in-case decision to > name the faster near-equivalent functionality PyErr_PROBE_SIGNALS instead > of PyErr_CHECK_SIGNALS. > > Cross-referencing to that (thread about making Ctrl-C "always" work): > > https://mail.python.org/pipermail/python-ideas/2017-November/047631.html > > > Of course, what we're doing with select() above can already apply for >> read() or other interruptible syscalls waiting for outside data... and >> that pattern is present a lot internally, especially since >> ?? >> PEP 475 ("Retry system calls failing with EINTR"). >> > >> Now, is the "sequentially consistent" ordering on is_tripped sufficient >> to guarantee that signals won't be missed on a weak-ordering platform? >> I *think* so, but an expert would need to check that code (or we >> cross our fingers and wait for a hypothetical bug report). >> >> > I think the question is: Do we know for sure that is_trippe?d has been > stored using sequentially consistent ordering prior to the call to > PyErr_CheckSignals(), even if an interruptible syscall is involved? I > suppose so? > > (But this is a separate question from the problem I was solving, of > course. I'm not proposing to remove PyErr_CheckSignals()) > > To continue on this: If I understand your question correctly, I'm hesitant to make strong statements about it. It would be interesting to know what we can assume about signals that happen at the same time with system calls, given the various platforms supported. Unfortunately, I don't know that. Maybe you are concerned about whether some nuances and recent changes to signal handling could lead to harmful change in behavior in some meaningful edge cases? I can at least say that my PyErr_PROBE_SIGNALS() proposal does not introduce such issues, if the difference is documented properly: """PyErr_PROBE_SIGNALS() is meant for performance-critical code and is not 100% guaranteed to always see the most recent signals. If a signal being deferred is a concern, use PyErr_CheckSignals() instead.""" But more generally, if we could assume that trip_signal() and PyErr_CheckSignals() always happen in the same "CPU thread", then we wouldn't need pyatomic.h here at all. The fact that the code currently assumes that all Python signal handlers should run in the same Python thread takes care of some of these concerns without needing locks etc. Some other concerns I can imagine by looking at some of the code in Modules/signalmodule.c: (1) If trip_signal() and PyErr_CheckSignals() are executed concurrently, trip_signal() might set a new signal flag (using relaxed memory order) while PyErr_CheckSignals is still running. Then if PyErr_CheckSignals() sets is_tripped to zero *after* trip_signal() sets it to 1, then the new signal might be deferred until the next time *some* new signal arrives, which could take an arbitrarily long amount of time, I suppose. However, it looks like this problem has been solved by always setting is_tripped to zero (with strict SEQ_CST memory order) *before* handling the individual signals. So if trip_signal() has already set is_tripped to 1 (with SEQ_CST), that prevents PyErr_CheckSignals from setting is_tripped to zero (with SEQ_CST) *and not* handling the signal. If trip_signal() has not yet finished, and therefore not set is_tripped to 1 yet, it will cause the next call to PyErr_CheckSignals to catch the signal. (2) Again, if trip_signal() and PyErr_CheckSignals() execute concurrently, it might happen that PyErr_CheckSignals() handles the signal *before* trip_signal sets is_tripped to 1. That would cause the next call to PyErr_CheckSignals() to think there's an unhandled signal, but will most likely not find one, because it was already handled on the previous call. But that just effectively means that nothing is done. In fact, there's a comment in the code that mentions this. (3, 4, ...) Of course there's more to take care of there, but that's unrelated to my PyErr_PROBE_SIGNALS() proposal. Anyway, at least (1) and (2) seem to already have been taken care of, and I assume you are aware of that. ??Koos -- + Koos Zevenhoven + http://twitter.com/k7hoven + -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Fri Nov 3 11:29:47 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 3 Nov 2017 16:29:47 +0100 Subject: [Python-ideas] Importance of noticing new signals References: <59E7D6EC.6090708@canterbury.ac.nz> <20171102132229.6e50913d@fsol> Message-ID: <20171103162947.7f9cfcdb@fsol> On Fri, 3 Nov 2017 17:22:07 +0200 Koos Zevenhoven wrote: > > Maybe you are concerned about whether some nuances and recent changes to > signal handling could lead to harmful change in behavior in some meaningful > edge cases? I can at least say that my PyErr_PROBE_SIGNALS() proposal does > not introduce such issues, if the difference is documented properly: > > """PyErr_PROBE_SIGNALS() is meant for performance-critical code and is not > 100% guaranteed to always see the most recent signals. If a signal being > deferred is a concern, use PyErr_CheckSignals() instead.""" Agreed. > But more generally, if we could assume that trip_signal() and > PyErr_CheckSignals() always happen in the same "CPU thread", then we > wouldn't need pyatomic.h here at all. The fact that the code currently > assumes that all Python signal handlers should run in the same Python > thread takes care of some of these concerns without needing locks etc. Hmm... *Python* signal handlers (registered with signal.signal()) all run in the same thread, but we have no way to ensure to ensure that trip_signal() (which is a C-level signal handler called by the OS) will run in the same thread. Even if we were to mask all signals in non-main threads started by Python (e.g. with threading.Thread), there can still be threads created by third-party C libraries that we don't have any control over. > Some other concerns I can imagine by looking at some of the code in > Modules/signalmodule.c: > > (1) If trip_signal() and PyErr_CheckSignals() are executed concurrently, > trip_signal() might set a new signal flag (using relaxed memory order) > while PyErr_CheckSignals is still running. Then if PyErr_CheckSignals() > sets is_tripped to zero *after* trip_signal() sets it to 1, then the new > signal might be deferred until the next time *some* new signal arrives, > which could take an arbitrarily long amount of time, I suppose. > > However, it looks like this problem has been solved by always setting > is_tripped to zero (with strict SEQ_CST memory order) *before* handling the > individual signals. So if trip_signal() has already set is_tripped to 1 > (with SEQ_CST), that prevents PyErr_CheckSignals from setting is_tripped to > zero (with SEQ_CST) *and not* handling the signal. If trip_signal() has not > yet finished, and therefore not set is_tripped to 1 yet, it will cause the > next call to PyErr_CheckSignals to catch the signal. > > (2) Again, if trip_signal() and PyErr_CheckSignals() execute concurrently, > it might happen that PyErr_CheckSignals() handles the signal *before* > trip_signal sets is_tripped to 1. That would cause the next call to > PyErr_CheckSignals() to think there's an unhandled signal, but will most > likely not find one, because it was already handled on the previous call. > But that just effectively means that nothing is done. In fact, there's a > comment in the code that mentions this. > > (3, 4, ...) Of course there's more to take care of there, but that's > unrelated to my PyErr_PROBE_SIGNALS() proposal. Anyway, at least (1) and > (2) seem to already have been taken care of, and I assume you are aware of > that. Yes, that was my analysis too (and why I implemented it this way in the first place). But I'm glad you confirm it, since on such topics it's always easy to miss a potential problem. Regards Antoine. From turnbull.stephen.fw at u.tsukuba.ac.jp Sat Nov 4 03:31:32 2017 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Sat, 4 Nov 2017 16:31:32 +0900 Subject: [Python-ideas] install pip packages from Python prompt In-Reply-To: References: <01f701d350eb$c4ee6210$4ecb2630$@sdamon.com> <021601d350ee$88f236d0$9ad6a470$@sdamon.com> <045101d351b9$54a25240$fde6f6c0$@sdamon.com> Message-ID: <23037.27860.762729.984015@turnbull.sk.tsukuba.ac.jp> Erik Bray writes: > Nope. I totally get that they don?t know what a shell or command prompt > is. THEY. NEED. TO. LEARN. I don't want to take a position on the proposal, and I agree that we should *strongly* encourage everyone to learn. But "THEY. NEED. TO. LEARN." is not obvious to me. Anecdotally, my students are doing remarkably (to me, as a teacher) complex modeling with graphical interfaces to statistical and simulation packages (SPSS/AMOS, Artisoc, respectively), and collection of large textual databases from SNS with cargo-culted Python programs. For the past twenty years teaching social scientists, these accidental barriers (as Fred Brooks would have called them) have dropped dramatically, to the point where it's possible to do superficially good-looking (= complex) but entirely meaningless :-/ empirical research. (In some ways I think this lowered cost has been horribly detrimental to my work as an educator in applied social science. ;-) The point being that "user-friendly" UI in many fields where (fairly) advanced computing is used is more than keeping up with the perceived needs of most computer users, while the essential (in the sense of Brooks) non-computing modeling difficulties of their jobs remain. By "perceived" I mean I want my students using TeX, but it's hard to force them when all their professors (except me and a couple mathematicians) use Word (speaking of irreproducible results). It's good enough for government work, and that's in fact where many of them end up (and the great majority are either in government or in equivalent corporate bureaucrat positions). Yes, I meant the deprecatory connotations of "perceived", but realistically, I admit that maybe they *don't* *need* the more polished tech that I could teach them. > Hiding it is not a good idea for anyone. Agreed. Command lines and REPLs teach humility, to me as well as my students. :-) Steve -- Associate Professor Division of Policy and Planning Science http://turnbull/sk.tsukuba.ac.jp/ Faculty of Systems and Information Email: turnbull at sk.tsukuba.ac.jp University of Tsukuba Tel: 029-853-5175 Tennodai 1-1-1, Tsukuba 305-8573 JAPAN From erik.m.bray at gmail.com Sat Nov 4 06:07:23 2017 From: erik.m.bray at gmail.com (Erik Bray) Date: Sat, 4 Nov 2017 11:07:23 +0100 Subject: [Python-ideas] install pip packages from Python prompt In-Reply-To: <23037.27860.762729.984015@turnbull.sk.tsukuba.ac.jp> References: <01f701d350eb$c4ee6210$4ecb2630$@sdamon.com> <021601d350ee$88f236d0$9ad6a470$@sdamon.com> <045101d351b9$54a25240$fde6f6c0$@sdamon.com> <23037.27860.762729.984015@turnbull.sk.tsukuba.ac.jp> Message-ID: On Nov 4, 2017 08:31, "Stephen J. Turnbull" < turnbull.stephen.fw at u.tsukuba.ac.jp> wrote: Erik Bray writes: > Nope. I totally get that they don?t know what a shell or command prompt > is. THEY. NEED. TO. LEARN. Just to be clear I did not write this. Someone replying to me did. I'm going to go over all the different proposals in this thread and see if I can synthesize a list of options. I think, even if it's not a solution that winds up in the stdlib, it would be good to have some user stories about how package installation from within an interactive prompt might work (even if not from the standard REPL, which it should be noted has had small improvements made to it over the years). I also have my doubts about whether this *shouldn't* be possible. I mean, to a lot of beginners starting out the basic REPL *is* Python. They're so new to the scene they don't even know what IPython or Jupyter is or why they might want that. They aren't experienced enough to even know what they're missing out on. In classrooms we can resolve that easily by pointing our students to whatever tools we think will work best for them, but not everyone has that privilege. Best, Erik I don't want to take a position on the proposal, and I agree that we should *strongly* encourage everyone to learn. But "THEY. NEED. TO. LEARN." is not obvious to me. Anecdotally, my students are doing remarkably (to me, as a teacher) complex modeling with graphical interfaces to statistical and simulation packages (SPSS/AMOS, Artisoc, respectively), and collection of large textual databases from SNS with cargo-culted Python programs. For the past twenty years teaching social scientists, these accidental barriers (as Fred Brooks would have called them) have dropped dramatically, to the point where it's possible to do superficially good-looking (= complex) but entirely meaningless :-/ empirical research. (In some ways I think this lowered cost has been horribly detrimental to my work as an educator in applied social science. ;-) The point being that "user-friendly" UI in many fields where (fairly) advanced computing is used is more than keeping up with the perceived needs of most computer users, while the essential (in the sense of Brooks) non-computing modeling difficulties of their jobs remain. By "perceived" I mean I want my students using TeX, but it's hard to force them when all their professors (except me and a couple mathematicians) use Word (speaking of irreproducible results). It's good enough for government work, and that's in fact where many of them end up (and the great majority are either in government or in equivalent corporate bureaucrat positions). Yes, I meant the deprecatory connotations of "perceived", but realistically, I admit that maybe they *don't* *need* the more polished tech that I could teach them. I remember when I first started out teaching Software Carpentry I made the embarrassing mistake (coming from Physics) of assuming that LaTex is de-facto in most other academic fields :) > Hiding it is not a good idea for anyone. Agreed. Command lines and REPLs teach humility, to me as well as my students. :-) Steve -- Associate Professor Division of Policy and Planning Science http://turnbull/sk.tsukuba.ac.jp/ Faculty of Systems and Information Email: turnbull at sk.tsukuba.ac.jp University of Tsukuba Tel: 029-853-5175 Tennodai 1-1-1, Tsukuba 305-8573 JAPAN -------------- next part -------------- An HTML attachment was scrubbed... URL: From tds333 at mailbox.org Sat Nov 4 07:25:57 2017 From: tds333 at mailbox.org (tds333 at mailbox.org) Date: Sat, 4 Nov 2017 12:25:57 +0100 (CET) Subject: [Python-ideas] Proposal to change Python version release cycle Message-ID: <45619991.35926.1509794757663@office.mailbox.org> Hello, one of my long standing ideas to improve Python is to adjust the release cycle and version number handling. In short, to simplify it. This is the first draft of the idea: Proposal to change Python version release cycle =============================================== Goal ---- Increment the major version number more frequently for every new language feature release. Change Python feature version counting from "major.minor" to "major". This is simpler, better to handle compatibility and allows adding new library features more frequently. The current Python version scheme is Major.Minor.Patch this will be kept but the major version number will be incremented faster and for every language feature release as it is now for the minor number. For example today the minor version number is increased for every big release done in 1,5 year cycle. In this new language features with new keywords can appear, new libraries can be included and other C level API changes can happen. There backward compatible feature improvements and new features also backward incompatible can happen. I suggest to change this to increment the major version for every new release of the 1,5 year cycle. And allow new Python standard library backward compatible changes for every minor release cycle every 6 months. Start with this from Python version 4 on. Benefit ------- Clear separation of new language features and C API changes and new backward compatible feature additions. So for a major version on Python level new keywords can be added, build ins can be added or removed and the C API can be changed (new features or other incompatible changes). For a minor version new standard libraries can be added and API can be improved in a backward compatible way. This can be done more frequently for example in a 0,5 year cycle. Allowing the Python library part to improve in a higher rate. For a patch version only none breaking bug and security fixes are allowed. (Same as it is now, or even a little bit stricter) With this change there is a clear separation of new language Features which are also tracked by other Python implementations and new library features. This can be seen very easy by only looking at the Python major version. The C API level is included in this standard and is not allowed to change in a minor version as it is now. This is because booth the Python language and the C API of CPython implementation is a standard and tracked by other implementations. For them it is easy to say I am compatible to Version 4. For everyone in computer business it is clear what is meant. The Python standard library can be improved in a higher rate because backward compatible feature changes are allowed in a higher rate for minor versions. Also provisional libraries can be improved in a faster rate for every minor version. (for example every 6 months instead of 1,5 years) For Unix in the Python executable the major version can be appended and this specifies the Python language standard used. This is done already but most of the time only EXECUTABLEmajor.minor is useful. Also more complicated to handle as a simple Number. Can be problematic on some platforms because of the dot ".". Python major version every 1,5 years (same as it is now for major.minor). Python minor version every 6 months (to allow faster standard library changes). Python patch version as needed (to fix only bugs and security related stuff) This can solve the problem of adding something as a standard library and then this is the dead of the library because it cannot be improved any further in a higher update rate. The Python standard library should not be a dead end for a library after inclusion. This allows Python code to improve faster than the C code. This is needed to include new features faster and because we have more people capable in Python programming than for C in the Python community. On C level the compatibility distinction is simple only track the major version and no longer a mixture between major and major.minor compatibility. Risk ---- Minimal because to track features the major.minor version already has to be tracked. It is a change some assumptions about Python versions are at this point no longer valid. On C level external libraries and wrappers must adopt it and changes are needed. Open questions -------------- The exact release cycle for minor, can also be every 4 months if needed. Clear separation what is part of the language standard and what is part of the C API. Some modules, for example "sys" can also be part of the language Standard and should not change between minor versions. The difference between the stable C API and that for every minor version can be removed. There is only one C API guarantee for a major version and it does not change for a minor version. None goals ---------- Change the release cycle of 1,5 years for new big releases. Change the versioning scheme to something other than "major.minor.patch". New big ground breaking changes to everything. To complicate something. Regards, Wolfgang From rosuav at gmail.com Sat Nov 4 07:35:50 2017 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 4 Nov 2017 22:35:50 +1100 Subject: [Python-ideas] Proposal to change Python version release cycle In-Reply-To: <45619991.35926.1509794757663@office.mailbox.org> References: <45619991.35926.1509794757663@office.mailbox.org> Message-ID: On Sat, Nov 4, 2017 at 10:25 PM, wrote: > I suggest to change this to increment the major version for every new release > of the 1,5 year cycle. > And allow new Python standard library backward compatible changes for every > minor release cycle every 6 months. The usual implication of a major version bump is that there is significant incompatibility. That's something that should happen roughly once a decade, not every couple of years. Massive -1.0 from me on this. ChrisA From tds333 at mailbox.org Sat Nov 4 08:00:59 2017 From: tds333 at mailbox.org (Wolfgang) Date: Sat, 4 Nov 2017 13:00:59 +0100 Subject: [Python-ideas] Proposal to change Python version release cycle In-Reply-To: References: <45619991.35926.1509794757663@office.mailbox.org> Message-ID: <856f31c2-556b-0382-81e0-3caaaa4c3163@mailbox.org> On 04.11.2017 12:35, Chris Angelico wrote: > On Sat, Nov 4, 2017 at 10:25 PM, wrote: >> I suggest to change this to increment the major version for every new release >> of the 1,5 year cycle. >> And allow new Python standard library backward compatible changes for every >> minor release cycle every 6 months. > > The usual implication of a major version bump is that there is > significant incompatibility. That's something that should happen > roughly once a decade, not every couple of years. Massive -1.0 from me > on this. > Yes that implication is a possible risk. I will add it to the section if needed. As noted by Guido a ground breaking change as it happened from version 2 to 3 should never happen in Python land again and a possible version 4 will simply be after 3.9. Other languages increment their major version already faster and track the language standard simply with the major version. The major version should be incremented for a ground breaking change but an increment requires not a ground breaking change to do this. For me even adding new keywords to a language is a major change. (Such as async and await) Or changes in the C API. And talking about language standard version 4 is simpler than a more complicated scheme of major.minor. Also other implementations such as pypy can use this first version number in their name to show compatibility. (pypy4 compatible to Python ) I think if the changes happens it will take some time for people to adjust but then life will be simpler. :-) Thank you for your feedback. Regards, Wolfgang From rosuav at gmail.com Sat Nov 4 08:54:56 2017 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 4 Nov 2017 23:54:56 +1100 Subject: [Python-ideas] Proposal to change Python version release cycle In-Reply-To: <856f31c2-556b-0382-81e0-3caaaa4c3163@mailbox.org> References: <45619991.35926.1509794757663@office.mailbox.org> <856f31c2-556b-0382-81e0-3caaaa4c3163@mailbox.org> Message-ID: On Sat, Nov 4, 2017 at 11:00 PM, Wolfgang wrote: > > > On 04.11.2017 12:35, Chris Angelico wrote: >> >> On Sat, Nov 4, 2017 at 10:25 PM, wrote: >>> >>> I suggest to change this to increment the major version for every new >>> release >>> of the 1,5 year cycle. >>> And allow new Python standard library backward compatible changes for >>> every >>> minor release cycle every 6 months. >> >> >> The usual implication of a major version bump is that there is >> significant incompatibility. That's something that should happen >> roughly once a decade, not every couple of years. Massive -1.0 from me >> on this. >> > > Yes that implication is a possible risk. I will add it to the section > if needed. > > As noted by Guido a ground breaking change as it happened from version 2 to > 3 should never happen in Python land again and a possible version 4 will > simply be after 3.9. > Other languages increment their major version already faster and track > the language standard simply with the major version. > > The major version should be incremented for a ground breaking change > but an increment requires not a ground breaking change to do this. > > For me even adding new keywords to a language is a major change. > (Such as async and await) > Or changes in the C API. > And talking about language standard version 4 is simpler than a > more complicated scheme of major.minor. > Also other implementations such as pypy can use this first version > number in their name to show compatibility. > (pypy4 compatible to Python ) > > I think if the changes happens it will take some time for people to > adjust but then life will be simpler. :-) > > Thank you for your feedback. If I write code for Python 3.4 and try to run it on Python 3.7, there is a small probability that it will break (eg if it tries to use "await" as a function name - I had a program like that, and once the new keywords were pushed through, I renamed it to "wait" to ensure compatibility). But it's a fairly small chance, and it's usually pretty easy to write code that's compatible with several minor versions (by targeting the oldest and then testing on the newer ones). With major version changes, it's much harder to be compatible with both, and you often end up with compatibility shims (eg "try: input = raw_input; except NameError: pass"). This applies to code, to documentation, to answers on Stack Overflow, to blogs, and to everything else that discusses Python in any way. At work, we have to use certain JavaScript packages that have had, I kid you not, more than one major version bump per year since initial release. One time we only discovered an issue after a student installed the latest version of something, and the setup instructions didn't work. Oh, great, version 10 breaks things compared to version 9. I do NOT enjoy dealing with that kind of incompatibility, and permitting it every 1.5 years is a bad thing. And if compatibility is to be guaranteed, why bump the major version number? That's the whole point of the three-part version. ChrisA From solipsis at pitrou.net Sat Nov 4 09:29:53 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 4 Nov 2017 14:29:53 +0100 Subject: [Python-ideas] Proposal to change Python version release cycle References: <45619991.35926.1509794757663@office.mailbox.org> Message-ID: <20171104142953.0821b213@fsol> Hello Wolfgang, On Sat, 4 Nov 2017 12:25:57 +0100 (CET) tds333 at mailbox.org wrote: > Hello, > > one of my long standing ideas to improve Python is to adjust the > release cycle and version number handling. In short, to simplify it. There has been ample discussion in the past about changing our release cycle one way or another. In short, the meta-problem is there are many contradicting interests which would each favour a different solution to the problem. See for example this PEP (ultimately rejected): https://www.python.org/dev/peps/pep-0407/ and the discussion that ensued: https://mail.python.org/pipermail/python-dev/2012-January/thread.html#115591 I haven't read your proposal in detail, but I suspect that it may be vulnerable to some of the same objections. The big objection being that a significant part of our ecosystem (that is, to put it roughly, the more corporate-minded segment, though I believe it is a simplification and may include other segments, such as Debian stable users and maintainers) doesn't want to deal more frequent feature releases. Regards Antoine. > > This is the first draft of the idea: > > Proposal to change Python version release cycle > =============================================== > > > Goal > ---- > > Increment the major version number more frequently for every new language > feature release. > Change Python feature version counting from "major.minor" to "major". > > This is simpler, better to handle compatibility and allows adding new > library features more frequently. > > The current Python version scheme is Major.Minor.Patch this will be kept > but the major version number will be incremented faster and for every > language feature release as it is now for the minor number. > > For example today the minor version number is increased for every big release > done in 1,5 year cycle. In this new language features with new keywords can > appear, new libraries can be included and other C level API changes can happen. > There backward compatible feature improvements and new features also backward > incompatible can happen. > > I suggest to change this to increment the major version for every new release > of the 1,5 year cycle. > And allow new Python standard library backward compatible changes for every > minor release cycle every 6 months. > > Start with this from Python version 4 on. > > > Benefit > ------- > > Clear separation of new language features and C API changes and new > backward compatible feature additions. > > So for a major version on Python level new keywords can be added, build ins > can be added or removed and the C API can be changed (new features or other > incompatible changes). > > For a minor version new standard libraries can be added and API can be improved > in a backward compatible way. This can be done more frequently for example in > a 0,5 year cycle. Allowing the Python library part to improve in a higher rate. > > For a patch version only none breaking bug and security fixes are allowed. > (Same as it is now, or even a little bit stricter) > > With this change there is a clear separation of new language Features which > are also tracked by other Python implementations and new library features. > This can be seen very easy by only looking at the Python major version. > The C API level is included in this standard and is not allowed to change > in a minor version as it is now. This is because booth the Python language > and the C API of CPython implementation is a standard and tracked by other > implementations. For them it is easy to say I am compatible to Version 4. > For everyone in computer business it is clear what is meant. > > The Python standard library can be improved in a higher rate because backward > compatible feature changes are allowed in a higher rate for minor versions. > Also provisional libraries can be improved in a faster rate for every minor > version. (for example every 6 months instead of 1,5 years) > > For Unix in the Python executable the major version can be appended and this > specifies the Python language standard used. This is done already but most of > the time only EXECUTABLEmajor.minor is useful. Also more complicated to handle > as a simple Number. Can be problematic on some platforms because of the dot ".". > > Python major version every 1,5 years (same as it is now for major.minor). > Python minor version every 6 months (to allow faster standard library changes). > Python patch version as needed (to fix only bugs and security related stuff) > > This can solve the problem of adding something as a standard library and then > this is the dead of the library because it cannot be improved any further > in a higher update rate. > The Python standard library should not be a dead end for a library after > inclusion. This allows Python code to improve faster than the C code. > This is needed to include new features faster and because we have more > people capable in Python programming than for C in the Python community. > > On C level the compatibility distinction is simple only track the major > version and no longer a mixture between major and major.minor compatibility. > > > Risk > ---- > > Minimal because to track features the major.minor version already has to be > tracked. > It is a change some assumptions about Python versions are at this point no > longer valid. > On C level external libraries and wrappers must adopt it and changes are > needed. > > > Open questions > -------------- > > The exact release cycle for minor, can also be every 4 months if needed. > Clear separation what is part of the language standard and what is part > of the C API. > Some modules, for example "sys" can also be part of the language Standard > and should not change between minor versions. > > The difference between the stable C API and that for every minor version > can be removed. There is only one C API guarantee for a major version > and it does not change for a minor version. > > > None goals > ---------- > > Change the release cycle of 1,5 years for new big releases. > Change the versioning scheme to something other than "major.minor.patch". > New big ground breaking changes to everything. > To complicate something. > > > Regards, > > Wolfgang > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From ncoghlan at gmail.com Sat Nov 4 10:05:33 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 5 Nov 2017 00:05:33 +1000 Subject: [Python-ideas] Moving typing out of the stdlib in Python 3.7? In-Reply-To: References: Message-ID: On 4 November 2017 at 00:35, Guido van Rossum wrote: > [A copy from https://github.com/python/typing/issues/495 to get more > people's attention to this issue.] > > I'm wondering if we should remove typing from the stdlib. Now's the time to > think about this, as the feature freeze for 3.7 is about 12 weeks away. > > Cons: > > People have to depend on a PyPI package to use typing (but they do anyway > for typing_extensions) > It's a backward incompatibility for users of Python 3.5 and 3.6 (but the > typing module was always provisional) > > Pros: > > The typing module can evolve much faster outside the stdlib > We could get rid of typing_extensions (and maybe even mypy_extensions) > > If we don't do this I worry that we're entering a period where many new > typesystem features end up in typing_extensions and users will be confused > about which items are in typing and which in typing_extensions (not to > mention mypy_extensions). Anything new to be added to typing (e.g. Const, > Final, Literal, or changing ABCs to Protocols) would have to be added to > typing_extensions instead, and users would be confused about which features > exist in which module. Moving typing out of the stdlib can make things > potentially simpler, at the cost of an extra pip install (but they'll need > one anyway for mypy). > > Thoughts? Perhaps typing could switch to being a bundled module, such that it had its own version, independent of the Python standard library version, but was still present by default in new installations? Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From wes.turner at gmail.com Sat Nov 4 10:16:08 2017 From: wes.turner at gmail.com (Wes Turner) Date: Sat, 4 Nov 2017 10:16:08 -0400 Subject: [Python-ideas] install pip packages from Python prompt In-Reply-To: References: <01f701d350eb$c4ee6210$4ecb2630$@sdamon.com> <021601d350ee$88f236d0$9ad6a470$@sdamon.com> <045101d351b9$54a25240$fde6f6c0$@sdamon.com> <23037.27860.762729.984015@turnbull.sk.tsukuba.ac.jp> Message-ID: On Saturday, November 4, 2017, Erik Bray wrote: > On Nov 4, 2017 08:31, "Stephen J. Turnbull" tsukuba.ac.jp > > > wrote: > > Erik Bray writes: > > > Nope. I totally get that they don?t know what a shell or command prompt > > is. THEY. NEED. TO. LEARN. > > > Just to be clear I did not write this. Someone replying to me did. > > I'm going to go over all the different proposals in this thread and see if > I can synthesize a list of options. > Here's are some of the challenges (both for a pip() function and a GUI): - Create reproducible software environments - Log the pip CLI commands - Log whether a version was specified - Be able to replay that log: - Which inputs need to be escaped before passing them through to a shell or exec*? (see: shlex.split()) - Please don't flood PyPi with unnecessary requests - (Turns out there are 5 invocations of `!pip install' buried somewhere in a script that's run 5 times in a hour by 25 students: even with pip caching package downloads, how many HTTP requests is that? Who pays for PyPi?) - There are a number of GUIs for pip: web-based, GUI based, CLI wrappers: - web: - CLI wrappers: pipenv, pipsi; ``conda install pip`` - GUI: PyCharm - Tcl/Tk is in the standard library. - ``conda install spyder`` installs At - pip does not have a stable API. - not even pip.main() - what about the shell CLI? - sarge is still safer than shlex.split() > > I think, even if it's not a solution that winds up in the stdlib, it > would be good to have some user stories about how package installation from > within an interactive prompt might work (even if not from the standard > REPL, which it should be noted has had small improvements made to it over > the years). > This works today but is inadvisable because it risks wasting bandwidth if the %run (!) instructions are frequently re-run unnecessarily: $ ipython >>> !pip install -U ipython >>> exit() $ ipython > I also have my doubts about whether this *shouldn't* be possible. I mean, > to a lot of beginners starting out the basic REPL *is* Python. They're so > new to the scene they don't even know what IPython or Jupyter is or why > they might want that. They aren't experienced enough to even know what > they're missing out on. In classrooms we can resolve that easily by > pointing our students to whatever tools we think will work best for them, > but not everyone has that privilege. > https://conda.io/docs/user-guide/install/index.html $ conda install -y pip ipython notebook spyder Or, just: $ conda install -y ipython notebook > > Best, > Erik > > I don't want to take a position on the proposal, and I agree that we > should *strongly* encourage everyone to learn. But "THEY. NEED. TO. > LEARN." is not obvious to me. > > Anecdotally, my students are doing remarkably (to me, as a teacher) > complex modeling with graphical interfaces to statistical and > simulation packages (SPSS/AMOS, Artisoc, respectively), and collection > of large textual databases from SNS with cargo-culted Python programs. > For the past twenty years teaching social scientists, these accidental > barriers (as Fred Brooks would have called them) have dropped > dramatically, to the point where it's possible to do superficially > good-looking (= complex) but entirely meaningless :-/ empirical > research. (In some ways I think this lowered cost has been horribly > detrimental to my work as an educator in applied social science. ;-) > > The point being that "user-friendly" UI in many fields where (fairly) > advanced computing is used is more than keeping up with the perceived > needs of most computer users, while the essential (in the sense of > Brooks) non-computing modeling difficulties of their jobs remain. > > By "perceived" I mean I want my students using TeX, but it's hard to > force them when all their professors (except me and a couple > mathematicians) use Word (speaking of irreproducible results). It's > good enough for government work, and that's in fact where many of them > end up (and the great majority are either in government or in > equivalent corporate bureaucrat positions). Yes, I meant the > deprecatory connotations of "perceived", but realistically, I admit > that maybe they *don't* *need* the more polished tech that I could > teach them. > > > I remember when I first started out teaching Software Carpentry I made the > embarrassing mistake (coming from Physics) of assuming that LaTex is > de-facto in most other academic fields :) > > > Hiding it is not a good idea for anyone. > > Agreed. Command lines and REPLs teach humility, to me as well as my > students. :-) > > Steve > > > -- > Associate Professor Division of Policy and Planning Science > http://turnbull/sk.tsukuba.ac.jp/ Faculty of Systems and Information > Email: turnbull at sk.tsukuba.ac.jp > > University of Tsukuba > Tel: 029-853-5175 Tennodai 1-1-1, Tsukuba 305-8573 JAPAN > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tds333 at mailbox.org Sat Nov 4 10:40:09 2017 From: tds333 at mailbox.org (Wolfgang) Date: Sat, 4 Nov 2017 15:40:09 +0100 Subject: [Python-ideas] Proposal to change Python version release cycle In-Reply-To: <20171104142953.0821b213@fsol> References: <45619991.35926.1509794757663@office.mailbox.org> <20171104142953.0821b213@fsol> Message-ID: <8dd82d18-5828-2356-ee4f-14f6cc9dc14e@mailbox.org> On 04.11.2017 14:29, Antoine Pitrou wrote: > > Hello Wolfgang, > > On Sat, 4 Nov 2017 12:25:57 +0100 (CET) > tds333 at mailbox.org wrote: >> Hello, >> >> one of my long standing ideas to improve Python is to adjust the >> release cycle and version number handling. In short, to simplify it. > > There has been ample discussion in the past about changing our release > cycle one way or another. In short, the meta-problem is there are many > contradicting interests which would each favour a different solution to > the problem. See for example this PEP (ultimately rejected): > https://www.python.org/dev/peps/pep-0407/ > > and the discussion that ensued: > https://mail.python.org/pipermail/python-dev/2012-January/thread.html#115591 > > I haven't read your proposal in detail, but I suspect that it may be > vulnerable to some of the same objections. The big objection being > that a significant part of our ecosystem (that is, to put it roughly, > the more corporate-minded segment, though I believe it is a > simplification and may include other segments, such as Debian stable > users and maintainers) doesn't want to deal more frequent feature > releases. I read this PEP and some of the discussion. The difference to my idea is not to propose a LTS version and feature preview releases. The simplest form of my change is to switch from today major.minor to simply major for the feature release cycle. Additional minor feature releases for the standard library can also be optional or limited to provisional parts of the standard library. In all cases they should be backward compatible. But have the option to change the Python parts faster than the core. My idea came up because I have seen the need to improve the standard library in a faster pace than the core language. And to show this also in the version number. Hopefully ending the if it is included into the Python standard library it will be dead. Resulting in more releases but for the core with the same release cycle as of now. The amount of minor releases is not high it will be kept lower as it is now. Another point is to show by a higher major version number how mature Python is. (marketing) ;-) For Linux distributions there is still the option to include a specific Python Version with a feature set and recommend it. If someone wants to be conservative it is save to rely on features only for the first major version. This will be the same as of now for major.minor. Regards, Wolfgang From ncoghlan at gmail.com Sat Nov 4 10:48:33 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 5 Nov 2017 00:48:33 +1000 Subject: [Python-ideas] Proposal to change Python version release cycle In-Reply-To: <20171104142953.0821b213@fsol> References: <45619991.35926.1509794757663@office.mailbox.org> <20171104142953.0821b213@fsol> Message-ID: On 4 November 2017 at 23:29, Antoine Pitrou wrote: > > Hello Wolfgang, > > On Sat, 4 Nov 2017 12:25:57 +0100 (CET) > tds333 at mailbox.org wrote: >> Hello, >> >> one of my long standing ideas to improve Python is to adjust the >> release cycle and version number handling. In short, to simplify it. > > There has been ample discussion in the past about changing our release > cycle one way or another. In short, the meta-problem is there are many > contradicting interests which would each favour a different solution to > the problem. See for example this PEP (ultimately rejected): > https://www.python.org/dev/peps/pep-0407/ > > and the discussion that ensued: > https://mail.python.org/pipermail/python-dev/2012-January/thread.html#115591 PEP 413 was another competing proposal from around the same time: https://www.python.org/dev/peps/pep-0413/ Technically neither proposal has been formally Rejected, but they're also Deferred/Withdrawn because we knew that if we *had* asked for an explicit determination, the answer would have been "No, too high a cost, for not enough of a demonstrated benefit" (and that hasn't really changed in the past 5 years). > I haven't read your proposal in detail, but I suspect that it may be > vulnerable to some of the same objections. The big objection being > that a significant part of our ecosystem (that is, to put it roughly, > the more corporate-minded segment, though I believe it is a > simplification and may include other segments, such as Debian stable > users and maintainers) doesn't want to deal more frequent feature > releases. It's not just redistributors that get affected - it's the compatibility testing matrices for community projects as well. With our current release cycles (and excluding the 2.7 LTS release from consideration) we tend to have the following situation for the 3.x branches: - 1 actively maintained version - 2 or 3 security-fix only branches This stems from the default commitment of security fixes ending around 5 years after an X.Y.0 release, and releases happening 18-24 months apart: * Month 0: X.Y.0 released * Month 18: X.(Y+1).0 released * Month 36: X.(Y+2).0 released * Month 54: X.(Y+3).0 released * Month 60: X.Y.Z goes end-of-life * Month 72: X.(Y+4).0 released That same pattern also held for the 2.x series until the 2.7 release in 2010 (and it only changed then because that was the last 2.x feature release). The current year or so where the number of supported branches drops down to only 3 means we have some scope to speed up the rate of feature releases a bit (i.e. a release cadence of every 15 months instead of every 18 still means the community compatibility testing matrix consistently stays around 4 versions, or 5 if you're also testing against the dev release), but going to 6 monthly version updates means folks will either keep their test matrices the same as they are now (increasing the odds of the standard-library-only updates breaking things and that not being detected for some time), or else feeling obliged to expand their test matrices to cover even more concurrently supported versions. There are potential ways around both of those problems, but they require making the defined support periods for minor releases significantly shorter, unless they're the last minor release before the next major release (which means that instead of being simpler, these kinds of arrangements tend to end up being *more* complicated than the "every feature release receives security updates for 5 years" status quo). So rather than making fundamental changes to the way Python is versioned just to make standard library updates more readily available on older base versions of Python, we've instead gone with a more needs driven approach: making backport modules available for older versions where we find it is beneficial to do so (e.g. contextlib2, unittest2, importlib2). While Python 2.7 is a major driver for that change in practice, the benefits apply to older 3.x releases as well: - the updated modules are available for all Python versions and implementations where the packaging tools work (which is most of them) - users can decide on a module-by-module basis whether they want the baseline version or fast track updates - those decisions are explicitly tracked in the project's dependency metadata, rather than being implicit in the choice of deployment platform Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sat Nov 4 11:01:50 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 5 Nov 2017 01:01:50 +1000 Subject: [Python-ideas] Proposal to change Python version release cycle In-Reply-To: <8dd82d18-5828-2356-ee4f-14f6cc9dc14e@mailbox.org> References: <45619991.35926.1509794757663@office.mailbox.org> <20171104142953.0821b213@fsol> <8dd82d18-5828-2356-ee4f-14f6cc9dc14e@mailbox.org> Message-ID: On 5 November 2017 at 00:40, Wolfgang wrote: > > > On 04.11.2017 14:29, Antoine Pitrou wrote: >> >> >> Hello Wolfgang, >> >> On Sat, 4 Nov 2017 12:25:57 +0100 (CET) >> tds333 at mailbox.org wrote: >>> >>> Hello, >>> >>> one of my long standing ideas to improve Python is to adjust the >>> release cycle and version number handling. In short, to simplify it. >> >> >> There has been ample discussion in the past about changing our release >> cycle one way or another. In short, the meta-problem is there are many >> contradicting interests which would each favour a different solution to >> the problem. See for example this PEP (ultimately rejected): >> https://www.python.org/dev/peps/pep-0407/ >> >> and the discussion that ensued: >> >> https://mail.python.org/pipermail/python-dev/2012-January/thread.html#115591 >> >> I haven't read your proposal in detail, but I suspect that it may be >> vulnerable to some of the same objections. The big objection being >> that a significant part of our ecosystem (that is, to put it roughly, >> the more corporate-minded segment, though I believe it is a >> simplification and may include other segments, such as Debian stable >> users and maintainers) doesn't want to deal more frequent feature >> releases. > > > I read this PEP and some of the discussion. > > The difference to my idea is not to propose a LTS version and feature > preview releases. > > The simplest form of my change is to switch from today major.minor to simply > major for the feature release cycle. We're currently more likely to go the other direction, and stick with the 3.x numbering for an extended period (potentially reaching 3.10, 3.11, 3.12, etc), so that the ongoing disruption caused by the 2.x -> 3.x transition has had a chance to settle down before we ask anyone to start calling anything "Python 4". While the problems Linux distros are facing with whether "python" should refer to "python2" or "python3" are at least arguably self-inflicted (especially for those that already dealt with the Python 1.5.2 -> Python 2.0 migration back in the early 2000s), nobody is really looking forward to having to figure out how to adjust to a potential future Python 4.0 that invalidates all the current "python3" based naming schemes that have been introduced to work around the fact that Python 3.x needed to be parallel installable with Python 2.x without breaking any existing Python 2.x applications. We're not even sure yet when we're going to update PEP 394 to say that we think it's reasonable for distros to start using "python" to mean "python3" (see https://www.python.org/dev/peps/pep-0394/ and https://github.com/python/redistributor-guide/issues/1 for more discussion on the latter point). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From tds333 at mailbox.org Sat Nov 4 11:10:32 2017 From: tds333 at mailbox.org (Wolfgang) Date: Sat, 4 Nov 2017 16:10:32 +0100 Subject: [Python-ideas] Proposal to change Python version release cycle In-Reply-To: References: <45619991.35926.1509794757663@office.mailbox.org> <20171104142953.0821b213@fsol> Message-ID: <150c4391-644f-b451-9224-6c4b8bfc8e3d@mailbox.org> Hi Nick, On 04.11.2017 15:48, Nick Coghlan wrote: > On 4 November 2017 at 23:29, Antoine Pitrou wrote: >> >> Hello Wolfgang, >> >> On Sat, 4 Nov 2017 12:25:57 +0100 (CET) >> tds333 at mailbox.org wrote: >>> Hello, >>> >>> one of my long standing ideas to improve Python is to adjust the >>> release cycle and version number handling. In short, to simplify it. >> >> There has been ample discussion in the past about changing our release >> cycle one way or another. In short, the meta-problem is there are many >> contradicting interests which would each favour a different solution to >> the problem. See for example this PEP (ultimately rejected): >> https://www.python.org/dev/peps/pep-0407/ >> >> and the discussion that ensued: >> https://mail.python.org/pipermail/python-dev/2012-January/thread.html#115591 > > PEP 413 was another competing proposal from around the same time: > https://www.python.org/dev/peps/pep-0413/ > > Technically neither proposal has been formally Rejected, but they're > also Deferred/Withdrawn because we knew that if we *had* asked for an > explicit determination, the answer would have been "No, too high a > cost, for not enough of a demonstrated benefit" (and that hasn't > really changed in the past 5 years). > >> I haven't read your proposal in detail, but I suspect that it may be >> vulnerable to some of the same objections. The big objection being >> that a significant part of our ecosystem (that is, to put it roughly, >> the more corporate-minded segment, though I believe it is a >> simplification and may include other segments, such as Debian stable >> users and maintainers) doesn't want to deal more frequent feature >> releases. > > It's not just redistributors that get affected - it's the > compatibility testing matrices for community projects as well. > > With our current release cycles (and excluding the 2.7 LTS release > from consideration) we tend to have the following situation for the > 3.x branches: > > - 1 actively maintained version > - 2 or 3 security-fix only branches > > This stems from the default commitment of security fixes ending around > 5 years after an X.Y.0 release, and releases happening 18-24 months > apart: > > * Month 0: X.Y.0 released > * Month 18: X.(Y+1).0 released > * Month 36: X.(Y+2).0 released > * Month 54: X.(Y+3).0 released > * Month 60: X.Y.Z goes end-of-life > * Month 72: X.(Y+4).0 released > > That same pattern also held for the 2.x series until the 2.7 release > in 2010 (and it only changed then because that was the last 2.x > feature release). > > The current year or so where the number of supported branches drops > down to only 3 means we have some scope to speed up the rate of > feature releases a bit (i.e. a release cadence of every 15 months > instead of every 18 still means the community compatibility testing > matrix consistently stays around 4 versions, or 5 if you're also > testing against the dev release), but going to 6 monthly version > updates means folks will either keep their test matrices the same as > they are now (increasing the odds of the standard-library-only updates > breaking things and that not being detected for some time), or else > feeling obliged to expand their test matrices to cover even more > concurrently supported versions. There are potential ways around both > of those problems, but they require making the defined support periods > for minor releases significantly shorter, unless they're the last > minor release before the next major release (which means that instead > of being simpler, these kinds of arrangements tend to end up being > *more* complicated than the "every feature release receives security > updates for 5 years" status quo). > > So rather than making fundamental changes to the way Python is > versioned just to make standard library updates more readily available > on older base versions of Python, we've instead gone with a more needs > driven approach: making backport modules available for older versions > where we find it is beneficial to do so (e.g. contextlib2, unittest2, > importlib2). > > While Python 2.7 is a major driver for that change in practice, the > benefits apply to older 3.x releases as well: > > - the updated modules are available for all Python versions and > implementations where the packaging tools work (which is most of them) > - users can decide on a module-by-module basis whether they want the > baseline version or fast track updates > - those decisions are explicitly tracked in the project's dependency > metadata, rather than being implicit in the choice of deployment > platform > > Cheers, > Nick. > This is a valid concern. So more pressure by more releases can be a problem. For my idea the only possibility to limit this is to have minor versions only for the provisional parts to allow improvements and to have the major version track the core features and the bug fixing only for the first major version. Resulting in something like 4.0.0, 4.0.1 and 4.1.0 for something like a feature update for provisional but not supported with bug fixing and not for compatibility testing. This is then something like PEP-407 which is rejected. And also makes things little more complicated. Another possibility is to change only the versioning to major.minor instead of major.minor.patch. Then having a simpler versioning scheme for other Python implementations as only benefit (and the simplification to spell compatibility). Thus major is for the compatibility and minor counts the bug fix releases. Thinking if a change then is needed. :-) Marketing benefits? Don't know then. Regards, Wolfgang From solipsis at pitrou.net Sat Nov 4 11:20:15 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 4 Nov 2017 16:20:15 +0100 Subject: [Python-ideas] Proposal to change Python version release cycle References: <45619991.35926.1509794757663@office.mailbox.org> <20171104142953.0821b213@fsol> <150c4391-644f-b451-9224-6c4b8bfc8e3d@mailbox.org> Message-ID: <20171104162015.34fa3b7d@fsol> On Sat, 4 Nov 2017 16:10:32 +0100 Wolfgang wrote: > > Another possibility is to change only the versioning > to major.minor instead of major.minor.patch. Then having > a simpler versioning scheme for other Python implementations > as only benefit (and the simplification to spell compatibility). I think the versioning scheme is a red herring here. The important topic is when Python releases are cut and what kinds of changes they are made of. > Thinking if a change then is needed. :-) Marketing benefits? > Don't know then. I don't think there are any marketing benefits. That might have been true 10 years ago, but nowadays people are so used to staggering increases in version numbers that they are not impressed anymore. Besides, IMHO Python is well past the point where it needed a concerted marketing effort to promote itself. Regards Antoine. From tds333 at mailbox.org Sat Nov 4 11:29:30 2017 From: tds333 at mailbox.org (Wolfgang) Date: Sat, 4 Nov 2017 16:29:30 +0100 Subject: [Python-ideas] Proposal to change Python version release cycle In-Reply-To: References: <45619991.35926.1509794757663@office.mailbox.org> <20171104142953.0821b213@fsol> <8dd82d18-5828-2356-ee4f-14f6cc9dc14e@mailbox.org> Message-ID: <5bc30002-d6a4-5fad-f1e0-96e2d74a059e@mailbox.org> Hi Nick, On 04.11.2017 16:01, Nick Coghlan wrote: > On 5 November 2017 at 00:40, Wolfgang wrote: >> >> >> On 04.11.2017 14:29, Antoine Pitrou wrote: >>> >>> >>> Hello Wolfgang, >>> >>> On Sat, 4 Nov 2017 12:25:57 +0100 (CET) >>> tds333 at mailbox.org wrote: >>>> >>>> Hello, >>>> >>>> one of my long standing ideas to improve Python is to adjust the >>>> release cycle and version number handling. In short, to simplify it. >>> >>> >>> There has been ample discussion in the past about changing our release >>> cycle one way or another. In short, the meta-problem is there are many >>> contradicting interests which would each favour a different solution to >>> the problem. See for example this PEP (ultimately rejected): >>> https://www.python.org/dev/peps/pep-0407/ >>> >>> and the discussion that ensued: >>> >>> https://mail.python.org/pipermail/python-dev/2012-January/thread.html#115591 >>> >>> I haven't read your proposal in detail, but I suspect that it may be >>> vulnerable to some of the same objections. The big objection being >>> that a significant part of our ecosystem (that is, to put it roughly, >>> the more corporate-minded segment, though I believe it is a >>> simplification and may include other segments, such as Debian stable >>> users and maintainers) doesn't want to deal more frequent feature >>> releases. >> >> >> I read this PEP and some of the discussion. >> >> The difference to my idea is not to propose a LTS version and feature >> preview releases. >> >> The simplest form of my change is to switch from today major.minor to simply >> major for the feature release cycle. > > We're currently more likely to go the other direction, and stick with > the 3.x numbering for an extended period (potentially reaching 3.10, > 3.11, 3.12, etc), so that the ongoing disruption caused by the 2.x -> > 3.x transition has had a chance to settle down before we ask anyone to > start calling anything "Python 4". A good point but two digits minor version numbers have the possibility to break a lot code. There is a lot of stuff out where a single digit major version is assumed. Even the official Python build for windows with python27.dll, python36.dll can be problematic because the dot is omitted between numbers. Other do the same for compilation they concatenate only majorminor for a name. Then version 3.10 is the same as 31.0. Ok I know this will take a while. But for the first 2.7.10 version even there some other library code was broken because of such assumptions. > While the problems Linux distros are facing with whether "python" > should refer to "python2" or "python3" are at least arguably > self-inflicted (especially for those that already dealt with the > Python 1.5.2 -> Python 2.0 migration back in the early 2000s), nobody > is really looking forward to having to figure out how to adjust to a > potential future Python 4.0 that invalidates all the current "python3" > based naming schemes that have been introduced to work around the fact > that Python 3.x needed to be parallel installable with Python 2.x > without breaking any existing Python 2.x applications. We're not even > sure yet when we're going to update PEP 394 to say that we think it's > reasonable for distros to start using "python" to mean "python3" (see > https://www.python.org/dev/peps/pep-0394/ and > https://github.com/python/redistributor-guide/issues/1 for more > discussion on the latter point). I know on some build computers I have the same problem. But solved it for my scripts by using the major number as compatibility specifier and it works fine for me. Also knowing it is not upward compatible. The whole problem can only be solved by something used for Windows where the Python launcher deals with this. A Python launcher (py) for Linux can solve this there. Allowing also distributions to configure a fall back. For example the user specified python3 but I know it will be compatible with python4 so the launcher can do something like python3 is not found then I use python4 if available. ex.: #!/usr/bin/env py -3 And handle it as configurable version hint. Or even allowing alternative implementation like pypy if compatible. Regards, Wolfgang From tds333 at mailbox.org Sat Nov 4 14:10:09 2017 From: tds333 at mailbox.org (Wolfgang) Date: Sat, 4 Nov 2017 19:10:09 +0100 Subject: [Python-ideas] Proposal to change Python version release cycle In-Reply-To: References: <45619991.35926.1509794757663@office.mailbox.org> <20171104142953.0821b213@fsol> <8dd82d18-5828-2356-ee4f-14f6cc9dc14e@mailbox.org> Message-ID: <249133f3-f083-19be-d75a-64a3cf7f8fec@mailbox.org> On 04.11.2017 16:44, M.-A. Lemburg wrote: > Just to clarify: Python 2.0 was called 2.0 because the BeOpen marketing > department thought it was good idea, not because there were major > incompatible changes going into that release. > > Porting code from Python 1.5.2 to 2.0 was relatively straight forward > and not much different from other minor releases. > > The first ever major backwards incompatibility release switch we > had in Python after the great renaming of the C APIs between > Python 1.1 and 1.2 (which was only visible to C extensions and > relatively easy to fix using a compatibility header file), > was the transition from Python 2.x to Python 3.x. > > IMO, the only reason to do this again would be the removal of > the GIL, but only if there's absolutely no other way to get > around such breakage. > > I would very much appreciate Python's releases not to introduce > such major changes anymore. A continuous gradual change is a lot > better to handle than non-continuous breaks and I'm pretty > sure both will get us to the same level functionality eventually, > perhaps even avoiding a few new mistakes along the way. > > With that approach, the versioning scheme would just be an > implementation detail, which is good. Whether we call it Python > 4.2 or Python 42 is really not all that important. Python > is mature enough to not have to pull marketing tricks anymore. Thanks for the interesting details. Have to admit I entered the Python land with version 1.4 and missed the C API change from 1.1 to 1.2 completely. :-) Getting more opinions about the idea confirms me slowly to something like, good to discuss but not worth the change. Because some arguments confirmed me that it is not worth to change this. Also the marketing argument is weak and you are right that it should not care us. The only remaining argument is to have shorter release cycles but I think they don't depend on the versioning scheme nor a change in it improves this. Looking forward to 2020 when 2.7 is out of business and the maintenance cost can be invested in Python 4 and a faster release schedule. :-) Regards, Wolfgang From rosuav at gmail.com Sat Nov 4 13:09:41 2017 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 5 Nov 2017 04:09:41 +1100 Subject: [Python-ideas] Proposal to change Python version release cycle In-Reply-To: <5bc30002-d6a4-5fad-f1e0-96e2d74a059e@mailbox.org> References: <45619991.35926.1509794757663@office.mailbox.org> <20171104142953.0821b213@fsol> <8dd82d18-5828-2356-ee4f-14f6cc9dc14e@mailbox.org> <5bc30002-d6a4-5fad-f1e0-96e2d74a059e@mailbox.org> Message-ID: On Sun, Nov 5, 2017 at 2:29 AM, Wolfgang wrote: > A good point but two digits minor version numbers have the possibility > to break a lot code. There is a lot of stuff out where a single digit > major version is assumed. Even the official Python build for windows > with python27.dll, python36.dll can be problematic because the dot > is omitted between numbers. > Other do the same for compilation they concatenate only majorminor for a > name. > Then version 3.10 is the same as 31.0. > Ok I know this will take a while. Python 1.0 was released in 1994. Then 2.0 came out in 2000 (six years), and 3.0 in 2008 (eight years). So far, we've been nine years into 3.0 and aren't looking at 4.0 yet, so it's going to be at least ten. If version 5.0 is another twelve years after that, and version 6.0 is fourteen later, we'll be looking for version 31.0 some time around the year 3000. I think we can let a future generation worry about pathing problems with DLL names. Seriously, I don't think there's any need to stress about version 31 of something as stable as Python. No matter how long it *actually* is between versions, it's going to be a lot longer than we can really plan for right now. In that much time, *anything* could change. ChrisA From desmoulinmichel at gmail.com Sat Nov 4 13:09:57 2017 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Sat, 4 Nov 2017 18:09:57 +0100 Subject: [Python-ideas] Moving typing out of the stdlib in Python 3.7? In-Reply-To: References: Message-ID: I could see the typing module staying but: - typing_extension (or the new official external module) append types in the typing module; - if typing_extension is not installed or too old and missing some types, any missing type being accessed returns some kind of mock object that avoid crashing; - any running type checker encountering mocked types has to show a warning - you can pip install typing (and it's mocking abilities) in python 2. This way somebody without the 3rd party module can still use the code without crashing it. Having types out of date or missing a brand new one won't crash the code, only have a warning in mypy or pycharm. This solves the compatibility problem, but also another major problem currently: several projects I worked on had a fake typing module I could use it in python 2 and 3 without having to manually install typing. Sometime I also create scripts and leave type annotations, and my users really don't want to install the exact typing extension to make it work. And I don't want to remove the annotations manually after ward. Le 03/11/2017 ? 15:51, Chris Angelico a ?crit?: > On Sat, Nov 4, 2017 at 1:49 AM, Guido van Rossum wrote: >> On Fri, Nov 3, 2017 at 7:45 AM, Chris Angelico wrote: >>> >>> On Sat, Nov 4, 2017 at 1:35 AM, Guido van Rossum wrote: >>>> [A copy from https://github.com/python/typing/issues/495 to get more >>>> people's attention to this issue.] >>>> >>>> I'm wondering if we should remove typing from the stdlib. Now's the time >>>> to >>>> think about this, as the feature freeze for 3.7 is about 12 weeks away. >>>> >>>> Cons: >>>> >>>> People have to depend on a PyPI package to use typing (but they do >>>> anyway >>>> for typing_extensions) >>> >>> If the lazy evaluation of annotations (PEP 563) also lands in 3.7, >>> then this would be a very minor downside. You'd need to pip-install >>> typing as well as mypy *for the actual type checking*, but at run >>> time, you could ignore both (all those List[...] annotations would be >>> stored unevaluated). Otherwise, it'd mean that any project that makes >>> use of type hints would require typing as a run-time dependency. >> >> >> This would not work if you use TypeVar, NewType, or any kind of type alias >> involving things imported from typing (e.g. Union or TypedDict). Also >> cast(). > > Ah, I forgot about those - they're not just used in the annotations. > Oh well, was a nice idea. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From mal at egenix.com Sat Nov 4 11:44:08 2017 From: mal at egenix.com (M.-A. Lemburg) Date: Sat, 4 Nov 2017 16:44:08 +0100 Subject: [Python-ideas] Proposal to change Python version release cycle In-Reply-To: References: <45619991.35926.1509794757663@office.mailbox.org> <20171104142953.0821b213@fsol> <8dd82d18-5828-2356-ee4f-14f6cc9dc14e@mailbox.org> Message-ID: Just to clarify: Python 2.0 was called 2.0 because the BeOpen marketing department thought it was good idea, not because there were major incompatible changes going into that release. Porting code from Python 1.5.2 to 2.0 was relatively straight forward and not much different from other minor releases. The first ever major backwards incompatibility release switch we had in Python after the great renaming of the C APIs between Python 1.1 and 1.2 (which was only visible to C extensions and relatively easy to fix using a compatibility header file), was the transition from Python 2.x to Python 3.x. IMO, the only reason to do this again would be the removal of the GIL, but only if there's absolutely no other way to get around such breakage. I would very much appreciate Python's releases not to introduce such major changes anymore. A continuous gradual change is a lot better to handle than non-continuous breaks and I'm pretty sure both will get us to the same level functionality eventually, perhaps even avoiding a few new mistakes along the way. With that approach, the versioning scheme would just be an implementation detail, which is good. Whether we call it Python 4.2 or Python 42 is really not all that important. Python is mature enough to not have to pull marketing tricks anymore. Cheers, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Nov 04 2017) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From prometheus235 at gmail.com Sat Nov 4 15:33:38 2017 From: prometheus235 at gmail.com (Nick Timkovich) Date: Sat, 4 Nov 2017 14:33:38 -0500 Subject: [Python-ideas] Proposal to change Python version release cycle In-Reply-To: References: <45619991.35926.1509794757663@office.mailbox.org> <20171104142953.0821b213@fsol> <8dd82d18-5828-2356-ee4f-14f6cc9dc14e@mailbox.org> Message-ID: On Sat, Nov 4, 2017 at 10:44 AM, M.-A. Lemburg wrote: > Just to clarify: Python 2.0 was called 2.0 because the BeOpen marketing > department thought it was good idea, not because there were major > incompatible changes going into that release. > Alternative history question: if it was just 1.6, then would Python 3000, probably called "Python 2000/2.x" in that world, with all the fixes have happened sooner because you would have "ran out" (yes, v1.10 > v1.9) of numbers leading up to 2? > With that approach, the versioning scheme would just be an > implementation detail, which is good. Whether we call it Python > 4.2 or Python 42 is really not all that important. Python > is mature enough to not have to pull marketing tricks anymore. But of course we'll skip Python 9, just like iPhones and Windowses ;) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sat Nov 4 16:22:25 2017 From: guido at python.org (Guido van Rossum) Date: Sat, 4 Nov 2017 13:22:25 -0700 Subject: [Python-ideas] Moving typing out of the stdlib in Python 3.7? In-Reply-To: References: Message-ID: On Sat, Nov 4, 2017 at 7:05 AM, Nick Coghlan wrote: > Perhaps typing could switch to being a bundled module, such that it > had its own version, independent of the Python standard library > version, but was still present by default in new installations? > This is beginning to sound like the most attractive solution. We could possibly do away with typing_extensions. Are there precedents of how to bundle a module in this way? Or is it going to be another special case like pip? -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Nov 4 23:07:05 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 5 Nov 2017 13:07:05 +1000 Subject: [Python-ideas] Proposal to change Python version release cycle In-Reply-To: <5bc30002-d6a4-5fad-f1e0-96e2d74a059e@mailbox.org> References: <45619991.35926.1509794757663@office.mailbox.org> <20171104142953.0821b213@fsol> <8dd82d18-5828-2356-ee4f-14f6cc9dc14e@mailbox.org> <5bc30002-d6a4-5fad-f1e0-96e2d74a059e@mailbox.org> Message-ID: On 5 November 2017 at 01:29, Wolfgang wrote: > On 04.11.2017 16:01, Nick Coghlan wrote: >> We're currently more likely to go the other direction, and stick with >> the 3.x numbering for an extended period (potentially reaching 3.10, >> 3.11, 3.12, etc), so that the ongoing disruption caused by the 2.x -> >> 3.x transition has had a chance to settle down before we ask anyone to >> start calling anything "Python 4". > > A good point but two digits minor version numbers have the possibility > to break a lot code. There is a lot of stuff out where a single digit > major version is assumed. Even the official Python build for windows > with python27.dll, python36.dll can be problematic because the dot > is omitted between numbers. > Other do the same for compilation they concatenate only majorminor for a > name. > Then version 3.10 is the same as 31.0. > Ok I know this will take a while. > But for the first 2.7.10 version even there some other library code > was broken because of such assumptions. Aye, we're aware :) We're not in a situation where we'll have any *good* options available, so the question we're considering is "What do we think will be the least bad approach?". At our current release cadence, 3.7 will be in mid 2018, 3.8 in late 2019 or early 2020, and then 3.9 in mid 2021. The open question will be what we call the release after that (in late 2022), with the main leading contenders being "4.0" (on the grounds that so many changes will have accumulated since 2008's 3.0 release by then that it makes sense to call them different major versions, similar to the rationale the Linux kernel now uses for major version updates), and "3.10" (on the grounds that the release won't be any more different from 3.9 than 3.9 will be from 3.8). Other variants (like making the major release number "40" or "2022", and using the minor version field for some new purpose other than indicating compatibility breaks in the compiler AST, interpreter opcode format, code caching tags, and C ABI), tend to introduce the same pragmatic questions that will already need to be considered in choosing between the 3.10 and 4.0 alternatives. It's not just the version numbering aesthetics that need to be considered either - in addition to the point you raise around how the Python version gets embedded in file path names (such that the 3-digit form is technically ambiguous without a separator, and may break parsers that are expecting exactly two digits), there are other backwards compatibility concerns around how folks do version compatibility checks based on sys.version and sys.version_info. Python and CPython will be more than 3 decades old by 2022, and an ecosystem can build up a *lot* of implicit assumptions regarding how a platform's versioning scheme works in that kind of time frame :) Personally, I doubt we'll make a firm decision on how we're going to handle this until some time after 2.7 goes EOL in 2020, as by then we'll necessarily have a better idea of how the various Linux distros (including the LTS commercial ones) ended up handling the Python 2 -> Python 3 switch, and the fact that the next release at that point will be 3.9 making it eminently clear that we've run out of time to continue postponing the question. We'll hopefully also have more experience with folks relying on the stable runtime ABI, and hence whether or not it might be beneficial to define new "py4" and "abi4" compatibility tags for the binary wheel distribution format. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sat Nov 4 23:46:59 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 5 Nov 2017 13:46:59 +1000 Subject: [Python-ideas] Moving typing out of the stdlib in Python 3.7? In-Reply-To: References: Message-ID: On 5 November 2017 at 06:22, Guido van Rossum wrote: > On Sat, Nov 4, 2017 at 7:05 AM, Nick Coghlan wrote: >> >> Perhaps typing could switch to being a bundled module, such that it >> had its own version, independent of the Python standard library >> version, but was still present by default in new installations? > > > This is beginning to sound like the most attractive solution. We could > possibly do away with typing_extensions. Are there precedents of how to > bundle a module in this way? Or is it going to be another special case like > pip? You'd be blazing new trails, but you'd be blazing them with a single-file pure Python module without any binary dependencies outside CPython itself, which is the simplest possible precedent setting scenario we could hope for. I can think of a couple of possible approaches you could take, depending on how you want to manage development on the typing module itself. = External dev = * Typing moves entirely out of the CPython source tree, with a separate issue tracker, repository, branch structure, etc (which you already have) * We add a "_bundled" directory to the CPython source tree in 3.7+ and put a typing sdist archive there * the release process PEP gains a "Confirm the bundled modules are up to date" step * the CPython build process is updated to use a venv to generate wheel files from bundled sdists * regrtest gains a "bundled" resource (and/or dedicated command line option) * when asked to test bundled modules, regrtest spins up a test venv, installs pip and the bundled modules, then runs the tests for those modules * ensurepip gains the ability to also install bundled wheel files * the Windows and Mac OS X installers start including the typing wheel file = In tree dev = * Similar to external dev for testing and distribution * Source code would stay in the main CPython tree and track CPython branch structure * The sdist file would be built by CPython's build process, rather than being checked in The external dev model is probably a better fit for your use case, though, since you'd like something that you can readily keep consistent across all supported 3.x versions (or at least 3.7+), and that's harder to do when you have multiple copies of the source code to keep in sync (vs having a single development branch where you run your CI testing across all supported Python versions). The external dev option also establishes a more useful precedent, since we could use it to officially migrate ongoing distutils maintenance into the setuptools project, and then bring it back via bundling of setuptools. Similarly, PEP 434 (which already exempts IDLE from the usual "No new features in maintenance releases" rules) could potentially be switched over to handling IDLE as an independently updated bundled application. The above idea is somewhat similar to what I suggested for the recommended modules list in https://mail.python.org/pipermail/python-ideas/2017-October/047508.html, but the difference is that to simplify the testing and installer building process, we'd bundle a suitable source archive in the git repo, rather than requiring PyPI access. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From gadgetsteve at live.co.uk Sun Nov 5 01:22:33 2017 From: gadgetsteve at live.co.uk (Steve Barnes) Date: Sun, 5 Nov 2017 05:22:33 +0000 Subject: [Python-ideas] Proposal to change Python version release cycle In-Reply-To: <20171104142953.0821b213@fsol> References: <45619991.35926.1509794757663@office.mailbox.org> <20171104142953.0821b213@fsol> Message-ID: On 04/11/2017 13:29, Antoine Pitrou wrote: > > Hello Wolfgang, > > On Sat, 4 Nov 2017 12:25:57 +0100 (CET) > tds333 at mailbox.org wrote: >> Hello, >> >> one of my long standing ideas to improve Python is to adjust the >> release cycle and version number handling. In short, to simplify it. > > There has been ample discussion in the past about changing our release > cycle one way or another. In short, the meta-problem is there are many > contradicting interests which would each favour a different solution to > the problem. See for example this PEP (ultimately rejected): > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.python.org%2Fdev%2Fpeps%2Fpep-0407%2F&data=02%7C01%7Cgadgetsteve%40live.co.uk%7Cf3da335b539641fb039b08d5238838ce%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636453990327468977&sdata=BJlayFVFBmTcA4DUoNwTwkeUO6uKrqy8RLlJHlqvOx8%3D&reserved=0 > > and the discussion that ensued: > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.python.org%2Fpipermail%2Fpython-dev%2F2012-January%2Fthread.html%23115591&data=02%7C01%7Cgadgetsteve%40live.co.uk%7Cf3da335b539641fb039b08d5238838ce%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636453990327468977&sdata=dodH8VnnkdkObTK7%2Bugu%2BinI5NACNF73WnvS8hjZZmE%3D&reserved=0 > > I haven't read your proposal in detail, but I suspect that it may be > vulnerable to some of the same objections. The big objection being > that a significant part of our ecosystem (that is, to put it roughly, > the more corporate-minded segment, though I believe it is a > simplification and may include other segments, such as Debian stable > users and maintainers) doesn't want to deal more frequent feature > releases. > > Regards > > Antoine. > > Another minus point from my prospective - I work for a huge corporate organisation, (~300k employees), and we have to seek approval from the legal departments, on a per division basis, before we can use any Open Source software, (I and several others know that the per division bit is a ridiculous overhead and we are trying to get it addressed). The approvers will let us get away with saying that we need approval to use version major.X, i.e. Python 3.x has been approved, but will not let us get away with requesting to use a tool without specifying the major version number. I suspect that the reasoning behind this is that most packages rightly consider a licence change a major revision. So for us this suggestion would result in us having to re-seek approval every 12-18 months. -- Steve (Gadget) Barnes Any opinions in this message are my personal opinions and do not reflect those of my employer. --- This email has been checked for viruses by AVG. http://www.avg.com From solipsis at pitrou.net Sun Nov 5 05:48:25 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 5 Nov 2017 11:48:25 +0100 Subject: [Python-ideas] Moving typing out of the stdlib in Python 3.7? References: Message-ID: <20171105114825.6c49bdff@fsol> On Sun, 5 Nov 2017 13:46:59 +1000 Nick Coghlan wrote: > * ensurepip gains the ability to also install bundled wheel files Why? Why wouldn't you put the wheel directly in site-packages on install? Regards Antoine. From p.f.moore at gmail.com Sun Nov 5 08:30:28 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 5 Nov 2017 13:30:28 +0000 Subject: [Python-ideas] Moving typing out of the stdlib in Python 3.7? In-Reply-To: <20171105114825.6c49bdff@fsol> References: <20171105114825.6c49bdff@fsol> Message-ID: On 5 November 2017 at 10:48, Antoine Pitrou wrote: > On Sun, 5 Nov 2017 13:46:59 +1000 > Nick Coghlan wrote: >> * ensurepip gains the ability to also install bundled wheel files > > Why? Why wouldn't you put the wheel directly in site-packages on > install? I'm not quite sure what you mean? It needs to be "installed", in the sense of being unpacked into site-packages, and the ensurepip mechanism is already able to do that for pip and setuptools, so adding an extra wheel to install wouldn't be too hard. If we don't install like that, people won't be able to easily upgrade typing by just using "pip install --upgrade typing". Wheels don't support simply being added to sys.path the way that eggs did, if that's what you mean. Paul From ncoghlan at gmail.com Sun Nov 5 08:44:29 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 5 Nov 2017 23:44:29 +1000 Subject: [Python-ideas] Moving typing out of the stdlib in Python 3.7? In-Reply-To: References: <20171105114825.6c49bdff@fsol> Message-ID: On 5 November 2017 at 23:30, Paul Moore wrote: > On 5 November 2017 at 10:48, Antoine Pitrou wrote: >> On Sun, 5 Nov 2017 13:46:59 +1000 >> Nick Coghlan wrote: >>> * ensurepip gains the ability to also install bundled wheel files >> >> Why? Why wouldn't you put the wheel directly in site-packages on >> install? > > I'm not quite sure what you mean? It needs to be "installed", in the > sense of being unpacked into site-packages, and the ensurepip > mechanism is already able to do that for pip and setuptools, so adding > an extra wheel to install wouldn't be too hard. If we don't install > like that, people won't be able to easily upgrade typing by just using > "pip install --upgrade typing". > > Wheels don't support simply being added to sys.path the way that eggs > did, if that's what you mean. What Paul said here. While wheels *can* be designed to support use-without-extraction (and pip's own wheel file is constructed that way so you can use a pip wheel to install itself), the general design principle is that we expect them to be installed prior to use, such that they get proper PEP 376 metadata entries, and so that their subcomponents end up in the right sysconfig directories for the deployment environment. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From mal at egenix.com Sun Nov 5 08:48:39 2017 From: mal at egenix.com (M.-A. Lemburg) Date: Sun, 5 Nov 2017 14:48:39 +0100 Subject: [Python-ideas] Proposal to change Python version release cycle In-Reply-To: References: <45619991.35926.1509794757663@office.mailbox.org> <20171104142953.0821b213@fsol> <8dd82d18-5828-2356-ee4f-14f6cc9dc14e@mailbox.org> Message-ID: <21698012-5e20-8b23-0539-a9f9b54f54f6@egenix.com> On 04.11.2017 20:33, Nick Timkovich wrote: > On Sat, Nov 4, 2017 at 10:44 AM, M.-A. Lemburg wrote: > >> Just to clarify: Python 2.0 was called 2.0 because the BeOpen marketing >> department thought it was good idea, not because there were major >> incompatible changes going into that release. >> > > Alternative history question: if it was just 1.6, then would Python 3000, > probably called "Python 2000/2.x" in that world, with all the fixes have > happened sooner because you would have "ran out" (yes, v1.10 > v1.9) of > numbers leading up to 2? There was a Python 1.6, but this was cut during the Python 2.0 release cycle to make CNRI happy. It included everything the CNRI team had developed after Python 1.5.2 before moving on to BeOpen to form the "PythonLabs": https://www.python.org/download/releases/1.6/ See "Overview of Changes Since 1.6" in https://www.python.org/download/releases/2.0/ for a list of changes between 1.6 and 2.0. The Python 1.6 never went into wide distribution, so for all practical purposes the move was from Python 1.5.2 to Python 2.0. More on all this is available in the "What's New in Python 2.0" document: https://docs.python.org/3/whatsnew/2.0.html -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Nov 05 2017) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From antoine at python.org Sun Nov 5 09:47:19 2017 From: antoine at python.org (Antoine Pitrou) Date: Sun, 5 Nov 2017 15:47:19 +0100 Subject: [Python-ideas] Moving typing out of the stdlib in Python 3.7? In-Reply-To: References: <20171105114825.6c49bdff@fsol> Message-ID: Le 05/11/2017 ? 14:30, Paul Moore a ?crit?: > On 5 November 2017 at 10:48, Antoine Pitrou wrote: >> On Sun, 5 Nov 2017 13:46:59 +1000 >> Nick Coghlan wrote: >>> * ensurepip gains the ability to also install bundled wheel files >> >> Why? Why wouldn't you put the wheel directly in site-packages on >> install? > > I'm not quite sure what you mean? It needs to be "installed", in the > sense of being unpacked into site-packages, and the ensurepip > mechanism is already able to do that for pip and setuptools, so adding > an extra wheel to install wouldn't be too hard. Ok, perhaps my question wasn't quite clear. Are you suggesting that people have to run "python -m ensurepip typing" after they installed Python? Or would the typing module be importable as soon as you have an installed Python, like stdlib modules are? Regards Antoine. From python at mrabarnett.plus.com Sun Nov 5 11:29:40 2017 From: python at mrabarnett.plus.com (MRAB) Date: Sun, 5 Nov 2017 16:29:40 +0000 Subject: [Python-ideas] Proposal to change Python version release cycle In-Reply-To: References: <45619991.35926.1509794757663@office.mailbox.org> <20171104142953.0821b213@fsol> <8dd82d18-5828-2356-ee4f-14f6cc9dc14e@mailbox.org> <5bc30002-d6a4-5fad-f1e0-96e2d74a059e@mailbox.org> Message-ID: On 2017-11-05 03:07, Nick Coghlan wrote: > On 5 November 2017 at 01:29, Wolfgang wrote: >> On 04.11.2017 16:01, Nick Coghlan wrote: >>> We're currently more likely to go the other direction, and stick with >>> the 3.x numbering for an extended period (potentially reaching 3.10, >>> 3.11, 3.12, etc), so that the ongoing disruption caused by the 2.x -> >>> 3.x transition has had a chance to settle down before we ask anyone to >>> start calling anything "Python 4". >> >> A good point but two digits minor version numbers have the possibility >> to break a lot code. There is a lot of stuff out where a single digit >> major version is assumed. Even the official Python build for windows >> with python27.dll, python36.dll can be problematic because the dot >> is omitted between numbers. >> Other do the same for compilation they concatenate only majorminor for a >> name. >> Then version 3.10 is the same as 31.0. >> Ok I know this will take a while. >> But for the first 2.7.10 version even there some other library code >> was broken because of such assumptions. > > Aye, we're aware :) > > We're not in a situation where we'll have any *good* options > available, so the question we're considering is "What do we think will > be the least bad approach?". > > At our current release cadence, 3.7 will be in mid 2018, 3.8 in late > 2019 or early 2020, and then 3.9 in mid 2021. > > The open question will be what we call the release after that (in late > 2022), with the main leading contenders being "4.0" (on the grounds > that so many changes will have accumulated since 2008's 3.0 release by > then that it makes sense to call them different major versions, > similar to the rationale the Linux kernel now uses for major version > updates), and "3.10" (on the grounds that the release won't be any > more different from 3.9 than 3.9 will be from 3.8). Other variants > (like making the major release number "40" or "2022", and using the > minor version field for some new purpose other than indicating > compatibility breaks in the compiler AST, interpreter opcode format, > code caching tags, and C ABI), tend to introduce the same pragmatic > questions that will already need to be considered in choosing between > the 3.10 and 4.0 alternatives. > > It's not just the version numbering aesthetics that need to be > considered either - in addition to the point you raise around how the > Python version gets embedded in file path names (such that the 3-digit > form is technically ambiguous without a separator, and may break > parsers that are expecting exactly two digits), there are other > backwards compatibility concerns around how folks do version > compatibility checks based on sys.version and sys.version_info. Python > and CPython will be more than 3 decades old by 2022, and an ecosystem > can build up a *lot* of implicit assumptions regarding how a > platform's versioning scheme works in that kind of time frame :) > > Personally, I doubt we'll make a firm decision on how we're going to > handle this until some time after 2.7 goes EOL in 2020, as by then > we'll necessarily have a better idea of how the various Linux distros > (including the LTS commercial ones) ended up handling the Python 2 -> > Python 3 switch, and the fact that the next release at that point will > be 3.9 making it eminently clear that we've run out of time to > continue postponing the question. We'll hopefully also have more > experience with folks relying on the stable runtime ABI, and hence > whether or not it might be beneficial to define new "py4" and "abi4" > compatibility tags for the binary wheel distribution format. > If there are 2 digits, it's major + minor. We could decide that if there are 3 digits, it's major + double-digit minor (e.g. "310" is "3.10"), and we could switch to major.minor when we get to Python 4.0 (or Python 10?). From p.f.moore at gmail.com Sun Nov 5 13:22:11 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 5 Nov 2017 18:22:11 +0000 Subject: [Python-ideas] Moving typing out of the stdlib in Python 3.7? In-Reply-To: References: <20171105114825.6c49bdff@fsol> Message-ID: On 5 November 2017 at 14:47, Antoine Pitrou wrote: > > Le 05/11/2017 ? 14:30, Paul Moore a ?crit : >> On 5 November 2017 at 10:48, Antoine Pitrou wrote: >>> On Sun, 5 Nov 2017 13:46:59 +1000 >>> Nick Coghlan wrote: >>>> * ensurepip gains the ability to also install bundled wheel files >>> >>> Why? Why wouldn't you put the wheel directly in site-packages on >>> install? >> >> I'm not quite sure what you mean? It needs to be "installed", in the >> sense of being unpacked into site-packages, and the ensurepip >> mechanism is already able to do that for pip and setuptools, so adding >> an extra wheel to install wouldn't be too hard. > > Ok, perhaps my question wasn't quite clear. Are you suggesting that > people have to run "python -m ensurepip typing" after they installed > Python? Or would the typing module be importable as soon as you have an > installed Python, like stdlib modules are? Ah, I get you now. I'd expect typing to be available exactly the same way as pip is. For me (on Windows) that means that it's available in a standard install. On Unix, I don't know (I think there's certain situations where you need to take extra steps in a custom build to ensure pip is available? I'd expect those steps to also install typing. So basically, yes, typing can be expected to be available in all installations, exactly the same as pip is. Paul From solipsis at pitrou.net Sun Nov 5 13:40:51 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 5 Nov 2017 19:40:51 +0100 Subject: [Python-ideas] Moving typing out of the stdlib in Python 3.7? References: <20171105114825.6c49bdff@fsol> Message-ID: <20171105194051.58293b33@fsol> On Sun, 5 Nov 2017 18:22:11 +0000 Paul Moore wrote: > On 5 November 2017 at 14:47, Antoine Pitrou wrote: > > > > Le 05/11/2017 ? 14:30, Paul Moore a ?crit : > >> On 5 November 2017 at 10:48, Antoine Pitrou wrote: > >>> On Sun, 5 Nov 2017 13:46:59 +1000 > >>> Nick Coghlan wrote: > >>>> * ensurepip gains the ability to also install bundled wheel files > >>> > >>> Why? Why wouldn't you put the wheel directly in site-packages on > >>> install? > >> > >> I'm not quite sure what you mean? It needs to be "installed", in the > >> sense of being unpacked into site-packages, and the ensurepip > >> mechanism is already able to do that for pip and setuptools, so adding > >> an extra wheel to install wouldn't be too hard. > > > > Ok, perhaps my question wasn't quite clear. Are you suggesting that > > people have to run "python -m ensurepip typing" after they installed > > Python? Or would the typing module be importable as soon as you have an > > installed Python, like stdlib modules are? > > Ah, I get you now. I'd expect typing to be available exactly the same > way as pip is. For me (on Windows) that means that it's available in a > standard install. On Unix, I don't know (I think there's certain > situations where you need to take extra steps in a custom build to > ensure pip is available? I'd expect those steps to also install > typing. I think typing shouldn't require any extra typing (ha) on Unix either. I don't remember what the rationale was for having to type "python -m ensurepip" to get pip installed, but typing is just a library, not an executable tool that may be able to mess with the system state, so I don't think it's worthwhile introducing an extra step for it. Regards Antoine. From apalala at gmail.com Sun Nov 5 13:49:18 2017 From: apalala at gmail.com (=?UTF-8?Q?Juancarlo_A=C3=B1ez?=) Date: Sun, 5 Nov 2017 14:49:18 -0400 Subject: [Python-ideas] About the efficiency of range() Message-ID: I found this interesting: https://stackoverflow.com/a/46996392 -- Juancarlo *A?ez* -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sun Nov 5 14:00:23 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 5 Nov 2017 19:00:23 +0000 Subject: [Python-ideas] Moving typing out of the stdlib in Python 3.7? In-Reply-To: <20171105194051.58293b33@fsol> References: <20171105114825.6c49bdff@fsol> <20171105194051.58293b33@fsol> Message-ID: On 5 November 2017 at 18:40, Antoine Pitrou wrote: > I think typing shouldn't require any extra typing (ha) on Unix either. > I don't remember what the rationale was for having to type > "python -m ensurepip" to get pip installed, but typing is just a > library, not an executable tool that may be able to mess with the > system state, so I don't think it's worthwhile introducing an extra > step for it. Yes, I agree. I didn't realise that "give me pip" was an extra step on Unix. I don't know what the arguments for doing it like that on Unix were, but they certainly don't seem to apply to typing. Paul From storchaka at gmail.com Sun Nov 5 14:14:48 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 5 Nov 2017 21:14:48 +0200 Subject: [Python-ideas] About the efficiency of range() In-Reply-To: References: Message-ID: 05.11.17 20:49, Juancarlo A?ez ????: > I found this interesting: > > https://stackoverflow.com/a/46996392 Please explain your suggestion. From storchaka at gmail.com Sun Nov 5 14:28:48 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 5 Nov 2017 21:28:48 +0200 Subject: [Python-ideas] Proposal to change Python version release cycle In-Reply-To: <5bc30002-d6a4-5fad-f1e0-96e2d74a059e@mailbox.org> References: <45619991.35926.1509794757663@office.mailbox.org> <20171104142953.0821b213@fsol> <8dd82d18-5828-2356-ee4f-14f6cc9dc14e@mailbox.org> <5bc30002-d6a4-5fad-f1e0-96e2d74a059e@mailbox.org> Message-ID: 04.11.17 17:29, Wolfgang ????: > A good point but two digits minor version numbers have the possibility > to break a lot code. There is a lot of stuff out where a single digit > major version is assumed. Even the official Python build for windows > with python27.dll, python36.dll can be problematic because the dot > is omitted between numbers. > Other do the same for compilation they concatenate only majorminor for a > name. > Then version 3.10 is the same as 31.0. Actually this is yet one argument against your idea. With your proposal 27 will be the same as 2.7. From stephanh42 at gmail.com Sun Nov 5 14:53:34 2017 From: stephanh42 at gmail.com (Stephan Houben) Date: Sun, 5 Nov 2017 20:53:34 +0100 Subject: [Python-ideas] Proposal to change Python version release cycle In-Reply-To: References: <45619991.35926.1509794757663@office.mailbox.org> <20171104142953.0821b213@fsol> <8dd82d18-5828-2356-ee4f-14f6cc9dc14e@mailbox.org> <5bc30002-d6a4-5fad-f1e0-96e2d74a059e@mailbox.org> Message-ID: After python39.dll I'd expect python3A.dll . Or possibly python3?.dll >>> len("python3?.dll") == len("python39.dll") True No worries, then. Stephan 2017-11-05 20:28 GMT+01:00 Serhiy Storchaka : > 04.11.17 17:29, Wolfgang ????: > >> A good point but two digits minor version numbers have the possibility >> to break a lot code. There is a lot of stuff out where a single digit >> major version is assumed. Even the official Python build for windows >> with python27.dll, python36.dll can be problematic because the dot >> is omitted between numbers. >> Other do the same for compilation they concatenate only majorminor for a >> name. >> Then version 3.10 is the same as 31.0. >> > > Actually this is yet one argument against your idea. With your proposal 27 > will be the same as 2.7. > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Sun Nov 5 15:02:09 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 5 Nov 2017 22:02:09 +0200 Subject: [Python-ideas] Proposal to change Python version release cycle In-Reply-To: References: <45619991.35926.1509794757663@office.mailbox.org> <20171104142953.0821b213@fsol> <8dd82d18-5828-2356-ee4f-14f6cc9dc14e@mailbox.org> <5bc30002-d6a4-5fad-f1e0-96e2d74a059e@mailbox.org> Message-ID: 05.11.17 21:53, Stephan Houben ????: > After python39.dll I'd expect python3A.dll . The problem will return in 3.36. > Or possibly python3?.dll > > >>> len("python3?.dll") == len("python39.dll") > True > > No worries, then. But len(os.fsencode("python3?.dll")) != len(os.fsencode("python39.dll")). From stephanh42 at gmail.com Sun Nov 5 15:11:37 2017 From: stephanh42 at gmail.com (Stephan Houben) Date: Sun, 5 Nov 2017 21:11:37 +0100 Subject: [Python-ideas] Proposal to change Python version release cycle In-Reply-To: References: <45619991.35926.1509794757663@office.mailbox.org> <20171104142953.0821b213@fsol> <8dd82d18-5828-2356-ee4f-14f6cc9dc14e@mailbox.org> <5bc30002-d6a4-5fad-f1e0-96e2d74a059e@mailbox.org> Message-ID: 2017-11-05 21:02 GMT+01:00 Serhiy Storchaka : > But len(os.fsencode("python3?.dll")) != len(os.fsencode("python39.dll")). > > I think it is on Windows. os.fsencode should use UTF-16 there. ? is in the BMP Presumably, non-Windows platforms wouldn't be interested in .dll files anyway... Stephan > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Sun Nov 5 18:01:15 2017 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 5 Nov 2017 18:01:15 -0500 Subject: [Python-ideas] About the efficiency of range() In-Reply-To: References: Message-ID: On 11/5/2017 1:49 PM, Juancarlo A?ez wrote: > I found this interesting: > https://stackoverflow.com/a/46996392 python-list is the place to post 'interesting' python-related items. If you are proposing a repeat(n) builtin, that was discussed on this list and rejected a year or two ago. It would be be a bit faster than even itertools.repeat(None, n) in a for loop. The speed argument for the proposal was countered then with the point made on the 3rd comment in the link: "But in practical code the body of the loop will be more complex, and dominate the over all timing. " An overhead of .23 microseconds per loop is small enough for real use cases. The better argument for repeat(n) was simplicity, not speed. The counter for that was and is that never needing the index for the lifetime of a particular loop, including learning, testing, debugging, and maintenance, is rare and not predictable. -- Terry Jan Reedy From barry at barrys-emacs.org Sun Nov 5 17:29:22 2017 From: barry at barrys-emacs.org (Barry Scott) Date: Sun, 05 Nov 2017 22:29:22 +0000 Subject: [Python-ideas] Moving typing out of the stdlib in Python 3.7? In-Reply-To: References: Message-ID: <6165094.eUCsvfha0D@varric.chelsea.private> On Saturday, 4 November 2017 20:22:25 GMT Guido van Rossum wrote: > On Sat, Nov 4, 2017 at 7:05 AM, Nick Coghlan wrote: > > Perhaps typing could switch to being a bundled module, such that it > > had its own version, independent of the Python standard library > > version, but was still present by default in new installations? > > This is beginning to sound like the most attractive solution. We could > possibly do away with typing_extensions. Are there precedents of how to > bundle a module in this way? Or is it going to be another special case like > pip? Is the outcome you want that you ship a version of typing with the python kit, but if you install from pip it overrides the one shipped in the python kit? That would be a matter of being having a suitable sys.path/site config I guess. pip folder before the bundled packages folder. If this is a mechanism that python kitting has then you would be able to bundle other packages like requests or six as well as typing, but because you can use pip to override the one shipped a user can optionally keep up with the latest versions. Barry From rosuav at gmail.com Sun Nov 5 18:08:57 2017 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 6 Nov 2017 10:08:57 +1100 Subject: [Python-ideas] Moving typing out of the stdlib in Python 3.7? In-Reply-To: <6165094.eUCsvfha0D@varric.chelsea.private> References: <6165094.eUCsvfha0D@varric.chelsea.private> Message-ID: On Mon, Nov 6, 2017 at 9:29 AM, Barry Scott wrote: > On Saturday, 4 November 2017 20:22:25 GMT Guido van Rossum wrote: >> On Sat, Nov 4, 2017 at 7:05 AM, Nick Coghlan wrote: >> > Perhaps typing could switch to being a bundled module, such that it >> > had its own version, independent of the Python standard library >> > version, but was still present by default in new installations? >> >> This is beginning to sound like the most attractive solution. We could >> possibly do away with typing_extensions. Are there precedents of how to >> bundle a module in this way? Or is it going to be another special case like >> pip? > > Is the outcome you want that you ship a version of typing with the python kit, > but if you install from pip it overrides the one shipped in the python kit? > > That would be a matter of being having a suitable sys.path/site config I > guess. pip folder before the bundled packages folder. > > If this is a mechanism that python kitting has then you would be able to > bundle other packages like requests or six as well as typing, but because > you can use pip to override the one shipped a user can optionally keep > up with the latest versions. If this were to happen, I would be inclined to put these "bootstrap" modules into their own directory in sys.path, after the rest of the stdlib. Then someone who's paranoid about stdlib shadowing could put pip-installed modules after the bulk of the stdlib (thus preventing any third-party package from overriding "import random", for instance) but still update modules that are specifically intended for updating; plus it'd mean you can get a directory listing of that, and go grab all the "blessed by python.org as an extension of the stdlib" packages. ChrisA From random832 at fastmail.com Sun Nov 5 21:16:15 2017 From: random832 at fastmail.com (Random832) Date: Sun, 05 Nov 2017 21:16:15 -0500 Subject: [Python-ideas] Proposal to change Python version release cycle In-Reply-To: <5bc30002-d6a4-5fad-f1e0-96e2d74a059e@mailbox.org> References: <45619991.35926.1509794757663@office.mailbox.org> <20171104142953.0821b213@fsol> <8dd82d18-5828-2356-ee4f-14f6cc9dc14e@mailbox.org> <5bc30002-d6a4-5fad-f1e0-96e2d74a059e@mailbox.org> Message-ID: <1509934575.1599416.1162610440.5EA807C1@webmail.messagingengine.com> On Sat, Nov 4, 2017, at 10:29, Wolfgang wrote: > For example the user specified python3 but I know it will be compatible > with python4 so the launcher can do something like > python3 is not found then I use python4 if available. > > ex.: > > #!/usr/bin/env py -3 If you actually try that, you'll find out the dirty little secret of the #! functionality - on many systems, it only supports one argument. From gadgetsteve at live.co.uk Sun Nov 5 14:18:30 2017 From: gadgetsteve at live.co.uk (Steve Barnes) Date: Sun, 5 Nov 2017 19:18:30 +0000 Subject: [Python-ideas] Possible enhancement to typing Message-ID: The discussions on moving typing to a standalone library prompted me to take a look at the documentation for typing which in turn lead to an idea that should be reasonably simple to implement but greatly add value to typing for some applications. If a group of iterators were to be added to the typing module it would be reasonably simple to automatically add and assert to any decorated modules to ensure that such modules were always called with the documented types. I am thinking of decorators such as: - @typing.assert_params_mismatch - this would provide a wrapper that had auto-generated asserts that all the parameters were of designated types. - @typing.debug_assert_params_mismatch - this would provide a wrapper that had auto-generated asserts that all the parameters were of designated types only if a DEBUG environmental variable was set or similar. - @typing.warn_params_mismatch - this would provide a wrapper that had auto-generated warnings issued to stderr if any of the parameters were of not of the designated types. - @typing.coerce_params - this would add a wrapper to attempt to convert any mismatched parameters to the specified type(s). I also think that this might increase the uptake of typing by giving some clear benefits outside of documentation and static type checking. Pushing this out for comment before considering a more formal, (PEP), proposal. -- Steve (Gadget) Barnes Any opinions in this message are my personal opinions and do not reflect those of my employer. --- This email has been checked for viruses by AVG. http://www.avg.com From ncoghlan at gmail.com Mon Nov 6 01:06:06 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 6 Nov 2017 16:06:06 +1000 Subject: [Python-ideas] Moving typing out of the stdlib in Python 3.7? In-Reply-To: References: <6165094.eUCsvfha0D@varric.chelsea.private> Message-ID: On 6 November 2017 at 09:08, Chris Angelico wrote: > On Mon, Nov 6, 2017 at 9:29 AM, Barry Scott wrote: >> If this is a mechanism that python kitting has then you would be able to >> bundle other packages like requests or six as well as typing, but because >> you can use pip to override the one shipped a user can optionally keep >> up with the latest versions. > > If this were to happen, I would be inclined to put these "bootstrap" > modules into their own directory in sys.path, after the rest of the > stdlib. Then someone who's paranoid about stdlib shadowing could put > pip-installed modules after the bulk of the stdlib (thus preventing > any third-party package from overriding "import random", for instance) > but still update modules that are specifically intended for updating; > plus it'd mean you can get a directory listing of that, and go grab > all the "blessed by python.org as an extension of the stdlib" > packages. When we say bundled, we mean bundled: the exact same bits you would get from PyPI, installed the same way, and if you upgrade it system wide, there's no trace of the one we bundled. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Mon Nov 6 01:07:56 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 6 Nov 2017 16:07:56 +1000 Subject: [Python-ideas] Moving typing out of the stdlib in Python 3.7? In-Reply-To: References: <20171105114825.6c49bdff@fsol> <20171105194051.58293b33@fsol> Message-ID: On 6 November 2017 at 05:00, Paul Moore wrote: > On 5 November 2017 at 18:40, Antoine Pitrou wrote: >> I think typing shouldn't require any extra typing (ha) on Unix either. >> I don't remember what the rationale was for having to type >> "python -m ensurepip" to get pip installed, but typing is just a >> library, not an executable tool that may be able to mess with the >> system state, so I don't think it's worthwhile introducing an extra >> step for it. > > Yes, I agree. I didn't realise that "give me pip" was an extra step on > Unix. I don't know what the arguments for doing it like that on Unix > were, but they certainly don't seem to apply to typing. It's the default on Unix as well - you have to do "make install ENSUREPIP=no" to avoid getting it. (And some distros also modify their Python installations so that pip is missing by default) We provide "python -m ensurepip" because we offer ways to skip doing it at install time (and venv creation time), so the explicit command gives folks a straightforward way to change their minds later. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From rosuav at gmail.com Mon Nov 6 01:16:30 2017 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 6 Nov 2017 17:16:30 +1100 Subject: [Python-ideas] Moving typing out of the stdlib in Python 3.7? In-Reply-To: References: <6165094.eUCsvfha0D@varric.chelsea.private> Message-ID: On Mon, Nov 6, 2017 at 5:06 PM, Nick Coghlan wrote: > On 6 November 2017 at 09:08, Chris Angelico wrote: >> On Mon, Nov 6, 2017 at 9:29 AM, Barry Scott wrote: >>> If this is a mechanism that python kitting has then you would be able to >>> bundle other packages like requests or six as well as typing, but because >>> you can use pip to override the one shipped a user can optionally keep >>> up with the latest versions. >> >> If this were to happen, I would be inclined to put these "bootstrap" >> modules into their own directory in sys.path, after the rest of the >> stdlib. Then someone who's paranoid about stdlib shadowing could put >> pip-installed modules after the bulk of the stdlib (thus preventing >> any third-party package from overriding "import random", for instance) >> but still update modules that are specifically intended for updating; >> plus it'd mean you can get a directory listing of that, and go grab >> all the "blessed by python.org as an extension of the stdlib" >> packages. > > When we say bundled, we mean bundled: the exact same bits you would > get from PyPI, installed the same way, and if you upgrade it system > wide, there's no trace of the one we bundled. Oh, that's even better! :+1: ChrisA From desmoulinmichel at gmail.com Mon Nov 6 01:30:35 2017 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Mon, 6 Nov 2017 07:30:35 +0100 Subject: [Python-ideas] Moving typing out of the stdlib in Python 3.7? In-Reply-To: References: <20171105114825.6c49bdff@fsol> <20171105194051.58293b33@fsol> Message-ID: Le 06/11/2017 ? 07:07, Nick Coghlan a ?crit?: > It's the default on Unix as well - you have to do "make install > ENSUREPIP=no" to avoid getting it. (And some distros also modify their > Python installations so that pip is missing by default) On debian and derivatives (so Ubuntu) you need to install python-pip to be able to use pip. Now it's annoying already. Because you have to write every tutorial to include a special case for them. But at least it's not a required step to run your program. However, if you do code using type hints and typing is not installed, you can't even run the program without installing something. So we finally have Python 3 by default on most Linux system, but still would not be able to assume we can run a modern script on it. From lukasz at langa.pl Mon Nov 6 01:42:56 2017 From: lukasz at langa.pl (Lukasz Langa) Date: Sun, 5 Nov 2017 22:42:56 -0800 Subject: [Python-ideas] Moving typing out of the stdlib in Python 3.7? In-Reply-To: References: <20171105114825.6c49bdff@fsol> <20171105194051.58293b33@fsol> Message-ID: <7F60DB5E-063E-461B-821B-CB4129299442@langa.pl> > On 5 Nov, 2017, at 10:30 PM, Michel Desmoulin wrote: > > Le 06/11/2017 ? 07:07, Nick Coghlan a ?crit : > >> It's the default on Unix as well - you have to do "make install >> ENSUREPIP=no" to avoid getting it. (And some distros also modify their >> Python installations so that pip is missing by default) > > On debian and derivatives (so Ubuntu) you need to install python-pip to > be able to use pip. > > Now it's annoying already. Because you have to write every tutorial to > include a special case for them. But at least it's not a required step > to run your program. > > However, if you do code using type hints and typing is not installed, > you can't even run the program without installing something. So we > finally have Python 3 by default on most Linux system, but still would > not be able to assume we can run a modern script on it. This is a valid concern. Although this particular problem is self-inflicted by Debian, I can understand their rationale behind explicit packaging. They need to have control over the entire package graph. I wonder if there's a way in .deb to specify a required installed package. I'm not calling it a "dependency" since obviously it would rather be "python3-typing" that depends on "python3". Barry, do you know? But even if Debian installs python3-typing by default, will "pip install -U typing" be possible in this scenario? I guess it wouldn't be terrible if that only worked in virtualenvs, although ideally it would work also for the raw host installation. - ? -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From desmoulinmichel at gmail.com Mon Nov 6 01:47:32 2017 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Mon, 6 Nov 2017 07:47:32 +0100 Subject: [Python-ideas] Looking for input to help with the pip situation Message-ID: Hello, Today I'm going to give a training in Python again. And again it will go the same way. On Mac I will have to make people install python, then tell them to use pip3. On Windows, I will have to warn them about checking the "add python executable to system path" (that one of them will ALWAYS miss anyway). Then tell them to use py -3.x -m pip because some of them will have several versions of Python installed. Then on linux, I will tell them to install python-pip and python-venv and use python3 -m pip. I'll talk about --user, but commands won't be usable on some machine where the Scripts or bin dir is not in the system path. Then I will make them create a virtualenv so that they can avoid messing with their system python and finally can just use "pip install" like in most tutorials on the Web. And eventually I'll talk about pipenv and conda. The first one so they don't have to think about activating the virtualenv everytime, or pip freeze, or create the venv, or add it to gitignore, etc. The second because anaconda is very popular on windows. There is no way a beginner is going to get any that by themselves without a lot of time and pain. They will read some tutorial on the web and struggle to make sens of what pip is and why "pip install" doesn't work and why "python sucks". I think Python is offering an incredible experience for first timer. However, the whole "where is walpip" shenanigans is not one of them. I really want some people from this list to discuss here so we can find a way to either unify a bit the way we install and use pip, or find a way to express a tutorial that always works for people on the most popular platforms and spread the word so that any doc uses it. Michel From steve at pearwood.info Mon Nov 6 02:13:57 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 6 Nov 2017 18:13:57 +1100 Subject: [Python-ideas] Possible enhancement to typing In-Reply-To: References: Message-ID: <20171106071357.GC15990@ando.pearwood.info> On Sun, Nov 05, 2017 at 07:18:30PM +0000, Steve Barnes wrote: > If a group of iterators were to be added to the typing module it would > be reasonably simple to automatically add and assert to any decorated > modules to ensure that such modules were always called with the > documented types. "Iterators"? > I am thinking of decorators such as: > > - @typing.assert_params_mismatch - this would provide a wrapper that > had auto-generated asserts that all the parameters were of designated types. > - @typing.debug_assert_params_mismatch - this would provide a wrapper > that had auto-generated asserts that all the parameters were of > designated types only if a DEBUG environmental variable was set or similar. That's what assert does: assert only runs when __DEBUG__ is true. That's not controlled by an environment variable, but by the -O flag to the interpreter. So your assert_params_mismatch and debug_assert_params_mismatch are effectively the same thing. But using assert to check to perform argument checks is often an abuse of assert. To be more specific, using assert to check the value of public arguments in library code (where the arguments come from outside the library) is wrong, since you (the library author) cannot guarantee that your type tests will even run. Using asserts for argument checking inside application code is more of a grey area, with arguments for and against using assert. But in my opinion, the deciding factor is nearly always that an AssertionError is the wrong sort of exception. Outside of some fairly limited circumstances, most of which don't involve type-checking function arguments, using assert robs the caller of some useful information: the *kind* of error. (TypeError, ValueError, etc.) See here for further discussion: https://import-that.dreamwidth.org/676.html In general, I don't think we want to encourage such runtime type testing. Obviously there are exceptions -- library code should probably type check arguments, applications perhaps not -- and we're not exactly discouraging it either. There are already a number of third-party libraries that provide argument type tests at runtime, and I think that's probably the right place for them. [...] > I also think that this might increase the uptake of typing by giving > some clear benefits outside of documentation and static type checking. Problem is, the benefits of runtime type checking aren't clear. But the costs certainly are: if you want slow code, do lots and lots of runtime type checks. -- Steve From gadgetsteve at live.co.uk Mon Nov 6 02:39:53 2017 From: gadgetsteve at live.co.uk (Steve Barnes) Date: Mon, 6 Nov 2017 07:39:53 +0000 Subject: [Python-ideas] Possible enhancement to typing In-Reply-To: <20171106071357.GC15990@ando.pearwood.info> References: <20171106071357.GC15990@ando.pearwood.info> Message-ID: On 06/11/2017 07:13, Steven D'Aprano wrote: > On Sun, Nov 05, 2017 at 07:18:30PM +0000, Steve Barnes wrote: > >> If a group of iterators were to be added to the typing module it would >> be reasonably simple to automatically add and assert to any decorated >> modules to ensure that such modules were always called with the >> documented types. > > "Iterators"? > > >> I am thinking of decorators such as: >> >> - @typing.assert_params_mismatch - this would provide a wrapper that >> had auto-generated asserts that all the parameters were of designated types. >> - @typing.debug_assert_params_mismatch - this would provide a wrapper >> that had auto-generated asserts that all the parameters were of >> designated types only if a DEBUG environmental variable was set or similar. > > That's what assert does: assert only runs when __DEBUG__ is true. That's > not controlled by an environment variable, but by the -O flag to the > interpreter. > Good point. > So your assert_params_mismatch and debug_assert_params_mismatch are > effectively the same thing. > > But using assert to check to perform argument checks is often an abuse > of assert. To be more specific, using assert to check the value of > public arguments in library code (where the arguments come from outside the > library) is wrong, since you (the library author) cannot guarantee > that your type tests will even run. > > Using asserts for argument checking inside application code is more of a > grey area, with arguments for and against using assert. > > But in my opinion, the deciding factor is nearly always that an > AssertionError is the wrong sort of exception. Outside of some fairly > limited circumstances, most of which don't involve type-checking > function arguments, using assert robs the caller of some useful > information: the *kind* of error. (TypeError, ValueError, etc.) > I see your point here. > See here for further discussion: > > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fimport-that.dreamwidth.org%2F676.html&data=02%7C01%7Cgadgetsteve%40live.co.uk%7C5e419db490b84018f2d208d524e612b4%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636455492928665911&sdata=EFZIvxGxO18vk5s97RnSHH9kAuLS3sXBxDNS9VbLWlw%3D&reserved=0 > > In general, I don't think we want to encourage such runtime type > testing. Obviously there are exceptions -- library code should > probably type check arguments, applications perhaps not -- and > we're not exactly discouraging it either. There are already a number of > third-party libraries that provide argument type tests at runtime, and > I think that's probably the right place for them. > I'll have to look out for them. > > [...] >> I also think that this might increase the uptake of typing by giving >> some clear benefits outside of documentation and static type checking. > > Problem is, the benefits of runtime type checking aren't clear. But the > costs certainly are: if you want slow code, do lots and lots of runtime > type checks. > > Too much time spent writing safety critical code on my part then! I'll drop the idea. -- Steve (Gadget) Barnes Any opinions in this message are my personal opinions and do not reflect those of my employer. --- This email has been checked for viruses by AVG. http://www.avg.com From ncoghlan at gmail.com Mon Nov 6 03:28:10 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 6 Nov 2017 18:28:10 +1000 Subject: [Python-ideas] Moving typing out of the stdlib in Python 3.7? In-Reply-To: <7F60DB5E-063E-461B-821B-CB4129299442@langa.pl> References: <20171105114825.6c49bdff@fsol> <20171105194051.58293b33@fsol> <7F60DB5E-063E-461B-821B-CB4129299442@langa.pl> Message-ID: On 6 November 2017 at 16:42, Lukasz Langa wrote: > >> On 5 Nov, 2017, at 10:30 PM, Michel Desmoulin wrote: >> >> Le 06/11/2017 ? 07:07, Nick Coghlan a ?crit : >> >>> It's the default on Unix as well - you have to do "make install >>> ENSUREPIP=no" to avoid getting it. (And some distros also modify their >>> Python installations so that pip is missing by default) >> >> On debian and derivatives (so Ubuntu) you need to install python-pip to >> be able to use pip. >> >> Now it's annoying already. Because you have to write every tutorial to >> include a special case for them. But at least it's not a required step >> to run your program. >> >> However, if you do code using type hints and typing is not installed, >> you can't even run the program without installing something. So we >> finally have Python 3 by default on most Linux system, but still would >> not be able to assume we can run a modern script on it. > > This is a valid concern. Although this particular problem is self-inflicted by Debian, I can understand their rationale behind explicit packaging. They need to have control over the entire package graph. I wonder if there's a way in .deb to specify a required installed package. I'm not calling it a "dependency" since obviously it would rather be "python3-typing" that depends on "python3". Fedora just lives with the circular dependency between python3 and python3-pip, which ensures both are installed by default (this arrangement allows "python3 -m venv" to still install pip, without actually needing a second copy of pip inside the python3 package) A bundled typing module that "python -m venv" installed by default would probably require similar treatment. > But even if Debian installs python3-typing by default, will "pip install -U typing" be possible in this scenario? I guess it wouldn't be terrible if that only worked in virtualenvs, although ideally it would work also for the raw host installation. "sudo pip install " remains a terrible idea on any distro, because it leads to pip and the system package manager getting into a fight over which tool is responsible for managing the affected files. "pip install --user --upgrade typing" should work OK, though. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Mon Nov 6 03:47:43 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 6 Nov 2017 18:47:43 +1000 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: Message-ID: On 6 November 2017 at 16:47, Michel Desmoulin wrote: > I really want some people from this list to discuss here so we can find > a way to either unify a bit the way we install and use pip, or find a > way to express a tutorial that always works for people on the most > popular platforms and spread the word so that any doc uses it. https://docs.python.org/3/installing/#basic-usage is as close as we've been able to get to that for the time being. For Linux, you'll still need to do the initial "python3 -m venv" to get your students out of the system Python. I expect you'll also need an initial "py -m venv" for Windows users, to get their PATH configured appropriately with both a "python" command and any scripts they install from PyPI. But the key point: do *NOT* try to teach without creating a virtual environment as the first step, because it doesn't actually make anything simpler, and in fact makes a lot of things harder and more platform dependent. The tutorial in the Python Packaging User Guide similarly starts with venv creation: https://packaging.python.org/tutorials/installing-packages/#creating-virtual-environments Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From stephanh42 at gmail.com Mon Nov 6 03:50:28 2017 From: stephanh42 at gmail.com (Stephan Houben) Date: Mon, 6 Nov 2017 09:50:28 +0100 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: Message-ID: Hi Michel, That's exactly why I proposed a `pip` function available from the Python prompt. I suppose you could still tell your students to copy/paste the following into their Python interpreter. def pip(args): import sys import subprocess subprocess.check_call([sys.executable, "-m", "pip"] + args.split()) print("Please restart Python now to use installed or upgraded packages.") I suppose an alternative is to set up jupyterhub https://jupyterhub.readthedocs.io/en/latest/ and let all your students just access that from a webbrowser. Stephan 2017-11-06 7:47 GMT+01:00 Michel Desmoulin : > Hello, > > Today I'm going to give a training in Python again. > > And again it will go the same way. > > On Mac I will have to make people install python, then tell them to use > pip3. > > On Windows, I will have to warn them about checking the "add python > executable to system path" (that one of them will ALWAYS miss anyway). > Then tell them to use py -3.x -m pip because some of them will have > several versions of Python installed. > > Then on linux, I will tell them to install python-pip and python-venv > and use python3 -m pip. > > I'll talk about --user, but commands won't be usable on some machine > where the Scripts or bin dir is not in the system path. > > Then I will make them create a virtualenv so that they can avoid messing > with their system python and finally can just use "pip install" like in > most tutorials on the Web. > > And eventually I'll talk about pipenv and conda. The first one so they > don't have to think about activating the virtualenv everytime, or pip > freeze, or create the venv, or add it to gitignore, etc. The second > because anaconda is very popular on windows. > > There is no way a beginner is going to get any that by themselves > without a lot of time and pain. They will read some tutorial on the web > and struggle to make sens of what pip is and why "pip install" doesn't > work and why "python sucks". > > I think Python is offering an incredible experience for first timer. > However, the whole "where is walpip" shenanigans is not one of them. > > I really want some people from this list to discuss here so we can find > a way to either unify a bit the way we install and use pip, or find a > way to express a tutorial that always works for people on the most > popular platforms and spread the word so that any doc uses it. > > Michel > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Mon Nov 6 04:00:16 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 6 Nov 2017 09:00:16 +0000 Subject: [Python-ideas] Moving typing out of the stdlib in Python 3.7? In-Reply-To: <7F60DB5E-063E-461B-821B-CB4129299442@langa.pl> References: <20171105114825.6c49bdff@fsol> <20171105194051.58293b33@fsol> <7F60DB5E-063E-461B-821B-CB4129299442@langa.pl> Message-ID: On 6 November 2017 at 06:42, Lukasz Langa wrote: >> Now it's annoying already. Because you have to write every tutorial to >> include a special case for them. But at least it's not a required step >> to run your program. >> >> However, if you do code using type hints and typing is not installed, >> you can't even run the program without installing something. So we >> finally have Python 3 by default on most Linux system, but still would >> not be able to assume we can run a modern script on it. > > This is a valid concern. Although this particular problem is self-inflicted by Debian, I can understand their rationale behind explicit packaging. They need to have control over the entire package graph. It's also a problem for virtual environments, which won't include typing by default, and don't see the system installed packages. As Nick says, we could modify venv to always include typing, but most users, and all tools (tox, pew, pipenv, etc) use virtualenv rather than venv still, so this would be of very limited benefit. > I wonder if there's a way in .deb to specify a required installed package. I'm not calling it a "dependency" since obviously it would rather be "python3-typing" that depends on "python3". Barry, do you know? There's always "include typing in the standard library" :-) > But even if Debian installs python3-typing by default, will "pip install -U typing" be possible in this scenario? I guess it wouldn't be terrible if that only worked in virtualenvs, although ideally it would work also for the raw host installation. It would push yet more people into doing "sudo pip install -U typing", which is something we've been trying really hard to encourage them to stop doing over on the pip issues list. We strongly recommend using system supplied packages, but I'd be surprised if Debian is any more able to keep up with the latest versions of typing than Python core is. Paul From ncoghlan at gmail.com Mon Nov 6 04:12:39 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 6 Nov 2017 19:12:39 +1000 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: Message-ID: On 6 November 2017 at 18:50, Stephan Houben wrote: > Hi Michel, > > That's exactly why I proposed a `pip` function available from the Python > prompt. > I suppose you could still tell your students to copy/paste the following > into their > Python interpreter. > > def pip(args): > import sys > import subprocess > subprocess.check_call([sys.executable, "-m", "pip"] + args.split()) > print("Please restart Python now to use installed or upgraded > packages.") Depending on where and how Python is installed, this may still not work (while I'll grant that Linux users are more likely to already be familiar with the command line than Windows and Mac OS X users, it's still a non-trivial step from there to realising why "sudo pip install" is almost always a bad idea) > I suppose an alternative is to set up jupyterhub > > https://jupyterhub.readthedocs.io/en/latest/ > > and let all your students just access that from a webbrowser. Yep, and lots of teachers use services like PythonAnywhere, trinket.io, and similar, precisely because the only thing the learners need locally is a web browser. The main downside is that learning that way isn't quite as transferable a skill, since it completely hides the code packaging and distribution step. However, it's a good option when the primary aim is to teach computational skills, and Python is just the vehicle used for that purpose. Software Carpentry starts out with the Anaconda distribution, as it not only improves the cross-platform UX consistent situation, it also deals with the external binary dependency problem (at least for the core set of packages provided either natively or via conda-forge). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From niki.spahiev at gmail.com Mon Nov 6 04:17:59 2017 From: niki.spahiev at gmail.com (Niki Spahiev) Date: Mon, 6 Nov 2017 11:17:59 +0200 Subject: [Python-ideas] array.array.index() and Sequence ABC Message-ID: Hello, I noticed that array.index() method takes single argument in python 3.6, while Sequence protocol specifies 2 optional arguments after the first. example from list: L.index(value, [start, [stop]]) -> integer -- return first index of value. I propose adding start and stop arguments to array.index() Regards, Niki From storchaka at gmail.com Mon Nov 6 05:14:02 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 6 Nov 2017 12:14:02 +0200 Subject: [Python-ideas] array.array.index() and Sequence ABC In-Reply-To: References: Message-ID: 06.11.17 11:17, Niki Spahiev ????: > I noticed that array.index() method takes single argument in python 3.6, > while Sequence protocol specifies 2 optional arguments after the first. > > example from list: > > L.index(value, [start, [stop]]) -> integer -- return first index of value. > > I propose adding start and stop arguments to array.index() Open an issue on the bug tracker and provide a patch. See also: https://bugs.python.org/issue28197 https://bugs.python.org/issue31942 From sf at fermigier.com Mon Nov 6 05:34:28 2017 From: sf at fermigier.com (=?UTF-8?Q?St=C3=A9fane_Fermigier?=) Date: Mon, 6 Nov 2017 11:34:28 +0100 Subject: [Python-ideas] Possible enhancement to typing In-Reply-To: References: <20171106071357.GC15990@ando.pearwood.info> Message-ID: A few random remarks: 1) This seems similar to the Typeguard project https://github.com/agronholm/typeguard I personally never used it. I'd rather have some machinery that would run the checks "magically" using the type annotations, without the need for explicit decoration. 2) I fully agree that there is a important difference between using runtime checks to verify your code (for instance, as part of an automated test suite, or even when manually testing an application), and for data validation. 3) In practice, however, I have never disabled assertions in production, but I agree if can be problematic if developers confuse AssertionErrors with TypeErros. 4) 10 years ago, when I was working on the EDOS project ( http://cordis.europa.eu/pub/ist/docs/directorate_d/st-ds/edos-project-story_en.pdf ), I ran a small experiment where I used, IIRC, the profile hook to intercept all function / method calls, and log information about arguments and return value types to a gigantic log file. Then the log file could be parsed and these information used to suggest type annotations. Except there were no type annotations at the time in Python. I know PyCharm can do a similar thing now: you run your program or your tests under the debugger, it logs runtime type information somewhere, and then can use it to suggest autocompletion or maybe type annotations. Now I believe something could be done along the lines: a) record runtime type information from test or regular runs b) massage these information and use them to annotate Python code with additional type information (up to the developer to then accept or not the proposed changes) c) also run a test suite or an app under some magical machinery, and either raise a TypeError or log warnings when discrepancies are detected between type annotation and runtime behaviour. (c) could be done independently from (a) and (b), (a) and (b) would use similar machinery, and (a), (b) and (c) would be probably a useful way to introduce type annotations to an existing code base without too much risk. (a) and (b) could also provide data for an interesting SE research project. S. On Mon, Nov 6, 2017 at 8:39 AM, Steve Barnes wrote: > > > On 06/11/2017 07:13, Steven D'Aprano wrote: > > On Sun, Nov 05, 2017 at 07:18:30PM +0000, Steve Barnes wrote: > > > >> If a group of iterators were to be added to the typing module it would > >> be reasonably simple to automatically add and assert to any decorated > >> modules to ensure that such modules were always called with the > >> documented types. > > > > "Iterators"? > > > > > >> I am thinking of decorators such as: > >> > >> - @typing.assert_params_mismatch - this would provide a wrapper that > >> had auto-generated asserts that all the parameters were of designated > types. > >> - @typing.debug_assert_params_mismatch - this would provide a > wrapper > >> that had auto-generated asserts that all the parameters were of > >> designated types only if a DEBUG environmental variable was set or > similar. > > > > That's what assert does: assert only runs when __DEBUG__ is true. That's > > not controlled by an environment variable, but by the -O flag to the > > interpreter. > > > Good point. > > So your assert_params_mismatch and debug_assert_params_mismatch are > > effectively the same thing. > > > > But using assert to check to perform argument checks is often an abuse > > of assert. To be more specific, using assert to check the value of > > public arguments in library code (where the arguments come from outside > the > > library) is wrong, since you (the library author) cannot guarantee > > that your type tests will even run. > > > > Using asserts for argument checking inside application code is more of a > > grey area, with arguments for and against using assert. > > > > But in my opinion, the deciding factor is nearly always that an > > AssertionError is the wrong sort of exception. Outside of some fairly > > limited circumstances, most of which don't involve type-checking > > function arguments, using assert robs the caller of some useful > > information: the *kind* of error. (TypeError, ValueError, etc.) > > > I see your point here. > > > See here for further discussion: > > > > https://nam06.safelinks.protection.outlook.com/?url= > https%3A%2F%2Fimport-that.dreamwidth.org%2F676.html& > data=02%7C01%7Cgadgetsteve%40live.co.uk%7C5e419db490b84018f2d208d524e6 > 12b4%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0% > 7C636455492928665911&sdata=EFZIvxGxO18vk5s97RnSHH9kAuLS3s > XBxDNS9VbLWlw%3D&reserved=0 > > > > In general, I don't think we want to encourage such runtime type > > testing. Obviously there are exceptions -- library code should > > probably type check arguments, applications perhaps not -- and > > we're not exactly discouraging it either. There are already a number of > > third-party libraries that provide argument type tests at runtime, and > > I think that's probably the right place for them. > > > I'll have to look out for them. > > > > [...] > >> I also think that this might increase the uptake of typing by giving > >> some clear benefits outside of documentation and static type checking. > > > > Problem is, the benefits of runtime type checking aren't clear. But the > > costs certainly are: if you want slow code, do lots and lots of runtime > > type checks. > > > > > > Too much time spent writing safety critical code on my part then! I'll > drop the idea. > -- > Steve (Gadget) Barnes > Any opinions in this message are my personal opinions and do not reflect > those of my employer. > > --- > This email has been checked for viruses by AVG. > http://www.avg.com > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier - http://linkedin.com/in/sfermigier Founder & CEO, Abilian - Enterprise Social Software - http://www.abilian.com/ Chairman, Free&OSS Group / Systematic Cluster - http://www.gt-logiciel-libre.org/ Co-Chairman, National Council for Free & Open Source Software (CNLL) - http://cnll.fr/ Founder & Organiser, PyData Paris - http://pydata.fr/ --- ?You never change things by ?ghting the existing reality. To change something, build a new model that makes the existing model obsolete.? ? R. Buckminster Fuller -------------- next part -------------- An HTML attachment was scrubbed... URL: From sf at fermigier.com Mon Nov 6 05:47:31 2017 From: sf at fermigier.com (=?UTF-8?Q?St=C3=A9fane_Fermigier?=) Date: Mon, 6 Nov 2017 11:47:31 +0100 Subject: [Python-ideas] Possible enhancement to typing In-Reply-To: References: <20171106071357.GC15990@ando.pearwood.info> Message-ID: On Mon, Nov 6, 2017 at 11:34 AM, St?fane Fermigier wrote: > [...] > > Now I believe something could be done along the lines: > > a) record runtime type information from test or regular runs > b) massage these information and use them to annotate Python code with > additional type information (up to the developer to then accept or not the > proposed changes) > c) also run a test suite or an app under some magical machinery, and > either raise a TypeError or log warnings when discrepancies are detected > between type annotation and runtime behaviour. > > [...] (a) and (b) would use similar machinery, > I mean, (a) and (c) > and (a), (b) and (c) would be probably a useful way to introduce type > annotations to an existing code base without too much risk. > > (a) and (b) could also provide data for an interesting SE research project. > Similar to "Measuring Polymorphism in Python Programs", by Beatrice Akerblom and Tobias Wrigstad: https://people.dsv.su.se/~beatrice/python/dls15_large_images.pdf S. -- Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier - http://linkedin.com/in/sfermigier Founder & CEO, Abilian - Enterprise Social Software - http://www.abilian.com/ Chairman, Free&OSS Group / Systematic Cluster - http://www.gt-logiciel-libre.org/ Co-Chairman, National Council for Free & Open Source Software (CNLL) - http://cnll.fr/ Founder & Organiser, PyData Paris - http://pydata.fr/ --- ?You never change things by ?ghting the existing reality. To change something, build a new model that makes the existing model obsolete.? ? R. Buckminster Fuller -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Mon Nov 6 06:49:41 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 6 Nov 2017 12:49:41 +0100 Subject: [Python-ideas] Moving typing out of the stdlib in Python 3.7? References: <20171105114825.6c49bdff@fsol> <20171105194051.58293b33@fsol> Message-ID: <20171106124941.4c00bf99@fsol> On Mon, 6 Nov 2017 07:30:35 +0100 Michel Desmoulin wrote: > Le 06/11/2017 ? 07:07, Nick Coghlan a ?crit?: > > > It's the default on Unix as well - you have to do "make install > > ENSUREPIP=no" to avoid getting it. (And some distros also modify their > > Python installations so that pip is missing by default) > > On debian and derivatives (so Ubuntu) you need to install python-pip to > be able to use pip. I suspect they do this because they want to steer people into trying to `apt install` their Python dependencies first. They have no reason to do any such thing for typing, IMO. Regards Antoine. From contact at brice.xyz Mon Nov 6 06:53:48 2017 From: contact at brice.xyz (Brice Parent) Date: Mon, 6 Nov 2017 12:53:48 +0100 Subject: [Python-ideas] Proposal to change Python version release cycle In-Reply-To: References: <45619991.35926.1509794757663@office.mailbox.org> <20171104142953.0821b213@fsol> <8dd82d18-5828-2356-ee4f-14f6cc9dc14e@mailbox.org> <5bc30002-d6a4-5fad-f1e0-96e2d74a059e@mailbox.org> Message-ID: Le 04/11/17 ? 18:09, Chris Angelico a ?crit?: > Python 1.0 was released in 1994. Then 2.0 came out in 2000 (six > years), and 3.0 in 2008 (eight years). So far, we've been nine years > into 3.0 and aren't looking at 4.0 yet, so it's going to be at least > ten. If version 5.0 is another twelve years after that, and version > 6.0 is fourteen later, we'll be looking for version 31.0 some time > around the year 3000. I think we can let a future generation worry > about pathing problems with DLL names. > > Seriously, I don't think there's any need to stress about version 31 > of something as stable as Python. No matter how long it*actually* is > between versions, it's going to be a lot longer than we can really > plan for right now. In that much time,*anything* could change. I think the only problem we can reach here, not only in our lifetimes, but in the next years, is not Python3.10 vs Python31.0 (Python3.x will be long dead when we reach this point!), but the ordering of versions, like (python310 < python40). But it probably is a false problem, as after a two-digit minor version, we can fix the length of minor versions to two digits when needed (python310 < python400). -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Mon Nov 6 06:51:22 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 6 Nov 2017 12:51:22 +0100 Subject: [Python-ideas] Looking for input to help with the pip situation References: Message-ID: <20171106125122.1e2e25c8@fsol> On Mon, 6 Nov 2017 19:12:39 +1000 Nick Coghlan wrote: > > > I suppose an alternative is to set up jupyterhub > > > > https://jupyterhub.readthedocs.io/en/latest/ > > > > and let all your students just access that from a webbrowser. > > Yep, and lots of teachers use services like PythonAnywhere, > trinket.io, and similar, precisely because the only thing the learners > need locally is a web browser. The main downside is that learning that > way isn't quite as transferable a skill, since it completely hides the > code packaging and distribution step. However, it's a good option when > the primary aim is to teach computational skills, and Python is just > the vehicle used for that purpose. > > Software Carpentry starts out with the Anaconda distribution, as it > not only improves the cross-platform UX consistent situation, it also > deals with the external binary dependency problem (at least for the > core set of packages provided either natively or via conda-forge). And it's quite a large set of core packages there too :-) (disclaimer: I work for Anaconda Inc, though not on the Anaconda distribution) Regards Antoine. From solipsis at pitrou.net Mon Nov 6 06:47:55 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 6 Nov 2017 12:47:55 +0100 Subject: [Python-ideas] Moving typing out of the stdlib in Python 3.7? References: <20171105114825.6c49bdff@fsol> <20171105194051.58293b33@fsol> Message-ID: <20171106124755.5d5f1711@fsol> On Mon, 6 Nov 2017 16:07:56 +1000 Nick Coghlan wrote: > On 6 November 2017 at 05:00, Paul Moore wrote: > > On 5 November 2017 at 18:40, Antoine Pitrou wrote: > >> I think typing shouldn't require any extra typing (ha) on Unix either. > >> I don't remember what the rationale was for having to type > >> "python -m ensurepip" to get pip installed, but typing is just a > >> library, not an executable tool that may be able to mess with the > >> system state, so I don't think it's worthwhile introducing an extra > >> step for it. > > > > Yes, I agree. I didn't realise that "give me pip" was an extra step on > > Unix. I don't know what the arguments for doing it like that on Unix > > were, but they certainly don't seem to apply to typing. > > It's the default on Unix as well - you have to do "make install > ENSUREPIP=no" to avoid getting it. (And some distros also modify their > Python installations so that pip is missing by default) Oh, thank you. I stand corrected then :-) Regards Antoine. From stephanh42 at gmail.com Mon Nov 6 07:19:00 2017 From: stephanh42 at gmail.com (Stephan Houben) Date: Mon, 6 Nov 2017 13:19:00 +0100 Subject: [Python-ideas] Proposal to change Python version release cycle In-Reply-To: References: <45619991.35926.1509794757663@office.mailbox.org> <20171104142953.0821b213@fsol> <8dd82d18-5828-2356-ee4f-14f6cc9dc14e@mailbox.org> <5bc30002-d6a4-5fad-f1e0-96e2d74a059e@mailbox.org> Message-ID: 2017-11-06 12:53 GMT+01:00 Brice Parent : > > I think the only problem we can reach here, not only in our lifetimes, but > in the next years, is not Python3.10 vs Python31.0 (Python3.x will be long > dead when we reach this point!), but the ordering of versions, like > (python310 < python40). But it probably is a false problem, as after a > two-digit minor version, we can fix the length of minor versions to two > digits when needed (python310 < python400). > No probs with either of my proposals: >>> "python39.dll" < "python3A.dll" < "python40.dll" True >>> "python39.dll" < "python3?.dll" < "python40.dll" True Stephan > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vano at mail.mipt.ru Mon Nov 6 09:06:08 2017 From: vano at mail.mipt.ru (Ivan Pozdeev) Date: Mon, 6 Nov 2017 17:06:08 +0300 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: Message-ID: <76bcd6fb-50cb-6987-37db-7bad70649ca3@mail.mipt.ru> On 06.11.2017 9:47, Michel Desmoulin wrote: > Hello, > > Today I'm going to give a training in Python again. > > And again it will go the same way. > > On Mac I will have to make people install python, then tell them to use > pip3. > > On Windows, I will have to warn them about checking the "add python > executable to system path" (that one of them will ALWAYS miss anyway). > Then tell them to use py -3.x -m pip because some of them will have > several versions of Python installed. > > Then on linux, I will tell them to install python-pip and python-venv > and use python3 -m pip. > > I'll talk about --user, but commands won't be usable on some machine > where the Scripts or bin dir is not in the system path. > > Then I will make them create a virtualenv so that they can avoid messing > with their system python and finally can just use "pip install" like in > most tutorials on the Web. > > And eventually I'll talk about pipenv and conda. The first one so they > don't have to think about activating the virtualenv everytime, or pip > freeze, or create the venv, or add it to gitignore, etc. The second > because anaconda is very popular on windows. > > There is no way a beginner is going to get any that by themselves > without a lot of time and pain. They will read some tutorial on the web > and struggle to make sens of what pip is and why "pip install" doesn't > work and why "python sucks". I don't see anything particularly bogging here. It's always like this when you have multiple versions of the same software on the system. There's only one PATH, after all. Heck, *the mere fact that Python allows to work like this is already a huge leap forward.* **Doing this cross-platform with exactly the same steps is something few could believe was even possible a couple of years ago!** With most other software packages, it's either not possible at all (and an installer for a different version of a package requires uninstalling the existing version first), or you need to compile everything from source with custom --prefix or something -- and deal with dependency hell looming over your head when they unexpectedly override some system components, or have to specify paths all the time. This is why system package managers in Linux distributions are such a huge help -- as long as you're using the same repository (or a closely-knitted group thereof), you're supposed to have a sane environment whatever packages you install, with nothing overriding anything else. To simplify things with Python, i do the following: * use the system's Python whenever possible * if using something else, only install the one version/environment that I'm using day-to-day and add it to PATH (system's if safe & convenient, personal otherwise) * prefer the system's/environment's package manager to pip to install 3rd-party modules, too, if there is one. For personal day-to-day use, it's very rarely critical to use precisely specific versions of Python and packages. For commerical use, I'll either mainly use one environment still (the one my production or my client has), or have to deal with multiple environments anyway, or set a up a dedicated CI server/environment and leave its scripts to deal with them. > I think Python is offering an incredible experience for first timer. > However, the whole "where is walpip" shenanigans is not one of them. > > I really want some people from this list to discuss here so we can find > a way to either unify a bit the way we install and use pip, or find a > way to express a tutorial that always works for people on the most > popular platforms and spread the word so that any doc uses it. > > Michel > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- Regards, Ivan From gyromagnetic at gmail.com Mon Nov 6 09:21:54 2017 From: gyromagnetic at gmail.com (Gyro Funch) Date: Mon, 6 Nov 2017 07:21:54 -0700 Subject: [Python-ideas] Moving typing out of the stdlib in Python 3.7? In-Reply-To: References: Message-ID: On 11/4/2017 2:22 PM, Guido van Rossum wrote: > On Sat, Nov 4, 2017 at 7:05 AM, Nick Coghlan > > wrote: > > Perhaps typing could switch to being a bundled module, such that it > had its own version, independent of the Python standard library > version, but was still present by default in new installations? > > > This is beginning to sound like the most attractive solution. We > could possibly do away with typing_extensions. Are there precedents > of how to bundle a module in this way? Or is it going to be another > special case like pip? > > -- > --Guido van Rossum (python.org/~guido ) > Would this mean that other packages in the stdlib with development cycles faster than those of python could use the same bundling mechanism? -gyro From brent.bejot at gmail.com Mon Nov 6 09:48:59 2017 From: brent.bejot at gmail.com (brent bejot) Date: Mon, 6 Nov 2017 09:48:59 -0500 Subject: [Python-ideas] Moving typing out of the stdlib in Python 3.7? In-Reply-To: References: Message-ID: On Mon, Nov 6, 2017 at 9:21 AM, Gyro Funch wrote: > On 11/4/2017 2:22 PM, Guido van Rossum wrote: > > On Sat, Nov 4, 2017 at 7:05 AM, Nick Coghlan > > > > wrote: > > > > Perhaps typing could switch to being a bundled module, such that it > > had its own version, independent of the Python standard library > > version, but was still present by default in new installations? > > > > > > This is beginning to sound like the most attractive solution. We > > could possibly do away with typing_extensions. Are there precedents > > of how to bundle a module in this way? Or is it going to be another > > special case like pip? > > > > -- > > --Guido van Rossum (python.org/~guido ) > > > > > Would this mean that other packages in the stdlib with development > cycles faster than those of python could use the same bundling > mechanism? > This is pretty much how I thought things worked for all built-in packages until this thread came up. Is there any reason to not do this for all of stdlib? -------------- next part -------------- An HTML attachment was scrubbed... URL: From facundobatista at gmail.com Mon Nov 6 10:38:05 2017 From: facundobatista at gmail.com (Facundo Batista) Date: Mon, 6 Nov 2017 12:38:05 -0300 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: Message-ID: 2017-11-06 3:47 GMT-03:00 Michel Desmoulin : > Today I'm going to give a training in Python again. > > And again it will go the same way. > > On Mac I will have to make people install python, then tell them to use > pip3. > > On Windows, I will have to warn them about checking the "add python > executable to system path" (that one of them will ALWAYS miss anyway). > Then tell them to use py -3.x -m pip because some of them will have > several versions of Python installed. > > Then on linux, I will tell them to install python-pip and python-venv > and use python3 -m pip. > > I'll talk about --user, but commands won't be usable on some machine > where the Scripts or bin dir is not in the system path. > > Then I will make them create a virtualenv so that they can avoid messing > with their system python and finally can just use "pip install" like in > most tutorials on the Web. > > And eventually I'll talk about pipenv and conda. The first one so they > don't have to think about activating the virtualenv everytime, or pip > freeze, or create the venv, or add it to gitignore, etc. The second > because anaconda is very popular on windows. > > There is no way a beginner is going to get any that by themselves > without a lot of time and pain. They will read some tutorial on the web > and struggle to make sens of what pip is and why "pip install" doesn't > work and why "python sucks". Do you know about "fades"? https://fades.readthedocs.io/en/release_6_0/ fades is a system that automatically handles the virtualenvs in the cases normally found when writing scripts and simple programs, and even helps to administer big projects. It will automagically create a new virtualenv (or reuse a previous created one), installing the necessary dependencies, and execute your script inside that virtualenv, with the only requirement of executing the script with fades and also marking the required dependencies. It works in linux, mac and windows. So your only thing to do for students to run projects inside virtualenvs with X and Y dependencies is to `fades -d X -d Y script.py`, in all environments. Regards, -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ Twitter: @facundobatista From joshua.morton13 at gmail.com Mon Nov 6 11:23:54 2017 From: joshua.morton13 at gmail.com (Joshua Morton) Date: Mon, 06 Nov 2017 16:23:54 +0000 Subject: [Python-ideas] Proposal to change Python version release cycle In-Reply-To: References: <45619991.35926.1509794757663@office.mailbox.org> <20171104142953.0821b213@fsol> <8dd82d18-5828-2356-ee4f-14f6cc9dc14e@mailbox.org> <5bc30002-d6a4-5fad-f1e0-96e2d74a059e@mailbox.org> Message-ID: Isn't this recent bit of discussion an argument in favor of a new stdlib module `version`? That would contain things like resolving a version tuple to an executable name (or a cross-platform piece of an executable name)? Obviously this doesn't actually answer the question of how naming should be done in the future, but it does mean that (new) code will have the actual choice abstracted away. --Josh On Mon, Nov 6, 2017 at 4:20 AM Stephan Houben wrote: > 2017-11-06 12:53 GMT+01:00 Brice Parent : > >> >> I think the only problem we can reach here, not only in our lifetimes, >> but in the next years, is not Python3.10 vs Python31.0 (Python3.x will be >> long dead when we reach this point!), but the ordering of versions, like >> (python310 < python40). But it probably is a false problem, as after a >> two-digit minor version, we can fix the length of minor versions to two >> digits when needed (python310 < python400). >> > > No probs with either of my proposals: > > >>> "python39.dll" < "python3A.dll" < "python40.dll" > True > >>> "python39.dll" < "python3?.dll" < "python40.dll" > True > > Stephan > > >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Mon Nov 6 11:51:33 2017 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 06 Nov 2017 08:51:33 -0800 Subject: [Python-ideas] Moving typing out of the stdlib in Python 3.7? In-Reply-To: References: Message-ID: <5A009315.8060200@stoneleaf.us> On 11/06/2017 06:48 AM, brent bejot wrote: > On Mon, Nov 6, 2017 at 9:21 AM, Gyro Funch wrote: >> Would this mean that other packages in the stdlib with development >> cycles faster than those of python could use the same bundling >> mechanism? > > This is pretty much how I thought things worked for all built-in packages until this thread came up. Is there any > reason to not do this for all of stdlib? Yes. Stability is probably the first (knowing what is support in a particular version). Second would be that nearly all modules in the stdlib are stable and don't need frequent updates -- in fact, I suspect typing is the only exception. -- ~Ethan~ From desmoulinmichel at gmail.com Mon Nov 6 12:45:17 2017 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Mon, 6 Nov 2017 18:45:17 +0100 Subject: [Python-ideas] Moving typing out of the stdlib in Python 3.7? In-Reply-To: References: Message-ID: <10482590-80c3-43ef-5bf2-c2ddb94d6e54@gmail.com> > > > This is pretty much how I thought things worked for all built-in > packages until this thread came up.? Is there any reason to not do this > for all of stdlib? > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > Yes: https://www.python.org/dev/peps/pep-0206/#batteries-included-philosophy I can't find it but there is a great story about a guy needing to do a source code analysis in total isolation (not even a usb key or a cd) and the computer has nothing installed but Python. He gets by because Python is very useful "as is". I can say the same. When I log in on bare debian / centos or if I'm on a locked mac os. When i'm on a place or in a remote place in Africa. When I'm with customers locking down all the network with NDA, security badge and paranoia for desert... I can count on Python to be there for me and hashlib the stuff, etree thingy, zipfile the foo and uuid all the bars. It's a major feature. I can live without typing, but I would hate having to pip install a json parser. From desmoulinmichel at gmail.com Mon Nov 6 12:49:27 2017 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Mon, 6 Nov 2017 18:49:27 +0100 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: Message-ID: <1057b1b9-9e77-b9d6-0ff8-d106860c8b29@gmail.com> > > Do you know about "fades"? > > https://fades.readthedocs.io/en/release_6_0/ So when they read any tutorial online they don't have reusable knowledge ? From desmoulinmichel at gmail.com Mon Nov 6 12:52:17 2017 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Mon, 6 Nov 2017 18:52:17 +0100 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: Message-ID: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> Le 06/11/2017 ? 09:47, Nick Coghlan a ?crit?: > On 6 November 2017 at 16:47, Michel Desmoulin wrote: >> I really want some people from this list to discuss here so we can find >> a way to either unify a bit the way we install and use pip, or find a >> way to express a tutorial that always works for people on the most >> popular platforms and spread the word so that any doc uses it. > > https://docs.python.org/3/installing/#basic-usage is as close as we've > been able to get to that for the time being. I know and you still: - have to use py -m on windows, python3 linux, python in virtualenv... - install pip manually on linux - make sure the system path is correctly set Stuff that they will forget on the next install, or miss when changing plateform. And assume that stuff in any tutorial you make they know this stuff. This is a strong barrier or entry IMO. From desmoulinmichel at gmail.com Mon Nov 6 12:55:36 2017 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Mon, 6 Nov 2017 18:55:36 +0100 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: Message-ID: <2861c930-1705-9e0f-0edc-def7618cd2f6@gmail.com> Le 06/11/2017 ? 09:50, Stephan Houben a ?crit?: > Hi Michel, > > That's exactly why I proposed a `pip` function available from the Python > prompt. > I suppose you could still tell your students to copy/paste the following > into their > Python interpreter. > > def pip(args): > ??? import sys > ??? import subprocess > ??? subprocess.check_call([sys.executable, "-m", "pip"] + args.split()) > ??? print("Please restart Python now to use installed or upgraded > packages.") You still need to install pip on platforms it's not available. And do that in a virtualenv or do --user. And ask for a restart. > I suppose an alternative is to set up jupyterhub > > https://jupyterhub.readthedocs.io/en/latest/ > I love jupyter, but you can't limit yourself to it. That won't give them autonomy and will limit very much what they can do. E.G: You can't use tkinter in jupyter easily. From desmoulinmichel at gmail.com Mon Nov 6 12:57:38 2017 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Mon, 6 Nov 2017 18:57:38 +0100 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: Message-ID: > > Software Carpentry starts out with the Anaconda distribution, as it > not only improves the cross-platform UX consistent situation, it also > deals with the external binary dependency problem (at least for the > core set of packages provided either natively or via conda-forge). > Yeah but the knowledge is not transposable. You can't understand a lot of tutorials that uses pip. Plus it has licences issues. And if somebody starts mixing conda and pip, it can results in strange things. I do use anaconda, but you can't rely solely on it. From desmoulinmichel at gmail.com Mon Nov 6 13:04:25 2017 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Mon, 6 Nov 2017 19:04:25 +0100 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: <76bcd6fb-50cb-6987-37db-7bad70649ca3@mail.mipt.ru> References: <76bcd6fb-50cb-6987-37db-7bad70649ca3@mail.mipt.ru> Message-ID: > I don't see anything particularly bogging here. > It's always like this when you have multiple versions of the same > software on the system. There's only one PATH, after all. > > Heck, *the mere fact that Python allows to work like this is already a > huge leap forward.* **Doing this cross-platform with exactly the same > steps is something few could believe was even possible a couple of years > ago!** I agree. > > To simplify things with Python, i do the following: > * use the system's Python whenever possible So python 2.7 on mac and some linux or none for windows... > * if using something else, only install the one version/environment that > I'm using day-to-day and add it to PATH (system's if safe & convenient, > personal otherwise) > * prefer the system's/environment's package manager to pip to install > 3rd-party modules, too, if there is one. We can't solve the situation perfectly, but we can unify a bit. E.G: - provide the "py" command on all OSes to avoid the various naming and aliases of python - promote it in documentation - promote the use of py -x.x -m pip instead of the myriads of alternatives - provide an empty pip and venv module. If they are not here, py -m pip says "your plateform doesn't provide pip by default, please do xxxxx" to install it. With "xxxx" being plateform specific. - check "add python executable to system path" on the windows installer once by defaut From chris.barker at noaa.gov Mon Nov 6 17:31:15 2017 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 6 Nov 2017 14:31:15 -0800 Subject: [Python-ideas] install pip packages from Python prompt In-Reply-To: References: <01f701d350eb$c4ee6210$4ecb2630$@sdamon.com> <021601d350ee$88f236d0$9ad6a470$@sdamon.com> <045101d351b9$54a25240$fde6f6c0$@sdamon.com> <23037.27860.762729.984015@turnbull.sk.tsukuba.ac.jp> Message-ID: On Sat, Nov 4, 2017 at 7:16 AM, Wes Turner wrote: > > This works today but is inadvisable because it risks wasting bandwidth if > the %run (!) instructions are frequently re-run unnecessarily: > > $ ipython > >>> !pip install -U ipython > >>> exit() > $ ipython > does it work on Windows? My experience is almost always negative with Windows' aggressive file locking -- I'd be unsurprised if you couldn't update a package that is running (particularly if it is has loaded a dll, AKA compiled extension module) -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Mon Nov 6 17:48:04 2017 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 6 Nov 2017 14:48:04 -0800 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> Message-ID: On Mon, Nov 6, 2017 at 9:52 AM, Michel Desmoulin wrote: > I know and you still: > > - have to use py -m on windows, python3 linux, python in virtualenv... > can't you use python3 -m pip install ..... everywhere? That's what I tell my beginner students to do, and I've never had a problem. (nce they got Python installed right in the first place) For that: The python.og installers for Windows and Mac pretty much "just work" Linux is a different story, but Linux users are more comfortable with eh whole idea of command lines and packages, etc -- so my Linux users have never been the hangup. If/when I'm teaching data analysis for scientific computing, I go straight to conda, but for basic python, and most web Development, pyton.org python and pip work great. A also DO NOT introduce virtualenv right off the bat -- it is another complication that is critical to real development, but not important for learning python. I have done it it the past -- it did not go well.... What's too bad now is that so many docs say "pip install the_package", and users with multiple python installs can get burnt. (though most don't) -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From vano at mail.mipt.ru Mon Nov 6 17:53:22 2017 From: vano at mail.mipt.ru (Ivan Pozdeev) Date: Tue, 7 Nov 2017 01:53:22 +0300 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> Message-ID: <2f371148-b93b-57d3-10a8-28b5bd5011d8@mail.mipt.ru> On 07.11.2017 1:48, Chris Barker wrote: > On Mon, Nov 6, 2017 at 9:52 AM, Michel Desmoulin > > wrote: > > I know and you still: > > - have to use py -m on windows, python3 linux, python in virtualenv... > > > can't you use python3 -m pip install ..... > > everywhere? You can't. Windows versions don't create versioned executables. Got bitten with this myself. ...Maybe they should? (This is python-ideas, after all ;-) ) -- Regards, Ivan -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Mon Nov 6 18:32:00 2017 From: barry at python.org (Barry Warsaw) Date: Mon, 06 Nov 2017 15:32:00 -0800 Subject: [Python-ideas] Proposal to change Python version release cycle In-Reply-To: References: <45619991.35926.1509794757663@office.mailbox.org> <20171104142953.0821b213@fsol> <8dd82d18-5828-2356-ee4f-14f6cc9dc14e@mailbox.org> <5bc30002-d6a4-5fad-f1e0-96e2d74a059e@mailbox.org> Message-ID: Nick Coghlan wrote: > The open question will be what we call the release after that (in late > 2022), with the main leading contenders being "4.0" (on the grounds > that so many changes will have accumulated since 2008's 3.0 release by > then that it makes sense to call them different major versions, > similar to the rationale the Linux kernel now uses for major version > updates) I predict by then that Larry will have finished the gilectomy, and while it will require backward incompatible changes to the C API, Python will run so blindingly fast in the quantum computing machine learning blockchain in the IOT cloud, bumping to 4.0 will be a no brainer[*]. Cheers, -Barry [*] Plus, our AI overlords will have made the decision for us. P.S. I suck at predictions. From barry at python.org Mon Nov 6 20:18:57 2017 From: barry at python.org (Barry Warsaw) Date: Mon, 06 Nov 2017 17:18:57 -0800 Subject: [Python-ideas] Proposal to change Python version release cycle In-Reply-To: References: <45619991.35926.1509794757663@office.mailbox.org> <20171104142953.0821b213@fsol> <8dd82d18-5828-2356-ee4f-14f6cc9dc14e@mailbox.org> Message-ID: M.-A. Lemburg wrote: > The first ever major backwards incompatibility release switch we > had in Python after the great renaming of the C APIs between > Python 1.1 and 1.2 (which was only visible to C extensions and > relatively easy to fix using a compatibility header file), > was the transition from Python 2.x to Python 3.x. Fond memories of some of my first contributions to Python: http://python-history.blogspot.com/2009/03/great-or-grand-renaming.html Cheers, -Barry From barry at python.org Mon Nov 6 20:22:22 2017 From: barry at python.org (Barry Warsaw) Date: Mon, 06 Nov 2017 17:22:22 -0800 Subject: [Python-ideas] Proposal to change Python version release cycle In-Reply-To: References: <45619991.35926.1509794757663@office.mailbox.org> <20171104142953.0821b213@fsol> <8dd82d18-5828-2356-ee4f-14f6cc9dc14e@mailbox.org> Message-ID: Nick Timkovich wrote: > Alternative history question: if it was just 1.6 Guido's time machine strikes again. There was both a Python 1.6 and a 1.6.1. https://www.python.org/download/releases/1.6/ https://www.python.org/download/releases/1.6.1/ The "Contractual Obligation" releases. (And another Monty Python's Flying Circus reference.) These resolved licensing and release issues related to the BeOpen adventure and GPL-compatibility. Cheers, -Barry From barry at python.org Mon Nov 6 20:27:31 2017 From: barry at python.org (Barry Warsaw) Date: Mon, 06 Nov 2017 17:27:31 -0800 Subject: [Python-ideas] Moving typing out of the stdlib in Python 3.7? In-Reply-To: References: Message-ID: Guido van Rossum wrote: > This is beginning to sound like the most attractive solution. We could > possibly do away with typing_extensions. Are there precedents of how to > bundle a module in this way? Or is it going to be another special case like > pip? With my downstream consumer hat on, *if* we go with a bundling solution, please make sure it's easy to unbundle. Downstream consumers may need to do this for policy, security, etc. reasons, so making this too difficult will mean delays in releases, convoluted hacks, policy violations, or all of the above. See as reference some of the hoops Debian has to go through to handle ensurepip. Cheers, -Barry From guido at python.org Mon Nov 6 20:58:02 2017 From: guido at python.org (Guido van Rossum) Date: Mon, 6 Nov 2017 17:58:02 -0800 Subject: [Python-ideas] Possible enhancement to typing In-Reply-To: References: <20171106071357.GC15990@ando.pearwood.info> Message-ID: On Mon, Nov 6, 2017 at 2:34 AM, St?fane Fermigier wrote: > 4) 10 years ago, when I was working on the EDOS project ( > http://cordis.europa.eu/pub/ist/docs/directorate_d/st-ds/ > edos-project-story_en.pdf ), I ran a small experiment where I used, IIRC, > the profile hook to intercept all function / method calls, and log > information about arguments and return value types to a gigantic log file. > Then the log file could be parsed and these information used to suggest > type annotations. Except there were no type annotations at the time in > Python. > > I know PyCharm can do a similar thing now: you run your program or your > tests under the debugger, it logs runtime type information somewhere, and > then can use it to suggest autocompletion or maybe type annotations. > I didn't know this. Do you know where there are docs for this feature? > Now I believe something could be done along the lines: > > a) record runtime type information from test or regular runs > b) massage these information and use them to annotate Python code with > additional type information (up to the developer to then accept or not the > proposed changes) > We have an early version of a tool that does this at Dropbox; I am planning to open-source it by the end of this year. So far the experience is that the annotations require a fair amount of manual cleanup though. [snip] > Similar to "Measuring Polymorphism in Python Programs", by Beatrice Akerblom and Tobias Wrigstad: > https://people.dsv.su.se/~beatrice/python/dls15_large_images.pdf Thanks for the link! -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Nov 6 23:35:40 2017 From: guido at python.org (Guido van Rossum) Date: Mon, 6 Nov 2017 20:35:40 -0800 Subject: [Python-ideas] Proposal to change Python version release cycle In-Reply-To: References: <45619991.35926.1509794757663@office.mailbox.org> <20171104142953.0821b213@fsol> <8dd82d18-5828-2356-ee4f-14f6cc9dc14e@mailbox.org> Message-ID: What a soap opera. :-) On Mon, Nov 6, 2017 at 5:22 PM, Barry Warsaw wrote: > Nick Timkovich wrote: > > > Alternative history question: if it was just 1.6 > > Guido's time machine strikes again. There was both a Python 1.6 and a > 1.6.1. > > https://www.python.org/download/releases/1.6/ > https://www.python.org/download/releases/1.6.1/ > > The "Contractual Obligation" releases. (And another Monty Python's > Flying Circus reference.) These resolved licensing and release issues > related to the BeOpen adventure and GPL-compatibility. > > Cheers, > -Barry > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From sf at fermigier.com Tue Nov 7 01:36:35 2017 From: sf at fermigier.com (=?UTF-8?Q?St=C3=A9fane_Fermigier?=) Date: Tue, 7 Nov 2017 07:36:35 +0100 Subject: [Python-ideas] Possible enhancement to typing In-Reply-To: References: <20171106071357.GC15990@ando.pearwood.info> Message-ID: On Tue, Nov 7, 2017 at 2:58 AM, Guido van Rossum wrote: > On Mon, Nov 6, 2017 at 2:34 AM, St?fane Fermigier > wrote: > >> 4) 10 years ago, when I was working on the EDOS project ( >> http://cordis.europa.eu/pub/ist/docs/directorate_d/st-ds/edo >> s-project-story_en.pdf ), I ran a small experiment where I used, IIRC, >> the profile hook to intercept all function / method calls, and log >> information about arguments and return value types to a gigantic log file. >> Then the log file could be parsed and these information used to suggest >> type annotations. Except there were no type annotations at the time in >> Python. >> >> I know PyCharm can do a similar thing now: you run your program or your >> tests under the debugger, it logs runtime type information somewhere, and >> then can use it to suggest autocompletion or maybe type annotations. >> > > I didn't know this. Do you know where there are docs for this feature? > This was described in this blog post when first introduced: https://blog.jetbrains.com/pycharm/2013/02/dynamic-runtime-type-inference-in-pycharm-2-7/ And more tersely, in the documentation: https://www.jetbrains.com/help/pycharm/python-debugger.html a) record runtime type information from test or regular runs >> b) massage these information and use them to annotate Python code with >> additional type information (up to the developer to then accept or not the >> proposed changes) >> > > We have an early version of a tool that does this at Dropbox; I am > planning to open-source it by the end of this year. So far the experience > is that the annotations require a fair amount of manual cleanup though. > That would be great ! S. -- Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier - http://linkedin.com/in/sfermigier Founder & CEO, Abilian - Enterprise Social Software - http://www.abilian.com/ Chairman, Free&OSS Group / Systematic Cluster - http://www.gt-logiciel-libre.org/ Co-Chairman, National Council for Free & Open Source Software (CNLL) - http://cnll.fr/ Founder & Organiser, PyData Paris - http://pydata.fr/ --- ?You never change things by ?ghting the existing reality. To change something, build a new model that makes the existing model obsolete.? ? R. Buckminster Fuller -------------- next part -------------- An HTML attachment was scrubbed... URL: From elazarg at gmail.com Tue Nov 7 05:29:33 2017 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Tue, 07 Nov 2017 10:29:33 +0000 Subject: [Python-ideas] Any chance on (slowly) deprecating `eval` and `exec` as builtins? Message-ID: Hi, The dangers of eval and exec are obvious and well known to advanced users, but the availability as built-in functions makes it too tempting for beginners or even medium-level programmers. You can see questions about these function pretty often in stackoverflow (roughly once a day , though sometimes the uses are legitimate). Maybe we could start a ten-year process of deprecating the use of `builtins.eval` (in the docs, and then with warnings)? `builtins.eval` will be a wrapper to the real evaluation function, moved to `unsafe.eval` or something obvious like that, so all you need to do to port your code is to add `from unsafe import unsafe_eval as eval, unsafe_exec as exec` at the top of the file; it will be a nice warning to the reader. The fact that it is a wrapper will slightly slow it down and make the stack traces noisier - both are good things, IMO. Also, it is unfortunate that `ast.literal_eval` is less accessible than `builtins.eval`. Giving it an alias in builtins might make it easier for programmers (and less scary - "ast" might sound like I need a PhD to use it). What do you think? Elazar -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Nov 7 07:44:18 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 7 Nov 2017 22:44:18 +1000 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> Message-ID: On 7 November 2017 at 03:52, Michel Desmoulin wrote: > > > Le 06/11/2017 ? 09:47, Nick Coghlan a ?crit : >> On 6 November 2017 at 16:47, Michel Desmoulin wrote: >>> I really want some people from this list to discuss here so we can find >>> a way to either unify a bit the way we install and use pip, or find a >>> way to express a tutorial that always works for people on the most >>> popular platforms and spread the word so that any doc uses it. >> >> https://docs.python.org/3/installing/#basic-usage is as close as we've >> been able to get to that for the time being. > > I know and you still: > > - have to use py -m on windows, python3 linux, python in virtualenv... Which is why we advise getting into a virtual environment ASAP, such that the only platform specific thing folks necessarily need to learn to get started is how to get to that first working virtual environment. > - install pip manually on linux s/Linux/Ubuntu/ Other distros (like Fedora) provide pip by default. > - make sure the system path is correctly set Recent python.org Windows installers do this automatically, but there are unfortunately still lots of ways for things to go wrong. > Stuff that they will forget on the next install, or miss when changing > plateform. Yes, because computers are awful, and incredibly user hostile. We don't have a magic wand to wave to fix that. > And assume that stuff in any tutorial you make they know this stuff. > > This is a strong barrier or entry IMO. Sure, but it's not one we can readily fix - the user hostility of command line environments and the compromises we have to make to abide by platform conventions are in the hands of operating system vendors, and there's only so much we can do to paper over those distinctions when user lock-in and putting barriers in the way of cross-device portability is a core part of commercial OS vendors' business models. This is a big part of why mobile client devices with cloud backends are so popular, even for development purposes: they allow for a much simpler developer experience that avoids decades of accumulated cruft in the desktop operating system command line experience. Even there though, you're faced with the fact that once you choose a provider, whatever you do there will probably be locked into that provider and not transferable elsewhere. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Tue Nov 7 07:52:00 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 7 Nov 2017 22:52:00 +1000 Subject: [Python-ideas] Proposal to change Python version release cycle In-Reply-To: References: <45619991.35926.1509794757663@office.mailbox.org> <20171104142953.0821b213@fsol> <8dd82d18-5828-2356-ee4f-14f6cc9dc14e@mailbox.org> <5bc30002-d6a4-5fad-f1e0-96e2d74a059e@mailbox.org> Message-ID: On 6 November 2017 at 22:19, Stephan Houben wrote: > 2017-11-06 12:53 GMT+01:00 Brice Parent : >> I think the only problem we can reach here, not only in our lifetimes, but >> in the next years, is not Python3.10 vs Python31.0 (Python3.x will be long >> dead when we reach this point!), but the ordering of versions, like >> (python310 < python40). But it probably is a false problem, as after a >> two-digit minor version, we can fix the length of minor versions to two >> digits when needed (python310 < python400). > > No probs with either of my proposals: > >>>> "python39.dll" < "python3A.dll" < "python40.dll" > True Ah, you're right, I forgot about that option (I think Ezio Melotti suggested it previously). Yes, going with "3.10", but encoding it as "3A" in lexically ambiguous contexts is another option that would let us get as far as 3.35 (aka "3Z") before encountering ambiguity problems. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From elazarg at gmail.com Tue Nov 7 08:06:27 2017 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Tue, 07 Nov 2017 13:06:27 +0000 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> Message-ID: On Tue, Nov 7, 2017 at 2:45 PM Nick Coghlan wrote: > On 7 November 2017 at 03:52, Michel Desmoulin > wrote: > And assume that stuff in any tutorial you make they know this stuff. > > > > This is a strong barrier or entry IMO. > > Sure, but it's not one we can readily fix - the user hostility of > command line environments and the compromises we have to make to abide > by platform conventions are in the hands of operating system vendors, > and there's only so much we can do to paper over those distinctions > when user lock-in and putting barriers in the way of cross-device > portability is a core part of commercial OS vendors' business models. > > I don't know if you are referring to Microsoft Windows here, but I want to note that from my personal experience the Windows subsystem for Linux ("Bash on Ubuntu on Windows") is easy to work with, so making Windows feel (CLI-wise) like Ubuntu is not so difficult. I'm not sure how easy it is for students to set up, but it is an option at least. Elazar -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjol at tjol.eu Tue Nov 7 08:07:38 2017 From: tjol at tjol.eu (Thomas Jollans) Date: Tue, 7 Nov 2017 14:07:38 +0100 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: <2f371148-b93b-57d3-10a8-28b5bd5011d8@mail.mipt.ru> References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2f371148-b93b-57d3-10a8-28b5bd5011d8@mail.mipt.ru> Message-ID: <0c38c6e3-8515-6c07-21c6-9694db4f8df8@tjol.eu> On 2017-11-06 23:53, Ivan Pozdeev via Python-ideas wrote: > On 07.11.2017 1:48, Chris Barker wrote: >> On Mon, Nov 6, 2017 at 9:52 AM, Michel Desmoulin >> > wrote: >> >> I know and you still: >> >> - have to use py -m on windows, python3 linux, python in virtualenv... >> >> >> can't you use python3 -m pip install ..... >> >> everywhere? > You can't. Windows versions don't create versioned executables. Got > bitten with this myself. Once you're worked around that in the first place (lesson: "this is how you start Python"), then figuring out whether you need "python -m pip" or "python3 -m pip" shouldn't be a big deal. Still the fact that the way you call Python is different on different platforms is rather unfortunate. -- Thomas From p.f.moore at gmail.com Tue Nov 7 08:08:17 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 7 Nov 2017 13:08:17 +0000 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> Message-ID: On 7 November 2017 at 12:44, Nick Coghlan wrote: >> - make sure the system path is correctly set > > Recent python.org Windows installers do this automatically, but there > are unfortunately still lots of ways for things to go wrong. I believe the latest installers switch it off again, because one of the ways things can go wrong is that stuff put at the start of the user path is still lower priority than stuff in the system path, and we now default to user installs. This is an actual problem with mixed python.org and Anaconda installations, for example - Anaconda adds itself to the system PATH, and overrides a default user install of python.org Python. So you can't prioritise python.org over Anaconda without manual path hacking. (This hit me when I installed Visual Studio and selected "include Python (Anaconda)" - I can't recall the exact option, but it broke my python.org install). As you say command-line environments are just inherently user-hostile, and we can't do a lot about that. People who buy into the command line experience learn how to deal with the complexity. People who use IDEs like Visual Studio or PyCharm can rely on the IDE vendor to provide a clean experience. But people who want to use core Python but who aren't comfortable fighting with the command line have a bit of a problem. We can't solve that problem for them, the best we can do is offer suggestions on best practices, or tools that help alleviate the issues. Paul From tjol at tjol.eu Tue Nov 7 08:04:58 2017 From: tjol at tjol.eu (Thomas Jollans) Date: Tue, 7 Nov 2017 14:04:58 +0100 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <76bcd6fb-50cb-6987-37db-7bad70649ca3@mail.mipt.ru> Message-ID: <479e4e16-7c73-df23-6a35-eff9b381ef31@tjol.eu> On 2017-11-06 19:04, Michel Desmoulin wrote: > >> I don't see anything particularly bogging here. >> It's always like this when you have multiple versions of the same >> software on the system. There's only one PATH, after all. >> >> Heck, *the mere fact that Python allows to work like this is already a >> huge leap forward.* **Doing this cross-platform with exactly the same >> steps is something few could believe was even possible a couple of years >> ago!** > > I agree. > >> >> To simplify things with Python, i do the following: >> * use the system's Python whenever possible > > So python 2.7 on mac and some linux or none for windows... > >> * if using something else, only install the one version/environment that >> I'm using day-to-day and add it to PATH (system's if safe & convenient, >> personal otherwise) >> * prefer the system's/environment's package manager to pip to install >> 3rd-party modules, too, if there is one. > > We can't solve the situation perfectly, but we can unify a bit. E.G: > > - provide the "py" command on all OSes to avoid the various naming and > aliases of python > - promote it in documentation > - promote the use of py -x.x -m pip instead of the myriads of alternatives I'm sure there's a reason why these acrobatics are required on Windows, but on other OS's separate "python3", "python3.6", "python2", "python2.7" and "python" executables/symlinks all in the PATH work perfectly fine. As Ivan said earlier, perhaps the Windows installers should provide a "python3" executable, so "python3 -m pip" works everywhere. I normally use "pip2" and "pip3", but those aren't available everywhere, and may be hard to support properly. > - provide an empty pip and venv module. If they are not here, py -m pip > says "your plateform doesn't provide pip by default, please do xxxxx" to > install it. With "xxxx" being plateform specific. Isn't pip installed by default on Windows and OSX? On Linux some distribution will probably do something unexpected and mess up your grand plan. > - check "add python executable to system path" on the windows installer > once by defaut I like this, but I have a suspicion that this is rather unorthodox. -- Thomas Jollans From p.f.moore at gmail.com Tue Nov 7 08:11:35 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 7 Nov 2017 13:11:35 +0000 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> Message-ID: On 7 November 2017 at 13:06, ????? wrote: > On Tue, Nov 7, 2017 at 2:45 PM Nick Coghlan wrote: >> >> On 7 November 2017 at 03:52, Michel Desmoulin >> wrote: >> >> > And assume that stuff in any tutorial you make they know this stuff. >> > >> > This is a strong barrier or entry IMO. >> >> Sure, but it's not one we can readily fix - the user hostility of >> command line environments and the compromises we have to make to abide >> by platform conventions are in the hands of operating system vendors, >> and there's only so much we can do to paper over those distinctions >> when user lock-in and putting barriers in the way of cross-device >> portability is a core part of commercial OS vendors' business models. >> > > I don't know if you are referring to Microsoft Windows here, but I want to > note that from my personal experience the Windows subsystem for Linux ("Bash > on Ubuntu on Windows") is easy to work with, so making Windows feel > (CLI-wise) like Ubuntu is not so difficult. I'm not sure how easy it is for > students to set up, but it is an option at least. It is, but like any such approach (Cygwin is similar, in principle if not in execution) that makes one OS "look like" another, whether it's appropriate is very dependent on circumstances. Training potential Windows developers in a bash/Ubuntu style environment leaves them at a disadvantage when they need to develop for actual Windows environments. It's certainly an option, but so is "train them on Linux". Paul From vano at mail.mipt.ru Tue Nov 7 08:28:11 2017 From: vano at mail.mipt.ru (Ivan Pozdeev) Date: Tue, 7 Nov 2017 16:28:11 +0300 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> Message-ID: <098c266d-088e-1008-b2c5-05aac4a9770b@mail.mipt.ru> On 07.11.2017 16:11, Paul Moore wrote: > It is, but like any such approach (Cygwin is similar, in principle if > not in execution) that makes one OS "look like" another, whether it's > appropriate is very dependent on circumstances. Training potential > Windows developers in a bash/Ubuntu style environment leaves them at a > disadvantage when they need to develop for actual Windows > environments. Moreover, one of the key selling points of Python is that it can interface with anything and everything found in any environment! Rather than e.g. conjuring up a closed circle with own replacements for everything than forces everyone into the lowest common denominator like Java does. *That's another key reason why I use & suggest using an installation "native" to the system whenever possible.* It's as close to "being an integral part of the environment rather than rejecting it and creating an inferior copy" philosophy as one can get. -- Regards, Ivan From storchaka at gmail.com Tue Nov 7 08:35:58 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 7 Nov 2017 15:35:58 +0200 Subject: [Python-ideas] Any chance on (slowly) deprecating `eval` and `exec` as builtins? In-Reply-To: References: Message-ID: 07.11.17 12:29, ????? ????: > Also, it is unfortunate that `ast.literal_eval` is less accessible than > `builtins.eval`. Giving it an alias in builtins might make it easier for > programmers (and less scary - "ast" might sound like I need a PhD to use > it). ast.literal_eval is not so safe as you think. Malicious input can cause a stack overflow in your program. [1] [1] https://bugs.python.org/issue31113 From steve at pearwood.info Tue Nov 7 09:41:16 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 8 Nov 2017 01:41:16 +1100 Subject: [Python-ideas] Any chance on (slowly) deprecating `eval` and `exec` as builtins? In-Reply-To: References: Message-ID: <20171107144116.GA19802@ando.pearwood.info> On Tue, Nov 07, 2017 at 03:35:58PM +0200, Serhiy Storchaka wrote: > 07.11.17 12:29, ????? ????: > >Also, it is unfortunate that `ast.literal_eval` is less accessible than > >`builtins.eval`. Giving it an alias in builtins might make it easier for > >programmers (and less scary - "ast" might sound like I need a PhD to use > >it). > > ast.literal_eval is not so safe as you think. Malicious input can cause > a stack overflow in your program. [1] > > [1] https://bugs.python.org/issue31113 I don't see anything about literal_eval in that bug report. In any case, I think that securing literal_eval is much simpler than securing eval: try: # a thousand character expression ought to be enough for # any legitimate purpose... value = literal_eval(tainted_string[:1000]) # untested except MemoryError: value = None versus, well, there's no obvious or simple way to secure eval. You might get something reasonable with: if '_' in tainted_string: raise ValueError("no underscores allowed!") globalns = {'__builtins__': None} # I think? eval(tainted_string[:1000], globalns, globalns) but even that is probably vulnerable to a clever enough attacker. I tried this on my computer with Python 3.5: # Build a deeply nested dict {'x':{'x':{...}}} as a string s = "{'x':0}" for i in range(1665): s = "{'x':%s}" % s assert len(s) < 10000 d = ast.literal_eval(s) and it failed with MemoryError rather than a segfault, so that's something. -- Steve From stephanh42 at gmail.com Tue Nov 7 10:34:20 2017 From: stephanh42 at gmail.com (Stephan Houben) Date: Tue, 7 Nov 2017 16:34:20 +0100 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: <2f371148-b93b-57d3-10a8-28b5bd5011d8@mail.mipt.ru> References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2f371148-b93b-57d3-10a8-28b5bd5011d8@mail.mipt.ru> Message-ID: One (smaller) suggestion on the PATH situation on Windows: I noticed that Visual C++ Build Tools installs a number of "Command prompts" under its Start menu item, each of which starts a cmd.exe with appropriate PATH set to the appropriate compiler (32/64 bits or ARM cross-compiler), and assorted environment variables set to the appropriate include/library directories. Could we do something similar for Python? I.e., Install under the "Python 3.6" start menu an additional "Python command prompt", which will start cmd.exe with an appropriate PATH so that python and pip run without further prefix. That way, the installer still doesn't need to mess with global PATH and you can easily have multiple versions of Python, each with their own "Python command prompt" submenu. At least for Windows users this would simplify the situation a bit. Stephan 2017-11-06 23:53 GMT+01:00 Ivan Pozdeev via Python-ideas < python-ideas at python.org>: > On 07.11.2017 1:48, Chris Barker wrote: > > On Mon, Nov 6, 2017 at 9:52 AM, Michel Desmoulin < > desmoulinmichel at gmail.com> wrote: > >> I know and you still: >> >> - have to use py -m on windows, python3 linux, python in virtualenv... >> > > can't you use python3 -m pip install ..... > > everywhere? > > You can't. Windows versions don't create versioned executables. Got bitten > with this myself. > > > ...Maybe they should? > (This is python-ideas, after all ;-) ) > > -- > Regards, > Ivan > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vano at mail.mipt.ru Tue Nov 7 13:07:47 2017 From: vano at mail.mipt.ru (Ivan Pozdeev) Date: Tue, 7 Nov 2017 21:07:47 +0300 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2f371148-b93b-57d3-10a8-28b5bd5011d8@mail.mipt.ru> Message-ID: <62e4ea68-e9ce-ab1e-dc33-ffdb225d9162@mail.mipt.ru> On 07.11.2017 18:34, Stephan Houben wrote: > One (smaller) suggestion on the PATH situation on Windows: > > I noticed that Visual C++ Build Tools installs a number of? "Command > prompts" > under its Start menu item, each of which starts a cmd.exe with > appropriate PATH > set to the appropriate compiler (32/64 bits or ARM cross-compiler), and > assorted environment variables set to the appropriate include/library > directories. > > Could we do something similar for Python? > > I.e., Install under the "Python 3.6" start menu an additional > "Python command prompt", which will > start cmd.exe with an appropriate PATH so that python and pip > run without further prefix. > > That way, the installer still doesn't need to mess with global PATH > and you can > easily have multiple versions of Python, each with their own > "Python command prompt" submenu. > > At least for Windows users this would simplify the situation a bit. > > Stephan > This suggestion is no different from environemnt activation scripts of anaconda/pyenv/virtualenv -- without the other benefits of virtualenv. IPython solves this by creating version-specific executables (even in Windows). I see no reason why executables of Python proper cannot do the same. In fact, the reason why Windows version went Py instead of this seems to rather be to solve a completely different problem -- the filetype association (when running a script via ShellExecute, it autodetects which Python version to invoke). > > 2017-11-06 23:53 GMT+01:00 Ivan Pozdeev via Python-ideas > >: > > On 07.11.2017 1:48, Chris Barker wrote: >> On Mon, Nov 6, 2017 at 9:52 AM, Michel Desmoulin >> > wrote: >> >> I know and you still: >> >> - have to use py -m on windows, python3 linux, python in >> virtualenv... >> >> >> can't you use python3 -m pip install ..... >> >> everywhere? > You can't. Windows versions don't create versioned executables. > Got bitten with this myself. > > > ...Maybe they should? > (This is python-ideas, after all ;-) ) > > -- > Regards, > Ivan > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > > > -- Regards, Ivan -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Nov 7 15:24:48 2017 From: guido at python.org (Guido van Rossum) Date: Tue, 7 Nov 2017 12:24:48 -0800 Subject: [Python-ideas] Proposal to change Python version release cycle In-Reply-To: References: <45619991.35926.1509794757663@office.mailbox.org> <20171104142953.0821b213@fsol> <8dd82d18-5828-2356-ee4f-14f6cc9dc14e@mailbox.org> <5bc30002-d6a4-5fad-f1e0-96e2d74a059e@mailbox.org> Message-ID: On Tue, Nov 7, 2017 at 4:52 AM, Nick Coghlan wrote: > Yes, going with "3.10", but encoding it as "3A" in lexically ambiguous > contexts is another option that would let us get as far as 3.35 (aka > "3Z") before encountering ambiguity problems. > No way. I call YAGNI on everything in this thread. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Nov 7 15:27:33 2017 From: guido at python.org (Guido van Rossum) Date: Tue, 7 Nov 2017 12:27:33 -0800 Subject: [Python-ideas] Possible enhancement to typing In-Reply-To: References: <20171106071357.GC15990@ando.pearwood.info> Message-ID: On Mon, Nov 6, 2017 at 10:36 PM, St?fane Fermigier wrote: > > > On Tue, Nov 7, 2017 at 2:58 AM, Guido van Rossum wrote: > >> On Mon, Nov 6, 2017 at 2:34 AM, St?fane Fermigier >> wrote: >> >>> 4) 10 years ago, when I was working on the EDOS project ( >>> http://cordis.europa.eu/pub/ist/docs/directorate_d/st-ds/edo >>> s-project-story_en.pdf ), I ran a small experiment where I used, IIRC, >>> the profile hook to intercept all function / method calls, and log >>> information about arguments and return value types to a gigantic log file. >>> Then the log file could be parsed and these information used to suggest >>> type annotations. Except there were no type annotations at the time in >>> Python. >>> >>> I know PyCharm can do a similar thing now: you run your program or your >>> tests under the debugger, it logs runtime type information somewhere, and >>> then can use it to suggest autocompletion or maybe type annotations. >>> >> >> I didn't know this. Do you know where there are docs for this feature? >> > > This was described in this blog post when first introduced: > > https://blog.jetbrains.com/pycharm/2013/02/dynamic- > runtime-type-inference-in-pycharm-2-7/ > > And more tersely, in the documentation: https://www. > jetbrains.com/help/pycharm/python-debugger.html > Oh. It sounds like it doesn't generate stubs or PEP 484 annotations. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Tue Nov 7 15:33:11 2017 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 7 Nov 2017 12:33:11 -0800 Subject: [Python-ideas] Any chance on (slowly) deprecating `eval` and `exec` as builtins? In-Reply-To: <20171107144116.GA19802@ando.pearwood.info> References: <20171107144116.GA19802@ando.pearwood.info> Message-ID: On Tue, Nov 7, 2017 at 6:41 AM, Steven D'Aprano wrote: > > >Also, it is unfortunate that `ast.literal_eval` is less accessible than > > >`builtins.eval`. Giving it an alias in builtins might make it easier for > > >programmers (and less scary - "ast" might sound like I need a PhD to use > > >it). > I agree -- literal_eval is a really nice utility, but folks are not likely to find it...or think it's as simple a tool as it is... In fact, a couple weeks ago, one of the students in my intro to python class used it -- good for him!), When my TA was reviewing the code (and she's a pretty experienced developer), she asked me: what is the ast module? I know it's the "Abstract Syntax Tree" module, so was very surprised that an intro student was messing about the the AST! when I looked at the code, I saw, oh that's literal_eval -- a simple (seeming, anyway) function that a beginner may just want to use (good old stack overflow...) I can see why liter_eval is needed in the AST module, but it's also a useful tool by itself,a nd that seems like a very strange place to store it... In any case, I think that securing literal_eval is much simpler than > securing eval: > > try: > # a thousand character expression ought to be enough for > # any legitimate purpose... > value = literal_eval(tainted_string[:1000]) # untested > except MemoryError: > value = None > sure -- though I'd use a lot more than 1000 characters -- not much these days, and you might want to unpack something like a JSON data package... And I'd raise an exception if it's too big, rather than trying to evaluate the subset... Maybe something like this should be patched into it? -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Tue Nov 7 15:38:45 2017 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 7 Nov 2017 12:38:45 -0800 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: <479e4e16-7c73-df23-6a35-eff9b381ef31@tjol.eu> References: <76bcd6fb-50cb-6987-37db-7bad70649ca3@mail.mipt.ru> <479e4e16-7c73-df23-6a35-eff9b381ef31@tjol.eu> Message-ID: On Tue, Nov 7, 2017 at 5:04 AM, Thomas Jollans wrote: > > As Ivan said earlier, perhaps the Windows installers should provide a > "python3" executable, so "python3 -m pip" works everywhere. > absolutely! I really, really thought it did!!!! (I'm amazed I never heard from a single student getting bit by that...) > I normally use "pip2" and "pip3", but those aren't available everywhere, > and may be hard to support properly. > there can be multiple python2 or 3s too... at least with: python? -m pip install you will get the pip that matches the python you use... with ensurepip, having pip no installed in a python is getting less common, so maybe this isn't needed anymore, but.... -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Tue Nov 7 15:59:02 2017 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 8 Nov 2017 07:59:02 +1100 Subject: [Python-ideas] Any chance on (slowly) deprecating `eval` and `exec` as builtins? In-Reply-To: References: <20171107144116.GA19802@ando.pearwood.info> Message-ID: On Wed, Nov 8, 2017 at 7:33 AM, Chris Barker wrote: > On Tue, Nov 7, 2017 at 6:41 AM, Steven D'Aprano wrote: >> In any case, I think that securing literal_eval is much simpler than >> securing eval: >> >> try: >> # a thousand character expression ought to be enough for >> # any legitimate purpose... >> value = literal_eval(tainted_string[:1000]) # untested >> except MemoryError: >> value = None > > > sure -- though I'd use a lot more than 1000 characters -- not much these > days, and you might want to unpack something like a JSON data package... That's the trouble, though. It's perfectly safe to literal_eval a large amount of well-formed data (say, a dict display with simple keys and good-sized strings as values), but you can cause major problems by literal_evalling a relatively small amount of malicious data (eg "["*100 bombs out with MemoryError, and I wouldn't trust that there isn't something far worse). If you're working with untrusted data, you probably should be using json.loads rather than ast.literal_eval. -1 on hiding eval/exec; these features exist in many languages, and they're identically dangerous everywhere. Basically, use eval only with text from the owner of the system, not from anyone untrusted. ChrisA From p.f.moore at gmail.com Tue Nov 7 16:26:38 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 7 Nov 2017 21:26:38 +0000 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <76bcd6fb-50cb-6987-37db-7bad70649ca3@mail.mipt.ru> <479e4e16-7c73-df23-6a35-eff9b381ef31@tjol.eu> Message-ID: On 7 November 2017 at 20:38, Chris Barker wrote: > On Tue, Nov 7, 2017 at 5:04 AM, Thomas Jollans wrote: >> >> As Ivan said earlier, perhaps the Windows installers should provide a >> "python3" executable, so "python3 -m pip" works everywhere. > > absolutely! I really, really thought it did!!!! (I'm amazed I never heard > from a single student getting bit by that...) On Windows, use py -X.Y to select the exact version of Python you want. Maybe Unix should have a launcher like this, too? It doesn't really need to be any more complex than exec pythonX.Y $@ relying on the existence of versioned executables on Unix. (Excuse my garbled don't-really-know-how-to-do-it shell scripting...) Paul From elazarg at gmail.com Tue Nov 7 16:39:41 2017 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Tue, 07 Nov 2017 21:39:41 +0000 Subject: [Python-ideas] Any chance on (slowly) deprecating `eval` and `exec` as builtins? In-Reply-To: References: <20171107144116.GA19802@ando.pearwood.info> Message-ID: ?????? ??? ??, 7 ????? 2017, 22:59, ??? Chris Angelico ?: > > -1 on hiding eval/exec; these features exist in many languages, and > they're identically dangerous everywhere. Basically, use eval only > with text from the owner of the system, not from anyone untrusted. > I am sorry. I don't understand the reasons you are giving here. One sentence is a fact, and I agree with the other, so I must be missing something. Elazar -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Tue Nov 7 16:50:49 2017 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 8 Nov 2017 08:50:49 +1100 Subject: [Python-ideas] Any chance on (slowly) deprecating `eval` and `exec` as builtins? In-Reply-To: References: <20171107144116.GA19802@ando.pearwood.info> Message-ID: On Wed, Nov 8, 2017 at 8:39 AM, ????? wrote: > > > ?????? ??? ??, 7 ????? 2017, 22:59, ??? Chris Angelico ?: >> >> >> -1 on hiding eval/exec; these features exist in many languages, and >> they're identically dangerous everywhere. Basically, use eval only >> with text from the owner of the system, not from anyone untrusted. > > > I am sorry. I don't understand the reasons you are giving here. One sentence > is a fact, and I agree with the other, so I must be missing something. If someone is using eval/exec with untrusted code, no amount of hiding-behind-imports is going to change that. A quick glance at the Stack Overflow search you linked to (just the search results themselves - I didn't dive deeper) shows only a few that would be affected by this change, and most of them are from people who seem to at least broadly understand what's going on. So the benefit isn't going to be huge, and a backward compatibility break is extremely annoying (even obscure functions like reduce incurred some backlash when they were "hidden" behind an import). Hence I'm -1 on changing this. Had Python always had eval off in some module, I wouldn't push for its promotion to builtin, but IMO the cost of moving it is greater than any benefit of protection. The dangers of eval/exec should be well known. ChrisA From guido at python.org Tue Nov 7 16:53:00 2017 From: guido at python.org (Guido van Rossum) Date: Tue, 7 Nov 2017 13:53:00 -0800 Subject: [Python-ideas] Any chance on (slowly) deprecating `eval` and `exec` as builtins? In-Reply-To: References: Message-ID: On Tue, Nov 7, 2017 at 2:29 AM, ????? wrote: > The dangers of eval and exec are obvious and well known to advanced users, > but the availability as built-in functions makes it too tempting for > beginners or even medium-level programmers. > I find it dubious to claim that these functions are dangerous to beginners. The dangers are related to attacks on servers that are exposed to the internet and beginners have no business running servers. Once you start exposing your code to attackers there are a lot of other things you have to worry about, and exec/eval are at least easily found using grep, unlike some other unsafe patterns. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From vano at mail.mipt.ru Tue Nov 7 16:55:58 2017 From: vano at mail.mipt.ru (Ivan Pozdeev) Date: Wed, 8 Nov 2017 00:55:58 +0300 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <76bcd6fb-50cb-6987-37db-7bad70649ca3@mail.mipt.ru> <479e4e16-7c73-df23-6a35-eff9b381ef31@tjol.eu> Message-ID: On 07.11.2017 23:38, Chris Barker wrote: > with ensurepip, having pip no installed in a python is getting less > common, so maybe this isn't needed anymore, but.... pip is problematic in environments that have their own package manager (i.e. anything but bare Windows) because it doesn't honor its conventions (e.g. if it's a non-default Python installation, all its packages must use versioned or otherwise custom names for executable modules - see e.g. https://stackoverflow.com/questions/40718770/pytest-running-with-another-version-of-python for an illustration). That's why it's not installed by default in environments where that's a significant problem. -- Regards, Ivan -------------- next part -------------- An HTML attachment was scrubbed... URL: From vano at mail.mipt.ru Tue Nov 7 17:06:00 2017 From: vano at mail.mipt.ru (Ivan Pozdeev) Date: Wed, 8 Nov 2017 01:06:00 +0300 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <76bcd6fb-50cb-6987-37db-7bad70649ca3@mail.mipt.ru> <479e4e16-7c73-df23-6a35-eff9b381ef31@tjol.eu> Message-ID: <8f93e5f2-8f6f-22d0-520c-ef254585c45d@mail.mipt.ru> On 08.11.2017 0:26, Paul Moore wrote: > On 7 November 2017 at 20:38, Chris Barker wrote: >> On Tue, Nov 7, 2017 at 5:04 AM, Thomas Jollans wrote: >>> As Ivan said earlier, perhaps the Windows installers should provide a >>> "python3" executable, so "python3 -m pip" works everywhere. >> absolutely! I really, really thought it did!!!! (I'm amazed I never heard >> from a single student getting bit by that...) > On Windows, use py -X.Y to select the exact version of Python you > want. Maybe Unix should have a launcher like this, too? It doesn't > really need to be any more complex than Native UNIX packages have been using `pythonX' and `pythonXY' for years, with incomplete names meaning "the latest installed version qualifying". This allows very convenient and reliable use in shebangs -- the script explicitly states which version it is written for (and even which installation, by specifying a full path - then even e.g. overriding `python' on PATH doesn't break system scripts). -- Regards, Ivan From storchaka at gmail.com Tue Nov 7 17:27:32 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 8 Nov 2017 00:27:32 +0200 Subject: [Python-ideas] Any chance on (slowly) deprecating `eval` and `exec` as builtins? In-Reply-To: <20171107144116.GA19802@ando.pearwood.info> References: <20171107144116.GA19802@ando.pearwood.info> Message-ID: 07.11.17 16:41, Steven D'Aprano ????: > On Tue, Nov 07, 2017 at 03:35:58PM +0200, Serhiy Storchaka wrote: >> 07.11.17 12:29, ????? ????: >>> Also, it is unfortunate that `ast.literal_eval` is less accessible than >>> `builtins.eval`. Giving it an alias in builtins might make it easier for >>> programmers (and less scary - "ast" might sound like I need a PhD to use >>> it). >> >> ast.literal_eval is not so safe as you think. Malicious input can cause >> a stack overflow in your program. [1] >> >> [1] https://bugs.python.org/issue31113 > > > I don't see anything about literal_eval in that bug report. Sorry, this particular issue isn't related to literal_eval. There was other recently fixed issue, but I forgot its number. But the point is that the compiler is recursive, and processing nested constructs consumes the C stack. There are some guards against too deep recursion (2.7 has less guards and more vulnerable), but it is hard to prove that all vulnerabilities are fixed. Your method (limiting the size of the input) helps against some attacks. Other methods -- restricting the set of characters and the number of parenthesis, braces and brackets. From guido at python.org Tue Nov 7 17:56:29 2017 From: guido at python.org (Guido van Rossum) Date: Tue, 7 Nov 2017 14:56:29 -0800 Subject: [Python-ideas] Moving typing out of the stdlib in Python 3.7? In-Reply-To: References: Message-ID: On Mon, Nov 6, 2017 at 5:27 PM, Barry Warsaw wrote: > > Guido van Rossum wrote: > > > This is beginning to sound like the most attractive solution. We could > > possibly do away with typing_extensions. Are there precedents of how to > > bundle a module in this way? Or is it going to be another special case > like > > pip? > > With my downstream consumer hat on, *if* we go with a bundling solution, > please make sure it's easy to unbundle. Downstream consumers may need > to do this for policy, security, etc. reasons, so making this too > difficult will mean delays in releases, convoluted hacks, policy > violations, or all of the above. > > See as reference some of the hoops Debian has to go through to handle > ensurepip. > Having skimmed this thread and some others, I am going in the opposite direction. Moving typing.py out of the stdlib and then bundling it will cause too many problems, both psychological and technical. Therefore I am withdrawing my idea of moving typing.py out of the stdlib, and replacing it with an extension of its (and PEP 484's) provisional status by at least one release cycle. This means that at least for the duration of 3.7 we'll be able to update typing.py pretty aggressively in bugfix releases. See https://github.com/python/peps/pull/451 -- basically we only need a restricted interpretation of provisional, where we can add new features, but won't break backwards compatibility *for documented aspects of the API*. With PEP 560, we can then attempt to speed up the import of typing.py (hopefully 10x by the time 3.8 rolls around). I propose that the schedule for typing.py's provisional status be added to PEP 563 (the __future__ import for annotations); I've got a PR with tentative update to PEP 484 ready (https://github.com/python/peps/pull/451). -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From fakedme+py at gmail.com Tue Nov 7 18:12:27 2017 From: fakedme+py at gmail.com (Soni L.) Date: Tue, 7 Nov 2017 21:12:27 -0200 Subject: [Python-ideas] Any chance on (slowly) deprecating `eval` and `exec` as builtins? In-Reply-To: References: Message-ID: <9025d232-2289-5193-a0d5-ac8946f7b3a6@gmail.com> On 2017-11-07 08:29 AM, ????? wrote: > Hi, > > The dangers of eval and exec are obvious and well known to advanced > users, but the availability as built-in functions makes it too > tempting for beginners or even medium-level programmers. You can see > questions about these function pretty often in stackoverflow (roughly > once a day > , though > sometimes the uses are legitimate). > > Maybe we could start a ten-year process of deprecating the use of > `builtins.eval` (in the docs, and then with warnings)? `builtins.eval` > will be a wrapper to the real evaluation function, moved to > `unsafe.eval` or something obvious like that, so all you need to do to > port your code is to add `from unsafe import unsafe_eval as eval, > unsafe_exec as exec` at the top of the file; it will be a nice warning > to the reader. > > The fact that it is a wrapper will slightly slow it down and make the > stack traces noisier - both are good things, IMO. > > Also, it is unfortunate that `ast.literal_eval` is less accessible > than `builtins.eval`. Giving it an alias in builtins might make it > easier for programmers (and less scary - "ast" might sound like I need > a PhD to use it). > > What do you think? Please don't! exec("def one(x):\n [r] = x\n return r")? # who says python doesn't have one-liners? (ofc, some would argue you should use: one = (lambda x: (lambda y: y)(*x)) but I digress) > > Elazar > > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Tue Nov 7 18:58:44 2017 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Tue, 7 Nov 2017 15:58:44 -0800 Subject: [Python-ideas] Any chance on (slowly) deprecating `eval` and `exec` as builtins? In-Reply-To: References: <20171107144116.GA19802@ando.pearwood.info> Message-ID: <-4084989005840651240@unknownmsgid> But the point is that the compiler is recursive, and processing nested constructs consumes the C stack. There are some guards against too deep recursion (2.7 has less guards and more vulnerable), but it is hard to prove that all vulnerabilities are fixed. Your method (limiting the size of the input) helps against some attacks. Other methods -- restricting the set of characters and the number of parenthesis, braces and brackets. Hmm ? I?d never really thought about it, bust presumably ast.literal_eval was designed for use in the compiler? or at least uses the compiler to do its real work. So maybe what we really need is a literal-eval that is DESIGNED to be a safe Python literal parser. Like a JSON parser but supporting the richer Python literal set. -CHB -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Tue Nov 7 19:32:27 2017 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 8 Nov 2017 11:32:27 +1100 Subject: [Python-ideas] Any chance on (slowly) deprecating `eval` and `exec` as builtins? In-Reply-To: <-4084989005840651240@unknownmsgid> References: <20171107144116.GA19802@ando.pearwood.info> <-4084989005840651240@unknownmsgid> Message-ID: On Wed, Nov 8, 2017 at 10:58 AM, Chris Barker - NOAA Federal wrote: > > > But the point is that the compiler is recursive, and processing nested > constructs consumes the C stack. There are some guards against too deep > recursion (2.7 has less guards and more vulnerable), but it is hard to prove > that all vulnerabilities are fixed. > > Your method (limiting the size of the input) helps against some attacks. > Other methods -- restricting the set of characters and the number of > parenthesis, braces and brackets. > > > Hmm ? I?d never really thought about it, bust presumably ast.literal_eval > was designed for use in the compiler? or at least uses the compiler to do > its real work. > > So maybe what we really need is a literal-eval that is DESIGNED to be a safe > Python literal parser. > > Like a JSON parser but supporting the richer Python literal set. I believe there are pure-Python implementations of literal_eval around, which would be a good basis for hacking on. ChrisA From schesis at gmail.com Tue Nov 7 20:32:58 2017 From: schesis at gmail.com (Zero Piraeus) Date: Tue, 7 Nov 2017 22:32:58 -0300 Subject: [Python-ideas] Fwd: Any chance on (slowly) deprecating `eval` and `exec` as builtins? In-Reply-To: References: Message-ID: : On 7 November 2017 at 18:53, Guido van Rossum wrote: > I find it dubious to claim that these functions are dangerous to beginners. > The dangers are related to attacks on servers that are exposed to the > internet and beginners have no business running servers. I can't be the only one whose experience is that it's very often those with the least business running servers that actually do run them, surely? +1 -[]z. From steve at pearwood.info Tue Nov 7 21:26:22 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 8 Nov 2017 13:26:22 +1100 Subject: [Python-ideas] Any chance on (slowly) deprecating `eval` and `exec` as builtins? In-Reply-To: References: Message-ID: <20171108022622.GC19802@ando.pearwood.info> On Tue, Nov 07, 2017 at 01:53:00PM -0800, Guido van Rossum wrote: > On Tue, Nov 7, 2017 at 2:29 AM, ????? wrote: > > > The dangers of eval and exec are obvious and well known to advanced users, > > but the availability as built-in functions makes it too tempting for > > beginners or even medium-level programmers. > > > > I find it dubious to claim that these functions are dangerous to beginners. I don't think its so much that eval/exec are in themselves dangerous to beginners as that their easy availability as builtins encourages bad habits that can last long after the programmer is no longer a beginner. I know the Python ecosystem is not quite the wild west as PHP and Javascript sometimes is, but code injection attacks do exist: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-9807 https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-9802 Sometimes they're written by beginners whose code isn't being reviewed carefully enough, and sometimes they're written by experienced coders who have simply learned bad habits and haven't learned better. I don't want to scare people away from using eval/exec, but it would be great if we could gently encourage them to think before using them, and to prefer literal_eval instead. -- Steve From guido at python.org Tue Nov 7 21:29:31 2017 From: guido at python.org (Guido van Rossum) Date: Tue, 7 Nov 2017 18:29:31 -0800 Subject: [Python-ideas] Any chance on (slowly) deprecating `eval` and `exec` as builtins? In-Reply-To: <20171108022622.GC19802@ando.pearwood.info> References: <20171108022622.GC19802@ando.pearwood.info> Message-ID: On Tue, Nov 7, 2017 at 6:26 PM, Steven D'Aprano wrote: > On Tue, Nov 07, 2017 at 01:53:00PM -0800, Guido van Rossum wrote: > > On Tue, Nov 7, 2017 at 2:29 AM, ????? wrote: > > > > > The dangers of eval and exec are obvious and well known to advanced > users, > > > but the availability as built-in functions makes it too tempting for > > > beginners or even medium-level programmers. > > > > > > > I find it dubious to claim that these functions are dangerous to > beginners. > > I don't think its so much that eval/exec are in themselves dangerous > to beginners as that their easy availability as builtins encourages bad > habits that can last long after the programmer is no longer a beginner. > > I know the Python ecosystem is not quite the wild west as PHP and > Javascript sometimes is, but code injection attacks do exist: > > https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-9807 > > https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-9802 > > Sometimes they're written by beginners whose code isn't being reviewed > carefully enough, and sometimes they're written by experienced coders > who have simply learned bad habits and haven't learned better. > > I don't want to scare people away from using eval/exec, but it would be > great if we could gently encourage them to think before using them, and > to prefer literal_eval instead. > Sure, I'm all for making sure the documentation is clear. But the proposal at hand is to remove them from the builtins, and I don't see the situation as grave as needing that. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From sf at fermigier.com Wed Nov 8 03:09:37 2017 From: sf at fermigier.com (=?UTF-8?Q?St=C3=A9fane_Fermigier?=) Date: Wed, 8 Nov 2017 09:09:37 +0100 Subject: [Python-ideas] Possible enhancement to typing In-Reply-To: References: <20171106071357.GC15990@ando.pearwood.info> Message-ID: On Tue, Nov 7, 2017 at 9:27 PM, Guido van Rossum wrote: > On Mon, Nov 6, 2017 at 10:36 PM, St?fane Fermigier > wrote: > >> >> >> On Tue, Nov 7, 2017 at 2:58 AM, Guido van Rossum >> wrote: >> >>> On Mon, Nov 6, 2017 at 2:34 AM, St?fane Fermigier >>> wrote: >>> >>>> 4) 10 years ago, when I was working on the EDOS project ( >>>> http://cordis.europa.eu/pub/ist/docs/directorate_d/st-ds/edo >>>> s-project-story_en.pdf ), I ran a small experiment where I used, IIRC, >>>> the profile hook to intercept all function / method calls, and log >>>> information about arguments and return value types to a gigantic log file. >>>> Then the log file could be parsed and these information used to suggest >>>> type annotations. Except there were no type annotations at the time in >>>> Python. >>>> >>>> I know PyCharm can do a similar thing now: you run your program or your >>>> tests under the debugger, it logs runtime type information somewhere, and >>>> then can use it to suggest autocompletion or maybe type annotations. >>>> >>> >>> I didn't know this. Do you know where there are docs for this feature? >>> >> >> This was described in this blog post when first introduced: >> >> https://blog.jetbrains.com/pycharm/2013/02/dynamic-runtime- >> type-inference-in-pycharm-2-7/ >> >> And more tersely, in the documentation: https://www.jet >> brains.com/help/pycharm/python-debugger.html >> > > Oh. It sounds like it doesn't generate stubs or PEP 484 annotations. > The blog post was written in 2013, PEP 484 was not yet adopted at the time, and so they targeted their own typechecker and typing annotation conventions, which were at the time based on docstrings conventions. I have just checked with the current PyCharm version, and when using the "add annotation" intention, they're using now either the "# type: (...) -> ...", or the PEP484 conventions, depending on the version of Python you're using for your particular workspace. Also, I found the the call signature collection mechanism during debug run seems to live in the PyDev.Debugger project ( https://github.com/fabioz/PyDev.Debugger/ which is common to PyDev and PyCharm), more specifically here: https://github.com/fabioz/PyDev.Debugger/blob/master/_pydevd_bundle/pydevd_signature.py S. -- Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier - http://linkedin.com/in/sfermigier Founder & CEO, Abilian - Enterprise Social Software - http://www.abilian.com/ Chairman, Free&OSS Group / Systematic Cluster - http://www.gt-logiciel-libre.org/ Co-Chairman, National Council for Free & Open Source Software (CNLL) - http://cnll.fr/ Founder & Organiser, PyData Paris - http://pydata.fr/ --- ?You never change things by ?ghting the existing reality. To change something, build a new model that makes the existing model obsolete.? ? R. Buckminster Fuller -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Wed Nov 8 13:08:20 2017 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 08 Nov 2017 10:08:20 -0800 Subject: [Python-ideas] Any chance on (slowly) deprecating `eval` and `exec` as builtins? In-Reply-To: <9025d232-2289-5193-a0d5-ac8946f7b3a6@gmail.com> References: <9025d232-2289-5193-a0d5-ac8946f7b3a6@gmail.com> Message-ID: <5A034814.7000008@stoneleaf.us> On 11/07/2017 03:12 PM, Soni L. wrote: > exec("def one(x):\n [r] = x\n return r") # who says python doesn't have one-liners? > > (ofc, some would argue you should use: > > one = (lambda x: (lambda y: y)(*x)) Most would argue that def one(x): [r] = x return r is the appropriate code. -- ~Ethan~ From breamoreboy at gmail.com Wed Nov 8 16:35:23 2017 From: breamoreboy at gmail.com (Mark Lawrence) Date: Wed, 8 Nov 2017 21:35:23 +0000 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: <2f371148-b93b-57d3-10a8-28b5bd5011d8@mail.mipt.ru> References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2f371148-b93b-57d3-10a8-28b5bd5011d8@mail.mipt.ru> Message-ID: On 06/11/17 22:53, Ivan Pozdeev via Python-ideas wrote: > You can't. Windows versions don't create versioned executables. Got > bitten with this myself. You either use py -x.y -mpip or you can directly use pip.exe, pepx.exe or pipx.y.exe. > > ...Maybe they should? > (This is python-ideas, after all ;-) ) > > -- > Regards, > Ivan > -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From desmoulinmichel at gmail.com Thu Nov 9 15:39:49 2017 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Thu, 9 Nov 2017 21:39:49 +0100 Subject: [Python-ideas] Any chance on (slowly) deprecating `eval` and `exec` as builtins? In-Reply-To: References: <20171107144116.GA19802@ando.pearwood.info> Message-ID: <34750410-751d-9676-0141-a4f71d1c8653@gmail.com> Le 07/11/2017 ? 22:39, ????? a ?crit?: > > > ?????? ??? ??, 7 ????? 2017, 22:59, ??? Chris Angelico > ?>: > > > -1 on hiding eval/exec; these features exist in many languages, and > they're identically dangerous everywhere. Basically, use eval only > with text from the owner of the system, not from anyone untrusted. > > > I am sorry. I don't understand the reasons you are giving here.?One > sentence is a fact, and I agree with the other, so I must be missing > something. > > Elazar? > > Also: why are eval and exec even builtins ? Builtins are for stuff you use often. I never saw in my entire career anybody use it often. From desmoulinmichel at gmail.com Fri Nov 10 01:48:35 2017 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Fri, 10 Nov 2017 07:48:35 +0100 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> Message-ID: <901dc0af-8df8-9421-f71e-afbe4aaafc9d@gmail.com> Le 06/11/2017 ? 23:48, Chris Barker a ?crit?: > On Mon, Nov 6, 2017 at 9:52 AM, Michel Desmoulin > > wrote: > > I know and you still: > > - have to use py -m on windows, python3 linux, python in virtualenv... > > > can't you use python3 -m pip install ..... > > everywhere? No: on windows you don't have python3, you have py -x.x. On linux you can't pip install, you need --users, admin rights or a virtualenv. > > That's what I tell my beginner students to do, and I've never had a > problem. (nce they got Python installed right in the first place) For that: > > The python.og installers for Windows and Mac pretty much? > "just work" Nope. E.G: in windows, if you don't check "add python executable to system path" people won't be able to type "python" in the shell. It's unchecked by default. > > Linux is a different story, but Linux users are more comfortable with eh > whole idea of command lines and packages, etc -- so my Linux users have > never been the hangup. Not anymore. There is a wave of beginners arriving in an industry using linux but being unfamiliar with the system. The problem is not that it's hard. The problem is that it's work to discover it. The first Python experience should be simple and straight forward. Another problem is that it's not portable knowledge from an OS to another. > > If/when I'm teaching data analysis for scientific computing, I go > straight to conda, but for basic python, and most web Development, > pyton.org python and pip work great. You can't teach conda first. It's not portable knowledge and would not let students benefit from the mass of online tutorials that use pip and virtualenv. > > A also DO NOT introduce virtualenv right off the bat -- it is another > complication that is critical to real development, but not important for > learning python. > > I have done it it the past -- it did not go well.... > > What's too bad now is that so many docs say "pip install the_package", > and users with multiple python installs can get burnt. (though most don't) > My point exactly. > -CHB > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R ? ? ? ? ? ?(206) 526-6959?? voice > 7600 Sand Point Way NE ??(206) 526-6329?? fax > Seattle, WA ?98115 ? ? ??(206) 526-6317?? main reception > > Chris.Barker at noaa.gov From desmoulinmichel at gmail.com Fri Nov 10 01:50:00 2017 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Fri, 10 Nov 2017 07:50:00 +0100 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2f371148-b93b-57d3-10a8-28b5bd5011d8@mail.mipt.ru> Message-ID: > > I.e., Install under the "Python 3.6" start menu an additional > "Python command prompt", which will > start cmd.exe with an appropriate PATH so that python and pip > run without further prefix. > > That way, the installer still doesn't need to mess with global PATH and > you can > easily have multiple versions of Python, each with their own > "Python command prompt" submenu. > > At least for Windows users this would simplify the situation a bit. They already do that on windows. But adding "yet another way" to do things differently on different OS doesn't feel like a solution ot me. From desmoulinmichel at gmail.com Fri Nov 10 01:54:06 2017 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Fri, 10 Nov 2017 07:54:06 +0100 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: <479e4e16-7c73-df23-6a35-eff9b381ef31@tjol.eu> References: <76bcd6fb-50cb-6987-37db-7bad70649ca3@mail.mipt.ru> <479e4e16-7c73-df23-6a35-eff9b381ef31@tjol.eu> Message-ID: <6396a8c5-398e-cc3a-f0f7-8d33419671f0@gmail.com> > Isn't pip installed by default on Windows and OSX? On Linux some > distribution will probably do something unexpected and mess up your > grand plan. Some installers download pip, which won't work if you don't have an internet connection or a proxy setup. But my point is, all this should be unified. All the points I mentioned worked in a A and B but not C, or a combination of those, different for each point. This is very unfortunate to write docs, tutorials, or any teaching material. From desmoulinmichel at gmail.com Fri Nov 10 01:55:55 2017 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Fri, 10 Nov 2017 07:55:55 +0100 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <76bcd6fb-50cb-6987-37db-7bad70649ca3@mail.mipt.ru> <479e4e16-7c73-df23-6a35-eff9b381ef31@tjol.eu> Message-ID: <5a35c08e-9a5b-2830-1840-24f5eea76817@gmail.com> > > there can be multiple python2 or 3s too... > > at least with: > > python? -m pip install > > you will get the pip that matches the python you use... Not on windows. You have the py command. My all point is that each of the specific issues I mentioned are A and B but not C propositions. It's not unified. > > with ensurepip, having pip no installed in a python is getting less > common, so maybe this isn't needed anymore, but.... ensurepip may depend of youself having an internet connection when you install it. And without a proxy. And it's not used on debian flavours. > > -CHB > ? > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R ? ? ? ? ? ?(206) 526-6959 ?? voice > 7600 Sand Point Way NE ??(206) 526-6329 ?? fax > Seattle, WA ?98115 ? ? ??(206) 526-6317 ?? main > reception > > Chris.Barker at noaa.gov > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From desmoulinmichel at gmail.com Fri Nov 10 01:56:40 2017 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Fri, 10 Nov 2017 07:56:40 +0100 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <76bcd6fb-50cb-6987-37db-7bad70649ca3@mail.mipt.ru> <479e4e16-7c73-df23-6a35-eff9b381ef31@tjol.eu> Message-ID: <7af4865a-478a-8da4-eb4a-92f6caac98c5@gmail.com> Le 07/11/2017 ? 22:26, Paul Moore a ?crit?: > On 7 November 2017 at 20:38, Chris Barker wrote: >> On Tue, Nov 7, 2017 at 5:04 AM, Thomas Jollans wrote: >>> >>> As Ivan said earlier, perhaps the Windows installers should provide a >>> "python3" executable, so "python3 -m pip" works everywhere. >> >> absolutely! I really, really thought it did!!!! (I'm amazed I never heard >> from a single student getting bit by that...) > > On Windows, use py -X.Y to select the exact version of Python you > want. Maybe Unix should have a launcher like this, too? It doesn't > really need to be any more complex than > > > exec pythonX.Y $@ That's my whole point. We should find a way to unify each of those issues. From desmoulinmichel at gmail.com Fri Nov 10 02:05:05 2017 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Fri, 10 Nov 2017 08:05:05 +0100 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> Message-ID: <2091af22-1b71-b123-9355-560e205972db@gmail.com> > Which is why we advise getting into a virtual environment ASAP, such > that the only platform specific thing folks necessarily need to learn > to get started is how to get to that first working virtual > environment. > You can't start by teaching virtualenv. I tried. It doesn't work. And it's a terrible prerequisit if you write docs, tutorial, teaching materials, etc. >> - install pip manually on linux > > s/Linux/Ubuntu/ > > Other distros (like Fedora) provide pip by default. s/Ubuntu/Debian derivates Debian, Ubuntu, Linux Mint, Linux Tail, etc. > >> - make sure the system path is correctly set > > Recent python.org Windows installers do this automatically, but there > are unfortunately still lots of ways for things to go wrong. Just gave a training for python 3.6 this week. It doesn't. You need to explicitly check a box to do so. Box that no beginer would ever check without being told to. Actually a box that some of my students even don't check despite being emphatically told to. > >> Stuff that they will forget on the next install, or miss when changing >> plateform > > Yes, because computers are awful, and incredibly user hostile. We > don't have a magic wand to wave to fix that. No, but we can make some adjustment to make lifes easier. For this particular issue, checking the box by default would help beginners. Sysadmin not wanting it would understand the problem and be able to take the proper decision. > >> And assume that stuff in any tutorial you make they know this stuff. >> >> This is a strong barrier or entry IMO. > > Sure, but it's not one we can readily fix - the user hostility of > command line environments and the compromises we have to make to abide > by platform conventions are in the hands of operating system vendors, > and there's only so much we can do to paper over those distinctions > when user lock-in and putting barriers in the way of cross-device > portability is a core part of commercial OS vendors' business models. That is avery pessimistic that would prevent use from making any progress in any thing. All the other participants offered at least 3 solutions to problems. They are workable. They may help. We should try them. > > This is a big part of why mobile client devices with cloud backends > are so popular, even for development purposes: they allow for a much > simpler developer experience that avoids decades of accumulated cruft > in the desktop operating system command line experience. Even there > though, you're faced with the fact that once you choose a provider, > whatever you do there will probably be locked into that provider and > not transferable elsewhere. I agree. From desmoulinmichel at gmail.com Fri Nov 10 02:07:53 2017 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Fri, 10 Nov 2017 08:07:53 +0100 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> Message-ID: Le 07/11/2017 ? 14:08, Paul Moore a ?crit?: > On 7 November 2017 at 12:44, Nick Coghlan wrote: >>> - make sure the system path is correctly set >> >> Recent python.org Windows installers do this automatically, but there >> are unfortunately still lots of ways for things to go wrong. > > I believe the latest installers switch it off again, because one of > the ways things can go wrong is that stuff put at the start of the > user path is still lower priority than stuff in the system path, and > we now default to user installs. This is an actual problem with mixed > python.org and Anaconda installations, for example - Anaconda adds > itself to the system PATH, and overrides a default user install of > python.org Python. So you can't prioritise python.org over Anaconda > without manual path hacking. (This hit me when I installed Visual > Studio and selected "include Python (Anaconda)" - I can't recall the > exact option, but it broke my python.org install). So what you are saying is that even if you do the right things, something the user may do could possibly reverse it. I don't see that as a reason not do to the right thing. From desmoulinmichel at gmail.com Fri Nov 10 02:09:23 2017 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Fri, 10 Nov 2017 08:09:23 +0100 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> Message-ID: <59ffc778-1d50-1f6c-ecea-aab905feef98@gmail.com> Le 07/11/2017 ? 14:06, ????? a ?crit?: > On Tue, Nov 7, 2017 at 2:45 PM Nick Coghlan > wrote: > > On 7 November 2017 at 03:52, Michel Desmoulin > > wrote: > > > And assume that stuff in any tutorial you make they know this stuff. > > > > This is a strong barrier or entry IMO. > > Sure, but it's not one we can readily fix - the user hostility of > command line environments and the compromises we have to make to abide > by platform conventions are in the hands of operating system vendors, > and there's only so much we can do to paper over those distinctions > when user lock-in and putting barriers in the way of cross-device > portability is a core part of commercial OS vendors' business models. > > > I don't know if you are referring to Microsoft Windows here, but I want > to note that from my personal experience the Windows subsystem for Linux > ("Bash on Ubuntu on Windows") is easy to work with, so making Windows > feel (CLI-wise) like Ubuntu is not so difficult. I'm not sure how easy > it is for students to set up, but it is an option at least. > It doesn't solve anything: - requires windows 10 - requires having the several Go install first - mix windows and linux, making the whole thing confusing to beginers - we still have the heterogenous setup between the 2 os being a problem. I love WSL, but it's not a solution to that particular problem. From ncoghlan at gmail.com Fri Nov 10 02:38:58 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 10 Nov 2017 17:38:58 +1000 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: <5a35c08e-9a5b-2830-1840-24f5eea76817@gmail.com> References: <76bcd6fb-50cb-6987-37db-7bad70649ca3@mail.mipt.ru> <479e4e16-7c73-df23-6a35-eff9b381ef31@tjol.eu> <5a35c08e-9a5b-2830-1840-24f5eea76817@gmail.com> Message-ID: On 10 November 2017 at 16:55, Michel Desmoulin wrote: > ensurepip may depend of youself having an internet connection when you > install it. And without a proxy. And it's not used on debian flavours. No, ensurepip doesn't depend on internet access (by design - it's the main reason why CPython bundles the wheel files). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Fri Nov 10 03:01:04 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 10 Nov 2017 18:01:04 +1000 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: <2091af22-1b71-b123-9355-560e205972db@gmail.com> References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> Message-ID: On 10 November 2017 at 17:05, Michel Desmoulin wrote: > >> Which is why we advise getting into a virtual environment ASAP, such >> that the only platform specific thing folks necessarily need to learn >> to get started is how to get to that first working virtual >> environment. >> > > You can't start by teaching virtualenv. I tried. It doesn't work. And > it's a terrible prerequisit if you write docs, tutorial, teaching > materials, etc. You can't have it both ways - the only way we can systematically mask the environmental differences between Windows, Linux and Mac OS X is by providing tooling that actually masks those differences, which means introducing that tooling becomes a prerequisite for teaching. It doesn't completely solve the problem (as getting into and out of the environment is still platform specific), but it does mean that the ubiquitous online instructions to run "pip install package-name" and "python -m command" will actually work once people are inside their working environment. That tooling is venv: * it ensures you have "pip" on your PATH * it ensures you have "python" on your PATH * it ensures that you have the required permissions to install new packages * it ensures that any commands you install from PyPI will be also on your PATH When we choose not to use venv, then it becomes necessary to ensure each of those things individually for each potential system starting state That said, we'd *like* the default to be is per-user package installations into the user home directory, but that creates additional problems: * "python" may be missing, and you'll have to use "python3" or "py" instead * "pip" may be missing (or mean "install for Python 2.7") * you have to specify "--user" on *every* call to pip, and most online guides won't say that * there's a major backwards compatibility problem with switching pip over to per-user package installs as the default (we still want to do it eventually, though) * on Windows, system-wide Python installs can't adjust per-user PATH settings, and per-user installs are subject to being broken by system-wide installs * on Windows, the distinction between a per-user install of Python, and per-user installs of a package is hard to teach * on Debian, I believe ~/.local/bin still isn't on PATH by default That said, I think there is one improvement we could feasibly make, which would be to introduce the notion of a "default user environment" into `venv`, such that there was a single "python -m venv shell" command that: * created a default user environment if it didn't already exist * launched a subshell with that environment already activated This wouldn't be a full environment manager like vex or pew - it would just be a way to bootstrap a single usable package management environment in a cross-platform way. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Fri Nov 10 04:50:22 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 10 Nov 2017 09:50:22 +0000 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> Message-ID: On 10 November 2017 at 08:01, Nick Coghlan wrote: > You can't have it both ways - the only way we can systematically mask > the environmental differences between Windows, Linux and Mac OS X is > by providing tooling that actually masks those differences, which > means introducing that tooling becomes a prerequisite for teaching. On Windows, which is the only platform I can reasonably comment on, the killer issue is that the installer doesn't make the commands "python" and "pip" available by default. Just checking my PC, both go and rust (which I installed *ages* ago) do appear to, so they "just work". Java sort-of also works like that (the runtime is on PATH, but the compilers need to be added manually). The biggest reason we don't add Python to PATH, as I understand it, is because we need to consider the implications of people having multiple versions of Python installed. As far as I know, no other language allows that. But equally, most beginners wouldn't actually *have* multiple versions installed. So maybe we should optimise for that case (only one version of Python installed). That would mean: 1. Go back to adding Python to PATH. Because our installers don't say "do you want to uninstall the old version", we should probably do a check for a "python" command on PATH in the installer, and if there is one, warn the user "You already have Python installed - if you are upgrading you should manually uninstall the old version first, otherwise your old installation will remain the default". We could get more complex at this point, but that depends on what capabilities we can include in the installer - I don't know how powerful the toolset is. 2. Explicitly document that multiple Python interpreters on one machine is considered "advanced", and users with that sort of setup should be prepared to manage PATH themselves. I'd put that as something like "It is possible to install multiple versions of Python at once, but if you do that, you should understand the implications - the average user should not need to do this". We still have to deal with the fact that basically every Unix environment is "advanced" in the above sense (the python2/python3 split). I don't have a solution for that (other than "upgrade to Windows" ;-)). > It doesn't completely solve the problem (as getting into and out of > the environment is still platform specific), but it does mean that the > ubiquitous online instructions to run "pip install package-name" and > "python -m command" will actually work once people are inside their > working environment. > > That tooling is venv: > > * it ensures you have "pip" on your PATH > * it ensures you have "python" on your PATH > * it ensures that you have the required permissions to install new packages > * it ensures that any commands you install from PyPI will be also on your PATH > > When we choose not to use venv, then it becomes necessary to ensure > each of those things individually for each potential system starting > state Currently, the reality is that people use virtualenv, not venv. All higher-level tools I'm aware of wrap virtualenv (to allow Python 2.7 support). Enhancing the capabilities of venv is fine, but promoting venv over virtualenv involves technical challenges across the whole toolset, not just documentation/education. But agreed, once we get people into a virtual environment (of any form) the portability issues become significantly reduced. The main outstanding issue is managing multiple environments, which could be handled by having a special "default" environment that is the only one we'd expect beginners to use/need. > That said, we'd *like* the default to be is per-user package > installations into the user home directory, but that creates > additional problems: > > * "python" may be missing, and you'll have to use "python3" or "py" instead > * "pip" may be missing (or mean "install for Python 2.7") > * you have to specify "--user" on *every* call to pip, and most online > guides won't say that > * there's a major backwards compatibility problem with switching pip > over to per-user package installs as the default (we still want to do > it eventually, though) > * on Windows, system-wide Python installs can't adjust per-user PATH > settings, and per-user installs are subject to being broken by > system-wide installs > * on Windows, the distinction between a per-user install of Python, > and per-user installs of a package is hard to teach > * on Debian, I believe ~/.local/bin still isn't on PATH by default Also on Windows, the per-user bin directory isn't added to PATH even if you add the system Python to PATH in the installer. > That said, I think there is one improvement we could feasibly make, > which would be to introduce the notion of a "default user environment" > into `venv`, such that there was a single "python -m venv shell" > command that: > > * created a default user environment if it didn't already exist > * launched a subshell with that environment already activated > > This wouldn't be a full environment manager like vex or pew - it would > just be a way to bootstrap a single usable package management > environment in a cross-platform way. "How do I run venv?" is no easier to answer in a cross-platform way than "how do I run pip?" You still need to be able to run python/python3/py. About the only improvement is that there's no legacy of documentation and advice on the web saying "run venv" like there is for "run pip"... Paul From ncoghlan at gmail.com Fri Nov 10 05:01:33 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 10 Nov 2017 20:01:33 +1000 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> Message-ID: On 10 November 2017 at 19:50, Paul Moore wrote: > On 10 November 2017 at 08:01, Nick Coghlan wrote: >> That tooling is venv: >> >> * it ensures you have "pip" on your PATH >> * it ensures you have "python" on your PATH >> * it ensures that you have the required permissions to install new packages >> * it ensures that any commands you install from PyPI will be also on your PATH >> >> When we choose not to use venv, then it becomes necessary to ensure >> each of those things individually for each potential system starting >> state > > Currently, the reality is that people use virtualenv, not venv. All > higher-level tools I'm aware of wrap virtualenv (to allow Python 2.7 > support). Enhancing the capabilities of venv is fine, but promoting > venv over virtualenv involves technical challenges across the whole > toolset, not just documentation/education. We already assume there will be a step in understanding from "working with the latest Python 3.x locally" to "dealing with multiple Python versions". Switching from venv to virtualenv just becomes part of that process (and will often be hidden behind a higher level tool like pipenv, pew, or vex anyway). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Fri Nov 10 05:07:23 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 10 Nov 2017 10:07:23 +0000 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> Message-ID: On 10 November 2017 at 10:01, Nick Coghlan wrote: > On 10 November 2017 at 19:50, Paul Moore wrote: >> On 10 November 2017 at 08:01, Nick Coghlan wrote: >>> That tooling is venv: >>> >>> * it ensures you have "pip" on your PATH >>> * it ensures you have "python" on your PATH >>> * it ensures that you have the required permissions to install new packages >>> * it ensures that any commands you install from PyPI will be also on your PATH >>> >>> When we choose not to use venv, then it becomes necessary to ensure >>> each of those things individually for each potential system starting >>> state >> >> Currently, the reality is that people use virtualenv, not venv. All >> higher-level tools I'm aware of wrap virtualenv (to allow Python 2.7 >> support). Enhancing the capabilities of venv is fine, but promoting >> venv over virtualenv involves technical challenges across the whole >> toolset, not just documentation/education. > > We already assume there will be a step in understanding from "working > with the latest Python 3.x locally" to "dealing with multiple Python > versions". Switching from venv to virtualenv just becomes part of that > process (and will often be hidden behind a higher level tool like > pipenv, pew, or vex anyway). OK, that's fair. Paul From phd at phdru.name Fri Nov 10 06:37:47 2017 From: phd at phdru.name (Oleg Broytman) Date: Fri, 10 Nov 2017 12:37:47 +0100 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: <901dc0af-8df8-9421-f71e-afbe4aaafc9d@gmail.com> References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <901dc0af-8df8-9421-f71e-afbe4aaafc9d@gmail.com> Message-ID: <20171110113747.GA1740@phdru.name> On Fri, Nov 10, 2017 at 07:48:35AM +0100, Michel Desmoulin wrote: > On linux you > can't pip install, you need --users, admin rights or a virtualenv. Isn't it the same on Windows? For an admin-installed Python you need --users, admin rights or a virtualenv. And a user-installed Python on Windows is equivalen to a user-compiled Python on Linux -- pip installs packages to the user-owned site-packages directory. Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From phd at phdru.name Fri Nov 10 06:41:59 2017 From: phd at phdru.name (Oleg Broytman) Date: Fri, 10 Nov 2017 12:41:59 +0100 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> Message-ID: <20171110114159.GB1740@phdru.name> On Fri, Nov 10, 2017 at 09:50:22AM +0000, Paul Moore wrote: > The biggest reason we don't add Python to PATH, as I understand it, is > because we need to consider the implications of people having multiple > versions of Python installed. Why not fix that the same way as on Unix -- by having versioned executables: python27.exe, python35.exe? Then python.exe in PATH will be from the most recent installed Python. > Paul Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From p.f.moore at gmail.com Fri Nov 10 06:51:35 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 10 Nov 2017 11:51:35 +0000 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: <20171110113747.GA1740@phdru.name> References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <901dc0af-8df8-9421-f71e-afbe4aaafc9d@gmail.com> <20171110113747.GA1740@phdru.name> Message-ID: On 10 November 2017 at 11:37, Oleg Broytman wrote: > On Fri, Nov 10, 2017 at 07:48:35AM +0100, Michel Desmoulin wrote: >> On linux you >> can't pip install, you need --users, admin rights or a virtualenv. > > Isn't it the same on Windows? For an admin-installed Python you need > --users, admin rights or a virtualenv. And a user-installed Python on > Windows is equivalen to a user-compiled Python on Linux -- pip installs > packages to the user-owned site-packages directory. It is - but the default install on Windows (using the python.org installer) is a per-user install. So beginners don't encounter admin-installed Python (unless they ask for it, in which case they made the choice so they should understand the implications ;-)) Paul From guido at python.org Fri Nov 10 11:43:44 2017 From: guido at python.org (Guido van Rossum) Date: Fri, 10 Nov 2017 08:43:44 -0800 Subject: [Python-ideas] PEP 560 (second post) In-Reply-To: References: Message-ID: Hey Ivan, There seem to be some action items from this thread that I haven't seen reflected in the PEP source code yet. - Change the title (I like "Core support for typing module and generic types" but maybe it can be shortened to "Core support for generics in the typing module" -- if you like that, go for it) - Change __subclass_base__ to __mro_entry__ - Add types.resolve_bases and related changes - Maybe some clarifications summarizing the feedback from this thread (esp. Nick's)? Then the next step I propose is a PR with a full implementation. After that I'll likely approve the PEP (or we'll have minor feedback based on trying the implementation). I don't require a new typing.py that uses these features to be part of the "full implementation" but it would be nice to make a start with that as well -- if we can make the case that it will speed up "import typing" a lot then that would be a powerful argument for the PEP. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From levkivskyi at gmail.com Fri Nov 10 11:54:51 2017 From: levkivskyi at gmail.com (Ivan Levkivskyi) Date: Fri, 10 Nov 2017 17:54:51 +0100 Subject: [Python-ideas] PEP 560 (second post) In-Reply-To: References: Message-ID: On 10 November 2017 at 17:43, Guido van Rossum wrote: > Hey Ivan, > > There seem to be some action items from this thread that I haven't seen > reflected in the PEP source code yet. > [...snip...] > Then the next step I propose is a PR with a full implementation. After > that I'll likely approve the PEP (or we'll have minor feedback based on > trying the implementation). > > Yes, sorry, I wanted to make updates to the PEP and reference implementation, but last two weeks were very busy. Hopefully, I will work on it this weekend. -- Ivan -------------- next part -------------- An HTML attachment was scrubbed... URL: From k7hoven at gmail.com Fri Nov 10 12:39:08 2017 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Fri, 10 Nov 2017 19:39:08 +0200 Subject: [Python-ideas] PEP 560 (second post) In-Reply-To: References: Message-ID: On Wed, Sep 27, 2017 at 12:28 PM, Ivan Levkivskyi wrote: > Previously I posted PEP 560 two weeks ago, while several other PEPs were > also posted, so it didn't get much of attention. Here I post the PEP 560 > again, now including the full text for convenience of commenting. > > ?[..]? ? > ? > After creating the class, > the original bases are saved in ``__orig_bases__`` (currently this is also > done by the metaclass). > > ?Those are *still* bases, right, even if they are not in the mro?? I'm not sure if this is a naming thing or something even more. > NOTE: These two method names are reserved for exclusive use by > the ``typing`` module and the generic types machinery, and any other use is > strongly discouraged. > ?Given the situation, that may be a good thing. But will it really work? I think it is also strongly discouraged to invent your own dunder method names, but people still do it.? > The reference implementation (with tests) can be found > in [4]_, the proposal was originally posted and discussed on > the ``typing`` tracker, see [5]_. > > > Backwards compatibility and impact on users who don't use ``typing``: > ===================================================================== > > This proposal may break code that currently uses the names > ``__class_getitem__`` and ``__subclass_base__``. > ? > ???Koos? [..] -- + Koos Zevenhoven + http://twitter.com/k7hoven + -------------- next part -------------- An HTML attachment was scrubbed... URL: From levkivskyi at gmail.com Fri Nov 10 13:33:08 2017 From: levkivskyi at gmail.com (Ivan Levkivskyi) Date: Fri, 10 Nov 2017 19:33:08 +0100 Subject: [Python-ideas] PEP 560 (second post) In-Reply-To: References: Message-ID: On 10 November 2017 at 18:39, Koos Zevenhoven wrote: > On Wed, Sep 27, 2017 at 12:28 PM, Ivan Levkivskyi > wrote: > >> ? >> > ? >> After creating the class, >> the original bases are saved in ``__orig_bases__`` (currently this is also >> done by the metaclass). >> >> > ?Those are *still* bases, right, even if they are not in the mro?? I'm not > sure if this is a naming thing or something even more. > The objects that have __subclass_base__ method (proposed to rename to __mro_entry__) are removed from __bases__ attributed of the newly created class. Otherwise they may cause a metaclass conflict. One can however still call them syntactic (or static?) bases. For example this is how it is going to be used by typing: from typing import List class Tokens(List[int]): ... assert Tokens.__bases__ == (list,) > NOTE: These two method names are reserved for exclusive use by >> the ``typing`` module and the generic types machinery, and any other use >> is >> strongly discouraged. >> > > ?Given the situation, that may be a good thing. But will it really work? I > think it is also strongly discouraged to invent your own dunder method > names, but people still do it.? > Terry had a similar comment. I will "soften" this formulation in the next revision of the PEP. -- Ivan -------------- next part -------------- An HTML attachment was scrubbed... URL: From k7hoven at gmail.com Fri Nov 10 15:19:00 2017 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Fri, 10 Nov 2017 22:19:00 +0200 Subject: [Python-ideas] PEP 560 (second post) In-Reply-To: References: Message-ID: On Fri, Nov 10, 2017 at 8:33 PM, Ivan Levkivskyi wrote: > On 10 November 2017 at 18:39, Koos Zevenhoven wrote: > >> On Wed, Sep 27, 2017 at 12:28 PM, Ivan Levkivskyi >> wrote: >> >>> ? >>> >> ? >>> After creating the class, >>> the original bases are saved in ``__orig_bases__`` (currently this is >>> also >>> done by the metaclass). >>> >>> >> ?Those are *still* bases, right, even if they are not in the mro?? I'm >> not sure if this is a naming thing or something even more. >> > > The objects that have __subclass_base__ method (proposed to rename to > __mro_entry__) > are removed from __bases__ attributed of the newly created class. > Otherwise they may cause a metaclass conflict. > One can however still call them syntactic (or static?) bases. For example > this is how it is going to be used by typing: > > from typing import List > > class Tokens(List[int]): > ... > > assert Tokens.__bases__ == (list,) > ?Why is List[int] not allowed to be the base? Neither method-lookup performance nor the metaclass conflict issue seem to depend on whether List[int] is in __bases__. > > >> NOTE: These two method names are reserved for exclusive use by >>> the ``typing`` module and the generic types machinery, and any other use >>> is >>> strongly discouraged. >>> >> >> ?Given the situation, that may be a good thing. But will it really work? >> I think it is also strongly discouraged to invent your own dunder method >> names, but people still do it.? >> > > Terry had a similar comment. I will "soften" this formulation in the next > revision of the PEP. > > ? Right, I assume you mean the one where he pointed out that implicitly turning the methods into staticmethods based on their names makes those names reserved words. ? ?-- Koos -- + Koos Zevenhoven + http://twitter.com/k7hoven + -------------- next part -------------- An HTML attachment was scrubbed... URL: From levkivskyi at gmail.com Fri Nov 10 15:26:36 2017 From: levkivskyi at gmail.com (Ivan Levkivskyi) Date: Fri, 10 Nov 2017 21:26:36 +0100 Subject: [Python-ideas] PEP 560 (second post) In-Reply-To: References: Message-ID: On 10 November 2017 at 21:19, Koos Zevenhoven wrote: > On Fri, Nov 10, 2017 at 8:33 PM, Ivan Levkivskyi > wrote: > >> On 10 November 2017 at 18:39, Koos Zevenhoven wrote: >> >>> On Wed, Sep 27, 2017 at 12:28 PM, Ivan Levkivskyi >>> wrote: >>> >>>> ? >>>> >>> ? >>>> After creating the class, >>>> the original bases are saved in ``__orig_bases__`` (currently this is >>>> also >>>> done by the metaclass). >>>> >>>> >>> ?Those are *still* bases, right, even if they are not in the mro?? I'm >>> not sure if this is a naming thing or something even more. >>> >> >> The objects that have __subclass_base__ method (proposed to rename to >> __mro_entry__) >> are removed from __bases__ attributed of the newly created class. >> Otherwise they may cause a metaclass conflict. >> One can however still call them syntactic (or static?) bases. For example >> this is how it is going to be used by typing: >> >> from typing import List >> >> class Tokens(List[int]): >> ... >> >> assert Tokens.__bases__ == (list,) >> > > ?Why is List[int] not allowed to be the base? Neither method-lookup > performance nor the metaclass conflict issue seem to depend on whether > List[int] is in __bases__. > > The situation is actually quite opposite. Interestingly, the whole discussion started from Mark Shannon pointing to these problems with List[int] at the Language Summit. The original discussion on typing tracker is referenced in the PEP draft. -- Ivan -------------- next part -------------- An HTML attachment was scrubbed... URL: From k7hoven at gmail.com Fri Nov 10 15:46:49 2017 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Fri, 10 Nov 2017 22:46:49 +0200 Subject: [Python-ideas] PEP 560 (second post) In-Reply-To: References: Message-ID: On Fri, Nov 10, 2017 at 10:26 PM, Ivan Levkivskyi wrote: > On 10 November 2017 at 21:19, Koos Zevenhoven wrote: > >> On Fri, Nov 10, 2017 at 8:33 PM, Ivan Levkivskyi >> wrote: >> >>> On 10 November 2017 at 18:39, Koos Zevenhoven wrote: >>> >>>> On Wed, Sep 27, 2017 at 12:28 PM, Ivan Levkivskyi >>> > wrote: >>>> >>>>> ? >>>>> >>>> ? >>>>> After creating the class, >>>>> the original bases are saved in ``__orig_bases__`` (currently this is >>>>> also >>>>> done by the metaclass). >>>>> >>>>> >>>> ?Those are *still* bases, right, even if they are not in the mro?? I'm >>>> not sure if this is a naming thing or something even more. >>>> >>> >>> The objects that have __subclass_base__ method (proposed to rename to >>> __mro_entry__) >>> are removed from __bases__ attributed of the newly created class. >>> Otherwise they may cause a metaclass conflict. >>> One can however still call them syntactic (or static?) bases. For >>> example this is how it is going to be used by typing: >>> >>> from typing import List >>> >>> class Tokens(List[int]): >>> ... >>> >>> assert Tokens.__bases__ == (list,) >>> >> >> ?Why is List[int] not allowed to be the base? Neither method-lookup >> performance nor the metaclass conflict issue seem to depend on whether >> List[int] is in __bases__. >> >> > The situation is actually quite opposite. Interestingly, the whole > discussion started from Mark Shannon pointing to these problems with > List[int] at the Language Summit. > The original discussion on typing tracker is referenced in the PEP draft. > > ?What do you mean? I don't see any mention of __bases__ in the discussion. Perhaps related, if the syntax indeed were like this (as suggested by Mark?): @implements(List[int]) class Tokens(list): ... ?then I would expect List[int] to be in __implements__ or something like that. But how to interpret that? Maybe something like "List describes the interface/API that list has"? -- Koos -- + Koos Zevenhoven + http://twitter.com/k7hoven + -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri Nov 10 16:27:46 2017 From: guido at python.org (Guido van Rossum) Date: Fri, 10 Nov 2017 13:27:46 -0800 Subject: [Python-ideas] PEP 562 In-Reply-To: References: Message-ID: Picking up this thread as part of the PEP 562 and PEP 549 review. I like PEP 562 most, but I propose to add special-casing for `__dir__`. Not quite as proposed above (making the C level module_dir() look for `__all__`) but a bit more general -- making module_dir() look for `__dir__` and call that if present and callable. Ivan what do you think of that idea? It should be simple to add to your existing implementation. ( https://github.com/ilevkivskyi/cpython/pull/3#issuecomment-343591293) On Tue, Sep 12, 2017 at 1:26 AM, Ivan Levkivskyi wrote: > @Anthony > > module.__getattr__ works pretty well for normal access, after being > > imported by another module, but it doesn't properly trigger loading by > > functions defined in the module's own namespace. > > The idea of my PEP is to be very simple (both semantically and in terms > of implementation). This is why I don't want to add any complex logic. > People who will want to use __getattr__ for lazy loading still can do this > by importing submodules. > > @Nathaniel @INADA > > The main two use cases I know of for this and PEP 549 are lazy imports > > of submodules, and deprecating attributes. > > Yes, lazy loading seems to be a popular idea :-) > I will add the simple recipe by Inada to the PEP since it will already > work. > > @Cody > > I still think the better way > > to solve the custom dir() would be to change the module __dir__ > > method to check if __all__ is defined and use it to generate the > > result if it exists. This seems like a logical enhancement to me, > > and I'm planning on writing a patch to implement this. Whether it > > would be accepted is still an open issue though. > > This seems a reasonable rule to me, I can also make this patch if > you will not have time. > > @Guido > What do you think about the above idea? > > -- > Ivan > > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From levkivskyi at gmail.com Fri Nov 10 16:51:47 2017 From: levkivskyi at gmail.com (Ivan Levkivskyi) Date: Fri, 10 Nov 2017 22:51:47 +0100 Subject: [Python-ideas] PEP 562 In-Reply-To: References: Message-ID: On 10 November 2017 at 22:27, Guido van Rossum wrote: > Picking up this thread as part of the PEP 562 and PEP 549 review. I like > PEP 562 most, but I propose to add special-casing for `__dir__`. Not quite > as proposed above (making the C level module_dir() look for `__all__`) but > a bit more general -- making module_dir() look for `__dir__` and call that > if present and callable. Ivan what do you think of that idea? It should be > simple to add to your existing implementation. (https://github.com/ > ilevkivskyi/cpython/pull/3#issuecomment-343591293) > > I like this idea. I was thinking about yet another option: *extending* the result of current dir() search by contents __all__ if present (not just returning contents of __all__). But it looks like your idea covers more use cases, so I would stick with your idea. -- Ivan -------------- next part -------------- An HTML attachment was scrubbed... URL: From desmoulinmichel at gmail.com Sun Nov 12 01:19:37 2017 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Sun, 12 Nov 2017 07:19:37 +0100 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> Message-ID: > On Windows, which is the only platform I can reasonably comment on, > the killer issue is that the installer doesn't make the commands > "python" and "pip" available by default. Just checking my PC, both go > and rust (which I installed *ages* ago) do appear to, so they "just > work". Java sort-of also works like that (the runtime is on PATH, but > the compilers need to be added manually). > > The biggest reason we don't add Python to PATH, as I understand it, is > because we need to consider the implications of people having multiple > versions of Python installed. As far as I know, no other language > allows that. But equally, most beginners wouldn't actually *have* > multiple versions installed. So maybe we should optimise for that case > (only one version of Python installed). That would mean: > That seems reasonable. Especially since somebody with several versions install is more likely to know how to deal with system path issues than a total beginner installing python for the first time. > 1. Go back to adding Python to PATH. Because our installers don't say > "do you want to uninstall the old version", we should probably do a > check for a "python" command on PATH in the installer, and if there is > one, warn the user "You already have Python installed - if you are > upgrading you should manually uninstall the old version first, > otherwise your old installation will remain the default". We could get > more complex at this point, but that depends on what capabilities we > can include in the installer - I don't know how powerful the toolset > is. You don't even have to do that. We can detect it and prompt : "which python version do you want to be available by default ?", and then list command to run the alternative versions. > 2. Explicitly document that multiple Python interpreters on one > machine is considered "advanced", and users with that sort of setup > should be prepared to manage PATH themselves. I'd put that as > something like "It is possible to install multiple versions of Python > at once, but if you do that, you should understand the implications - > the average user should not need to do this" > > We still have to deal with the fact that basically every Unix > environment is "advanced" in the above sense (the python2/python3 > split). I don't have a solution for that (other than "upgrade to > Windows" ;-)). Provide the "py" command on linux and mac. And make it the default recommended way in the documentation. >> It doesn't completely solve the problem (as getting into and out of >> the environment is still platform specific), but it does mean that the >> ubiquitous online instructions to run "pip install package-name" and >> "python -m command" will actually work once people are inside their >> working environment. >> >> That tooling is venv: >> >> * it ensures you have "pip" on your PATH >> * it ensures you have "python" on your PATH >> * it ensures that you have the required permissions to install new packages >> * it ensures that any commands you install from PyPI will be also on your PATH >> >> When we choose not to use venv, then it becomes necessary to ensure >> each of those things individually for each potential system starting >> state > > Currently, the reality is that people use virtualenv, not venv. All > higher-level tools I'm aware of wrap virtualenv (to allow Python 2.7 > support). Enhancing the capabilities of venv is fine, but promoting > venv over virtualenv involves technical challenges across the whole > toolset, not just documentation/education. > > But agreed, once we get people into a virtual environment (of any > form) the portability issues become significantly reduced. The main > outstanding issue is managing multiple environments, which could be > handled by having a special "default" environment that is the only one > we'd expect beginners to use/need. Well, not exactly. Do you do python -m venv, or py -x.x -m venv or pythonx -m venv ? Wait, it's not installed by default on debian. And then virtualenv have their own issues: - it's a contextual mode that you need to activate AND be aware of at all time - you need to config tooling to use it (IDE, builders, etc) - the location of virtualenv is very important - you should not commit it to git - you can't relocate it easily - install jupyter / mymy / flake8 outside a virtualenv and the command will still seems to work inside a virtualenv it's not installed in. With terrible consequences. Virtualenvs are a hard tool to use for beginners. A lot of people on this list have forgotten their early years it seems. > Also on Windows, the per-user bin directory isn't added to PATH even > if you add the system Python to PATH in the installer. > >> That said, I think there is one improvement we could feasibly make, >> which would be to introduce the notion of a "default user environment" >> into `venv`, such that there was a single "python -m venv shell" >> command that: >> >> * created a default user environment if it didn't already exist >> * launched a subshell with that environment already activated >> >> This wouldn't be a full environment manager like vex or pew - it would >> just be a way to bootstrap a single usable package management >> environment in a cross-platform way. > > "How do I run venv?" is no easier to answer in a cross-platform way > than "how do I run pip?" You still need to be able to run > python/python3/py. About the only improvement is that there's no > legacy of documentation and advice on the web saying "run venv" like > there is for "run pip"... > > Paul > Exactly. From desmoulinmichel at gmail.com Sun Nov 12 01:20:38 2017 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Sun, 12 Nov 2017 07:20:38 +0100 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> Message-ID: <82885924-5a82-54a1-9df7-458ef2d9ab4a@gmail.com> Le 10/11/2017 ? 09:01, Nick Coghlan a ?crit?: > On 10 November 2017 at 17:05, Michel Desmoulin > wrote: >> >>> Which is why we advise getting into a virtual environment ASAP, such >>> that the only platform specific thing folks necessarily need to learn >>> to get started is how to get to that first working virtual >>> environment. >>> >> >> You can't start by teaching virtualenv. I tried. It doesn't work. And >> it's a terrible prerequisit if you write docs, tutorial, teaching >> materials, etc. > > You can't have it both ways - the only way we can systematically mask > the environmental differences between Windows, Linux and Mac OS X is > by providing tooling that actually masks those differences, which > means introducing that tooling becomes a prerequisite for teaching. > > It doesn't completely solve the problem (as getting into and out of > the environment is still platform specific), but it does mean that the > ubiquitous online instructions to run "pip install package-name" and > "python -m command" will actually work once people are inside their > working environment. > > That tooling is venv: > > * it ensures you have "pip" on your PATH > * it ensures you have "python" on your PATH > * it ensures that you have the required permissions to install new packages > * it ensures that any commands you install from PyPI will be also on your PATH > > When we choose not to use venv, then it becomes necessary to ensure > each of those things individually for each potential system starting > state > The way we do things is not the only way. Take JS: they don't have this problem, you npm install the same way everywhere. You don't have virtualenv but local node_modules. And you have transpilers to help with the langage version differences. Now I'm not advocating we do it the JS way. I'm just saying that you are very keen to defend a statu quo instead of offering ideas to solve the problem. Besides, using venv have the same issues. It's not installed on linux by defaut. And on windows you'll have to do py -x.x -m but on mac pythonx -m. From desmoulinmichel at gmail.com Sun Nov 12 01:21:03 2017 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Sun, 12 Nov 2017 07:21:03 +0100 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: <20171110114159.GB1740@phdru.name> References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> <20171110114159.GB1740@phdru.name> Message-ID: Le 10/11/2017 ? 12:41, Oleg Broytman a ?crit?: > On Fri, Nov 10, 2017 at 09:50:22AM +0000, Paul Moore wrote: >> The biggest reason we don't add Python to PATH, as I understand it, is >> because we need to consider the implications of people having multiple >> versions of Python installed. > > Why not fix that the same way as on Unix -- by having versioned > executables: python27.exe, python35.exe? Then python.exe in PATH will be > from the most recent installed Python. > >> Paul > > Oleg. > That or provider the "py" command on Unix. Either would work. From ncoghlan at gmail.com Sun Nov 12 04:55:43 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 12 Nov 2017 19:55:43 +1000 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: <82885924-5a82-54a1-9df7-458ef2d9ab4a@gmail.com> References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> <82885924-5a82-54a1-9df7-458ef2d9ab4a@gmail.com> Message-ID: On 12 November 2017 at 16:20, Michel Desmoulin wrote: > Le 10/11/2017 ? 09:01, Nick Coghlan a ?crit : >> On 10 November 2017 at 17:05, Michel Desmoulin >> wrote: >> When we choose not to use venv, then it becomes necessary to ensure >> each of those things individually for each potential system starting >> state >> > The way we do things is not the only way. Take JS: they don't have this > problem, you npm install the same way everywhere. You don't have > virtualenv but local node_modules. And you have transpilers to help with > the langage version differences. And they have browser manufacturers pouring millions of dollars a year into their tooling ecosystem into order to influence whose ad networks get clicked on most often. Python doesn't have that level of investment, but we do have fine folks volunteering to work on projects like `pip`, `virtualenv`, PyPI, and the packaging.python.org documentation project, as well as backports of standard library modules to earlier Python versions. When someone attempts to explain to you the practical challenges that limit both python-dev's and PyPA's ability to reliably control the starting experience of new Python users, the appropriate response is to *start listening*, not harangue them for failing to immediately follow your peremptory orders to make things simpler for you. "Do it because I said so" is bad management style even in an actual company - it's even worse in a peer production environment like an open source community. > Now I'm not advocating we do it the JS way. I'm just saying that you are > very keen to defend a statu quo instead of offering ideas to solve the > problem. I already opened https://github.com/pypa/python-packaging-user-guide/issues/396 to track possible areas of concrete near term improvement (I would have tagged you on the issue, but I couldn't find a GitHub account under your name). > Besides, using venv have the same issues. It's not installed on linux by > defaut. That depends greatly on which Linux you install - the choice on whether to deliberately cripple "python3 -m venv" or not is made by the maintainers for that distribution, and Python-friendly distros make sure Python's native tooling works properly by default. > And on windows you'll have to do py -x.x -m but on mac pythonx -m. CPython updates take years to reliably roll out to end user systems, so if you're looking to have an impact in a shorter time frame than that, the differences in cross-platform invocation are a constraint you're going to have to learn to live with. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Sun Nov 12 07:20:45 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 12 Nov 2017 12:20:45 +0000 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> Message-ID: On 12 November 2017 at 06:19, Michel Desmoulin wrote: >> 1. Go back to adding Python to PATH. Because our installers don't say >> "do you want to uninstall the old version", we should probably do a >> check for a "python" command on PATH in the installer, and if there is >> one, warn the user "You already have Python installed - if you are >> upgrading you should manually uninstall the old version first, >> otherwise your old installation will remain the default". We could get >> more complex at this point, but that depends on what capabilities we >> can include in the installer - I don't know how powerful the toolset >> is. > > You don't even have to do that. We can detect it and prompt : "which > python version do you want to be available by default ?", and then list > command to run the alternative versions. I deliberately avoided suggesting automatically changing the default version, because that's fraught with problems. You need to remove the previous default from PATH, and if you're doing a user install but the previous version is in the system PATH you don't have the privileges to do that. If the previous version was another distribution (e.g. Anaconda) you've no way to know that and conversely you don't know how to remove that distribution's path entries (is there a Scripts directory to remove, did they use bin instead, etc). The list of potential problems is endless. >> We still have to deal with the fact that basically every Unix >> environment is "advanced" in the above sense (the python2/python3 >> split). I don't have a solution for that (other than "upgrade to >> Windows" ;-)). > > Provide the "py" command on linux and mac. And make it the default > recommended way in the documentation. +1 from me. > Well, not exactly. Do you do python -m venv, or py -x.x -m venv or > pythonx -m venv ? Wait, it's not installed by default on debian. Seriously? Debian don't provide venv in the standard Python install? That's just broken. > Virtualenvs are a hard tool to use for beginners. Agreed. Genuine beginners just install Python, then use it. If they install extra packages, they want them to be available to all of their scripts. > A lot of people on this list have forgotten their early years it seems. Maybe. But the default beginner approach *does* have its problems, and guiding beginners to a better approach is a good idea. It's just that the starting point needs to be showing them why the problems solved by tools like virtualenv matter to them. At this point, though, we've moved into a much bigger issue than "it's hard to get started with pip". We should keep the discussion focused on the immediate problem, and not try to solve everything at once. Paul From ncoghlan at gmail.com Sun Nov 12 08:18:26 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 12 Nov 2017 23:18:26 +1000 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> Message-ID: On 12 November 2017 at 22:20, Paul Moore wrote: > On 12 November 2017 at 06:19, Michel Desmoulin > wrote: >>> We still have to deal with the fact that basically every Unix >>> environment is "advanced" in the above sense (the python2/python3 >>> split). I don't have a solution for that (other than "upgrade to >>> Windows" ;-)). >> >> Provide the "py" command on linux and mac. And make it the default >> recommended way in the documentation. > > +1 from me. The current Python launcher is Window specific, and doesn't work on POSIX systems. So the first barrier here is "Write a py command that presents the same CLI on POSIX that the Python launcher does on Windows". The second problem is that even though we can add a "py" command to the python.org Mac OS X installers and have that take effect for new python.org downloads, there are a *lot* of people on Mac OS X and Linux that don't actually obtain their pre-built Python from python.org. While some of those would probably adapt and ship a new command fairly quickly (the cross-platform commercial redistributors, rolling release and 6-month cadence Linux distros, community redistributors like pythonz, pyenv, and homebrew), other important platforms wouldn't (most notably, LTS Linux distros). The fact it wouldn't actually solve the bootstrapping problem due to the update policies of various popular platforms then greatly reduces the incentive to work on the first part of the problem (i.e. actually creating a POSIX-compatible version of that CLI). >> Well, not exactly. Do you do python -m venv, or py -x.x -m venv or >> pythonx -m venv ? Wait, it's not installed by default on debian. > > Seriously? Debian don't provide venv in the standard Python install? > That's just broken. Yup. And RHEL/CentOS don't provide Python 3.x by default at all - you need to grab it via other means. >> Virtualenvs are a hard tool to use for beginners. > > Agreed. Genuine beginners just install Python, then use it. If they > install extra packages, they want them to be available to all of their > scripts. Also agreed, but without them (or a functional equivalent, like `conda env` or `pyenv`), it's incredibly hard to get an arbitrary user on an arbitrary system to the point where all the following assumptions are satisfied: * the user is working at a command prompt (whether inside their IDE, or in an actual system shell) * the command `python` will run the desired version of Python * the commands `pip install` and `python -m pip install` will be essentially equivalent (aside from the differences when upgrading pip itself on Windows) and install packages into the desired version of Python * any executables installed that way will be executable from that command prompt * any Python import packages installed that way will be importable from a Python shell started in that environment (List of assumptions taken from https://github.com/pypa/python-packaging-user-guide/issues/396) Trying to drop any of those assumptions is problematic: * non-command-shell learning environments vary even more than command shells do * the `python` command has a much long history than any of the alternatives (including both `py` and `python3`), and third party guides reflect that * the `pip install` option really is nicer looking than `python -m pip install`, and it only has actual problems in the presence of multiple Python versions and when upgrading pip itself on Windows (plus: lots of third party guides recommend it, as do pypi.org project pages) * if `pip install devtool` doesn't let you run `command-from-devtool` in your command shell, that's thoroughly unhelpful * if `pip install dist-package` doesn't let you run `import library_from_dist_package` in your Python code, that's even less helpful It *would* be much nicer if a single user install of the Windows python.org installer met these criteria by default, though - while the `py` launcher is a good way of handling multiple versions (and nicely fills the role of handling file associations for Python file extensions), it isn't a particularly nice alternative to a properly configured user PATH on a machine with only one Python installation. >> A lot of people on this list have forgotten their early years it seems. > > Maybe. But the default beginner approach *does* have its problems, and > guiding beginners to a better approach is a good idea. We haven't forgotten our early years - we've just spent years (both individually and collectively) working on the problem of helping people get started with software development, and thus have a very good idea as to what *doesn't* work, as well as what *does* work. The latter list is currently fairly short: * having a friend or colleague walk them through it * "Install Parties", like those Django Girls runs (i.e. running a pre-tutorial event, specifically focused on getting a working environment set up) * highly prescriptive learning environments, whether online ones (like Grok Learning, trinket.io, PythonAnywhere, etc), or locally installed ones (like PyCharm Educational Edition, the Anaconda distribution, etc) Without the kinds of constraints suggested in the last option, there are too many potential starting points, and it isn't even possible to ask potential learners to self-assess what their starting point actually is, since it's a tech support problem where the first task is "assess the current state of the user's system" (hence the first two options). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Sun Nov 12 08:48:39 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 12 Nov 2017 13:48:39 +0000 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> Message-ID: On 12 November 2017 at 13:18, Nick Coghlan wrote: >> Seriously? Debian don't provide venv in the standard Python install? >> That's just broken. > > Yup. And RHEL/CentOS don't provide Python 3.x by default at all - you > need to grab it via other means. Wow. I have no problem with not providing Python 3 by default, but shipping a version that omits features that are non-optional parts of the standard library is ridiculous. I'm getting more and more inclined to make my default response to bug reports from people on Debian/Ubuntu be "report it to your vendor or demonstrate it on a vanilla build of Python" :-( Paul From brenbarn at brenbarn.net Sun Nov 12 11:59:19 2017 From: brenbarn at brenbarn.net (Brendan Barnwell) Date: Sun, 12 Nov 2017 08:59:19 -0800 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> Message-ID: <5A087DE7.4050306@brenbarn.net> On 2017-11-12 05:18, Nick Coghlan wrote: > * the `pip install` option really is nicer looking than `python -m pip > install`, and it only has actual problems in the presence of multiple > Python versions and when upgrading pip itself on Windows (plus: lots > of third party guides recommend it, as do pypi.org project pages) Is there any *advantage* to using `pip install` instead of `python -m install`? If not, could we at least change everything under Python/pip control (e.g., pip documentation) to never recommend `pip` and always recommend `python -m pip` instead, and encourage all third-party documentation to always use `python -m pip` and never use `pip`? Obviously this isn't a full solution, but in the end there's no way we change external third-party documentation, which will always eventually become outdated. Absent that, it seems worthwhile to regularize existing official documentation. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown From stephanh42 at gmail.com Sun Nov 12 13:02:12 2017 From: stephanh42 at gmail.com (Stephan Houben) Date: Sun, 12 Nov 2017 19:02:12 +0100 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> Message-ID: 2017-11-12 13:20 GMT+01:00 Paul Moore : > On 12 November 2017 at 06:19, Michel Desmoulin > wrote: > > > Well, not exactly. Do you do python -m venv, or py -x.x -m venv or > > pythonx -m venv ? Wait, it's not installed by default on debian. > > Seriously? Debian don't provide venv in the standard Python install? > That's just broken. > I think that it is wrong to think of Debian's "python3" package as purporting to provide the "standard Python install". This is really more properly though of as the Python *runtime* environment, i.e. the minimum stuff you need to run programs written in Python. Developers are supposed to install additional packages. Debian does the same for other languages,, i.e. their "node" package doesn't contain npm. I suppose it makes sense; a modern Linux desktop contains applications written in Python/Perl/Node/Ruby/Tcl/OCaml/..., if you start including all development tools in the base packages you are probably ending doubling a typical install. It's still annoying, I have personally decided that it is simpler to just install the tarball from python.org than to chase all the individual packages over which they split the Python install. Stephan -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sun Nov 12 13:38:51 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 12 Nov 2017 19:38:51 +0100 Subject: [Python-ideas] venv *is* provided in the standard Python install on Debian/Ubuntu References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> Message-ID: <20171112193851.3c24085f@fsol> On Sun, 12 Nov 2017 12:20:45 +0000 Paul Moore wrote: > > > Well, not exactly. Do you do python -m venv, or py -x.x -m venv or > > pythonx -m venv ? Wait, it's not installed by default on debian. > > Seriously? Debian don't provide venv in the standard Python install? > That's just broken. Frankly, I don't know where the current discussion comes from, but on two recent Debian and Ubuntu setups, I get: $ dpkg -S /usr/lib/python3.5/venv/__init__.py libpython3.5-stdlib:amd64: /usr/lib/python3.5/venv/__init__.py Which, for the uninitiated, means "the venv module is provided by the Debian/Ubuntu package named libpython3.5-stdlib". That package is, in turn, a dependency of the "python3.5" package. Regards Antoine. From solipsis at pitrou.net Sun Nov 12 13:51:47 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 12 Nov 2017 19:51:47 +0100 Subject: [Python-ideas] Looking for input to help with the pip situation References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> Message-ID: <20171112195147.759cc838@fsol> On Sun, 12 Nov 2017 23:18:26 +1000 Nick Coghlan wrote: > > We haven't forgotten our early years - we've just spent years (both > individually and collectively) working on the problem of helping > people get started with software development, and thus have a very > good idea as to what *doesn't* work, as well as what *does* work. > > The latter list is currently fairly short: > > * having a friend or colleague walk them through it > * "Install Parties", like those Django Girls runs (i.e. running a > pre-tutorial event, specifically focused on getting a working > environment set up) > * highly prescriptive learning environments, whether online ones (like > Grok Learning, trinket.io, PythonAnywhere, etc), or locally installed > ones (like PyCharm Educational Edition, the Anaconda distribution, > etc) Not wanting to nitpick, but I don't think the Anaconda distribution is "highly prescriptive". It's a software distribution with scientific computing as its main focus, but perfectly usable for ordinary Python programming. It's not more prescriptive than Debian, which by its philosophy is directed primarily towards sysadmin crowds but also used by some people on their personal desktops. Also, you can create custom Anaconda-like distributions to provide the desired environment to your students in an executable installer (*). I'm not sure how usable that option is, but it definitely exists, is open source and cross-platform. (*) https://github.com/conda/constructor Regards Antoine. From p.f.moore at gmail.com Sun Nov 12 14:12:43 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 12 Nov 2017 19:12:43 +0000 Subject: [Python-ideas] venv *is* provided in the standard Python install on Debian/Ubuntu In-Reply-To: <20171112193851.3c24085f@fsol> References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> <20171112193851.3c24085f@fsol> Message-ID: On 12 November 2017 at 18:38, Antoine Pitrou wrote: > On Sun, 12 Nov 2017 12:20:45 +0000 > Paul Moore wrote: >> >> > Well, not exactly. Do you do python -m venv, or py -x.x -m venv or >> > pythonx -m venv ? Wait, it's not installed by default on debian. >> >> Seriously? Debian don't provide venv in the standard Python install? >> That's just broken. > > Frankly, I don't know where the current discussion comes from, but on > two recent Debian and Ubuntu setups, I get: > > $ dpkg -S /usr/lib/python3.5/venv/__init__.py > libpython3.5-stdlib:amd64: /usr/lib/python3.5/venv/__init__.py > > > Which, for the uninitiated, means "the venv module is provided by the > Debian/Ubuntu package named libpython3.5-stdlib". That package is, in > turn, a dependency of the "python3.5" package. Thanks for the clarification. I was surprised by the assertion, but didn't have a Debian system to check, so it's good to know it's false. Paul From stephanh42 at gmail.com Sun Nov 12 14:24:35 2017 From: stephanh42 at gmail.com (Stephan Houben) Date: Sun, 12 Nov 2017 20:24:35 +0100 Subject: [Python-ideas] venv *is* provided in the standard Python install on Debian/Ubuntu In-Reply-To: <20171112193851.3c24085f@fsol> References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> <20171112193851.3c24085f@fsol> Message-ID: Hi Antoine, The venv module is included, however the pyvenv script is in a separate package python3.5-venv . By the way, I was totally confused by the following text form the doc. https://docs.python.org/3/library/venv.html ======== Deprecated since version 3.6: pyvenv was the recommended tool for creating virtual environments for Python 3.3 and 3.4, and is deprecated in Python 3.6 . Changed in version 3.5: The use of venv is now recommended for creating virtual environments. ======== So many questions: * What is the status of "pyenv" in 3.5? Apparently it is not deprecated there. * What is it replaced by? Apparently "venv", but it doesn't say so explicitly. * Is "venv" the same thing as "python -m venv" discussed earlier? Or is it a different thing? With so many things names so similarly, it is hard to tell * What does it mean for "venv" to be recommend in 3.5 if "pyvenv" is not deprecated there? Also, the link brings us to a long page of "What's New in Python 3.6" . Just searching for "venv" only gives the apparently irrelevant: "venv accepts a new parameter --prompt. This parameter provides an alternative prefix for the virtual environment. (Proposed by ?ukasz Balcerzak and ported to 3.6 by St?phane Wirtel in bpo-22829 .) " I suppose at that point the newbie gave up and downloaded node.js ;-) Stephan On Sun, 12 Nov 2017 12:20:45 +0000 > Paul Moore wrote: > > > > > Well, not exactly. Do you do python -m venv, or py -x.x -m venv or > > > pythonx -m venv ? Wait, it's not installed by default on debian. > > > > Seriously? Debian don't provide venv in the standard Python install? > > That's just broken. > > Frankly, I don't know where the current discussion comes from, but on > two recent Debian and Ubuntu setups, I get: > > $ dpkg -S /usr/lib/python3.5/venv/__init__.py > libpython3.5-stdlib:amd64: /usr/lib/python3.5/venv/__init__.py > > > Which, for the uninitiated, means "the venv module is provided by the > Debian/Ubuntu package named libpython3.5-stdlib". That package is, in > turn, a dependency of the "python3.5" package. > > Regards > > Antoine. > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From antoine at python.org Sun Nov 12 14:31:25 2017 From: antoine at python.org (Antoine Pitrou) Date: Sun, 12 Nov 2017 20:31:25 +0100 Subject: [Python-ideas] venv *is* provided in the standard Python install on Debian/Ubuntu In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> <20171112193851.3c24085f@fsol> Message-ID: <01575482-ae34-045e-4cde-7de476b893b1@python.org> Hi Stephan, Le 12/11/2017 ? 20:24, Stephan Houben a ?crit?: > Hi Antoine, > > The venv module is included, > however the pyvenv script is in a separate package > python3.5-venv . Thanks for the correction. That's definitely a silly decision -- especially as the "pyvenv" script must be very small anyway, perhaps even smaller than the package metadata that goes with it? It's not even internally consistent within Debian, as the "pyvenv" script is packaged separately but the "pydoc" script is bundled with the "python package": $ dpkg -S `which pyvenv` python3-venv: /usr/bin/pyvenv $ dpkg -S `which pydoc` python: /usr/bin/pydoc (note also how "pyvenv" and not "pyvenv3" is installed by "python3-venv"...) > So many questions: > * What is the status of "pyenv" in 3.5? Apparently it is not deprecated > there. > * What is it replaced by? Apparently "venv", but it doesn't say so > explicitly. > * Is "venv" the same thing as "python -m venv" discussed earlier? Or is it a > ? different thing? With so many things names so similarly, it is hard to > tell I guess so... Personally, I don't think that deprecation was a good idea. As Nick says, there is point to be made for short easy-to-remember commands such as "pyvenv" or "pip install". > * What does it mean for "venv" to be recommend in 3.5 if "pyvenv" is not > deprecated there? Probably that "python -m venv" is the recommended replacement. > I suppose at that point the newbie gave up and downloaded node.js ;-) Perhaps not a good idea, but hey :-)) Regards Antoine. From rosuav at gmail.com Sun Nov 12 16:11:05 2017 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 13 Nov 2017 08:11:05 +1100 Subject: [Python-ideas] venv *is* provided in the standard Python install on Debian/Ubuntu In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> <20171112193851.3c24085f@fsol> Message-ID: On Mon, Nov 13, 2017 at 6:24 AM, Stephan Houben wrote: > Hi Antoine, > > The venv module is included, > however the pyvenv script is in a separate package > python3.5-venv . > > By the way, I was totally confused by the following text form the doc. > > https://docs.python.org/3/library/venv.html > > ======== > Deprecated since version 3.6: pyvenv was the recommended tool for creating > virtual environments for Python 3.3 and 3.4, and is deprecated in Python > 3.6. > > Changed in version 3.5: The use of venv is now recommended for creating > virtual environments. > > ======== Not sure where you're reading that. I'm seeing: """ Note The pyvenv script has been deprecated as of Python 3.6 in favor of using python3 -m venv to help prevent any potential confusion as to which Python interpreter a virtual environment will be based on. """ I think that's pretty clear. "python3 -m venv env" is the standard and recommended way to spin up a virtual environment. ChrisA From ncoghlan at gmail.com Sun Nov 12 18:18:48 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 13 Nov 2017 09:18:48 +1000 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: <5A087DE7.4050306@brenbarn.net> References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> <5A087DE7.4050306@brenbarn.net> Message-ID: On 13 November 2017 at 02:59, Brendan Barnwell wrote: > On 2017-11-12 05:18, Nick Coghlan wrote: >> >> * the `pip install` option really is nicer looking than `python -m pip >> install`, and it only has actual problems in the presence of multiple >> Python versions and when upgrading pip itself on Windows (plus: lots >> of third party guides recommend it, as do pypi.org project pages) > > > Is there any *advantage* to using `pip install` instead of `python > -m install`? If not, could we at least change everything under Python/pip > control (e.g., pip documentation) to never recommend `pip` and always > recommend `python -m pip` instead, and encourage all third-party > documentation to always use `python -m pip` and never use `pip`? We've already changed most of them (pypi.org itself is the main one that we haven't changed it yet). However, there are still per-project READMEs out there that suggest "easy_install project" and direct invocation of "python setup.py install", so it really isn't appealing to layer yet another mandatory change in the recommended spelling of the installation command and create yet another point of confusion - it will be much nicer overall if we can retroactively make the existing "pip install" instructions correct for most users, and leave "python -m pip install" to the "Multiple versions of Python" and "Self-upgrading pip on Windows". Cheers, Nick. P.S. As a user, it's also genuinely irritating to have to always type the "python -m " prefix when inside an active virtual environment, as in that case, there isn't any ambiguity about which environment pip should be manipulating or which version of Python it should be using. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sun Nov 12 18:25:27 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 13 Nov 2017 09:25:27 +1000 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: <20171112195147.759cc838@fsol> References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> <20171112195147.759cc838@fsol> Message-ID: On 13 November 2017 at 04:51, Antoine Pitrou wrote: > On Sun, 12 Nov 2017 23:18:26 +1000 > Nick Coghlan wrote: >> >> We haven't forgotten our early years - we've just spent years (both >> individually and collectively) working on the problem of helping >> people get started with software development, and thus have a very >> good idea as to what *doesn't* work, as well as what *does* work. >> >> The latter list is currently fairly short: >> >> * having a friend or colleague walk them through it >> * "Install Parties", like those Django Girls runs (i.e. running a >> pre-tutorial event, specifically focused on getting a working >> environment set up) >> * highly prescriptive learning environments, whether online ones (like >> Grok Learning, trinket.io, PythonAnywhere, etc), or locally installed >> ones (like PyCharm Educational Edition, the Anaconda distribution, >> etc) > > Not wanting to nitpick, but I don't think the Anaconda distribution is > "highly prescriptive". It's a software distribution with scientific > computing as its main focus, but perfectly usable for ordinary Python > programming. It's not more prescriptive than Debian, which by its > philosophy is directed primarily towards sysadmin crowds but also used > by some people on their personal desktops. I meant prescriptive in the sense of "Here's how to get the binaries and manage upgrades". The non-prescriptive alternative is "Use any Python binary you like, and we'll try to adjust to that". The latter can potentially work given sufficiently experienced instructors (or a suitable mix of instructors with knowledge of different platforms), but it's not easy. (The downside of this being that prescriptive environments for particular workshops is one of the most common ways for beginners to end up with multiple versions of Python installed: whatever they found on their own, plus the version their workshop instructors told them to use). > Also, you can create custom Anaconda-like distributions to provide the > desired environment to your students in an executable installer (*). > I'm not sure how usable that option is, but it definitely exists, is > open source and cross-platform. > > (*) https://github.com/conda/constructor Oh, cool - I didn't know about that. Yeah, that could definitely provide an interesting middle ground between "Choose your own adventure" and "Here's the entire scientific Python stack". Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sun Nov 12 18:29:02 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 13 Nov 2017 09:29:02 +1000 Subject: [Python-ideas] venv *is* provided in the standard Python install on Debian/Ubuntu In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> <20171112193851.3c24085f@fsol> Message-ID: On 13 November 2017 at 07:11, Chris Angelico wrote: > On Mon, Nov 13, 2017 at 6:24 AM, Stephan Houben wrote: >> Hi Antoine, >> >> The venv module is included, >> however the pyvenv script is in a separate package >> python3.5-venv . >> >> By the way, I was totally confused by the following text form the doc. >> >> https://docs.python.org/3/library/venv.html >> >> ======== >> Deprecated since version 3.6: pyvenv was the recommended tool for creating >> virtual environments for Python 3.3 and 3.4, and is deprecated in Python >> 3.6. >> >> Changed in version 3.5: The use of venv is now recommended for creating >> virtual environments. >> >> ======== > > Not sure where you're reading that. I'm seeing: > > """ > Note > The pyvenv script has been deprecated as of Python 3.6 in favor of > using python3 -m venv to help prevent any potential confusion as to > which Python interpreter a virtual environment will be based on. > """ > > I think that's pretty clear. "python3 -m venv env" is the standard and > recommended way to spin up a virtual environment. It's further down in the page, under https://docs.python.org/3/library/venv.html#creating-virtual-environments I think the deprecation notice for pyvenv should just be deleted, since it renders like the *module* is deprecated. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sun Nov 12 18:30:06 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 13 Nov 2017 09:30:06 +1000 Subject: [Python-ideas] venv *is* provided in the standard Python install on Debian/Ubuntu In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> <20171112193851.3c24085f@fsol> Message-ID: On 13 November 2017 at 09:29, Nick Coghlan wrote: > On 13 November 2017 at 07:11, Chris Angelico wrote: >> On Mon, Nov 13, 2017 at 6:24 AM, Stephan Houben wrote: >>> Hi Antoine, >>> >>> The venv module is included, >>> however the pyvenv script is in a separate package >>> python3.5-venv . >>> >>> By the way, I was totally confused by the following text form the doc. >>> >>> https://docs.python.org/3/library/venv.html >>> >>> ======== >>> Deprecated since version 3.6: pyvenv was the recommended tool for creating >>> virtual environments for Python 3.3 and 3.4, and is deprecated in Python >>> 3.6. >>> >>> Changed in version 3.5: The use of venv is now recommended for creating >>> virtual environments. >>> >>> ======== >> >> Not sure where you're reading that. I'm seeing: >> >> """ >> Note >> The pyvenv script has been deprecated as of Python 3.6 in favor of >> using python3 -m venv to help prevent any potential confusion as to >> which Python interpreter a virtual environment will be based on. >> """ >> >> I think that's pretty clear. "python3 -m venv env" is the standard and >> recommended way to spin up a virtual environment. > > It's further down in the page, under > https://docs.python.org/3/library/venv.html#creating-virtual-environments > > I think the deprecation notice for pyvenv should just be deleted, > since it renders like the *module* is deprecated. That is, the confusing one starting with "Deprecated since version 3.6: ...". The note Chris quoted is fine, and should be kept. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From tritium-list at sdamon.com Sun Nov 12 18:32:07 2017 From: tritium-list at sdamon.com (Alex Walters) Date: Sun, 12 Nov 2017 18:32:07 -0500 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> <5A087DE7.4050306@brenbarn.net> Message-ID: <1b8f6701d35c0e$7517d4f0$5f477ed0$@sdamon.com> > -----Original Message----- > From: Python-ideas [mailto:python-ideas-bounces+tritium- > list=sdamon.com at python.org] On Behalf Of Nick Coghlan > Sent: Sunday, November 12, 2017 6:19 PM > To: Brendan Barnwell > Cc: python-ideas at python.org > Subject: Re: [Python-ideas] Looking for input to help with the pip situation > > On 13 November 2017 at 02:59, Brendan Barnwell > wrote: > > On 2017-11-12 05:18, Nick Coghlan wrote: > >> > >> * the `pip install` option really is nicer looking than `python -m pip > >> install`, and it only has actual problems in the presence of multiple > >> Python versions and when upgrading pip itself on Windows (plus: lots > >> of third party guides recommend it, as do pypi.org project pages) > > > > > > Is there any *advantage* to using `pip install` instead of `python > > -m install`? If not, could we at least change everything under Python/pip > > control (e.g., pip documentation) to never recommend `pip` and always > > recommend `python -m pip` instead, and encourage all third-party > > documentation to always use `python -m pip` and never use `pip`? > > We've already changed most of them (pypi.org itself is the main one > that we haven't changed it yet). > > However, there are still per-project READMEs out there that suggest > "easy_install project" and direct invocation of "python setup.py > install", so it really isn't appealing to layer yet another mandatory > change in the recommended spelling of the installation command and > create yet another point of confusion - it will be much nicer overall > if we can retroactively make the existing "pip install" instructions > correct for most users, and leave "python -m pip install" to the > "Multiple versions of Python" and "Self-upgrading pip on Windows". > > Cheers, > Nick. > > P.S. As a user, it's also genuinely irritating to have to always type > the "python -m " prefix when inside an active virtual environment, as > in that case, there isn't any ambiguity about which environment pip > should be manipulating or which version of Python it should be using. > You kinda should be typing `env/bin/python -m ...` (or env\Scripts\python.exe -m ...) but saying that proves your previous point. > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From rosuav at gmail.com Sun Nov 12 18:32:27 2017 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 13 Nov 2017 10:32:27 +1100 Subject: [Python-ideas] venv *is* provided in the standard Python install on Debian/Ubuntu In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> <20171112193851.3c24085f@fsol> Message-ID: On Mon, Nov 13, 2017 at 10:29 AM, Nick Coghlan wrote: > On 13 November 2017 at 07:11, Chris Angelico wrote: >> On Mon, Nov 13, 2017 at 6:24 AM, Stephan Houben wrote: >>> Hi Antoine, >>> >>> The venv module is included, >>> however the pyvenv script is in a separate package >>> python3.5-venv . >>> >>> By the way, I was totally confused by the following text form the doc. >>> >>> https://docs.python.org/3/library/venv.html >>> >>> ======== >>> Deprecated since version 3.6: pyvenv was the recommended tool for creating >>> virtual environments for Python 3.3 and 3.4, and is deprecated in Python >>> 3.6. >>> >>> Changed in version 3.5: The use of venv is now recommended for creating >>> virtual environments. >>> >>> ======== >> >> Not sure where you're reading that. I'm seeing: >> >> """ >> Note >> The pyvenv script has been deprecated as of Python 3.6 in favor of >> using python3 -m venv to help prevent any potential confusion as to >> which Python interpreter a virtual environment will be based on. >> """ >> >> I think that's pretty clear. "python3 -m venv env" is the standard and >> recommended way to spin up a virtual environment. > > It's further down in the page, under > https://docs.python.org/3/library/venv.html#creating-virtual-environments > > I think the deprecation notice for pyvenv should just be deleted, > since it renders like the *module* is deprecated. Ah, I see it now, thanks. Agreed; or maybe downgrade it to a parenthetical comment. Focus on "this is how to do the obvious thing", and only as an afterthought mention "it used to be done differently" in case someone greps for pyvenv. ChrisA From ncoghlan at gmail.com Sun Nov 12 21:32:06 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 13 Nov 2017 12:32:06 +1000 Subject: [Python-ideas] venv *is* provided in the standard Python install on Debian/Ubuntu In-Reply-To: <20171112193851.3c24085f@fsol> References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> <20171112193851.3c24085f@fsol> Message-ID: On 13 November 2017 at 04:38, Antoine Pitrou wrote: > On Sun, 12 Nov 2017 12:20:45 +0000 > Paul Moore wrote: >> >> > Well, not exactly. Do you do python -m venv, or py -x.x -m venv or >> > pythonx -m venv ? Wait, it's not installed by default on debian. >> >> Seriously? Debian don't provide venv in the standard Python install? >> That's just broken. > > Frankly, I don't know where the current discussion comes from, but on > two recent Debian and Ubuntu setups, I get: > > $ dpkg -S /usr/lib/python3.5/venv/__init__.py > libpython3.5-stdlib:amd64: /usr/lib/python3.5/venv/__init__.py The discussion comes from the fact that even though the code is present it doesn't actually work: $ docker run --rm -it ncoghlan/debian-python bash root at e9c0aa482aeb:/# python3 -m venv test_venv Error: Command '['/test_venv/bin/python3', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1 (I just refreshed my Docker image to 9.2, so that's the current default behaviour after "apt-get install -y python3") So technically it's ensurepip that's broken by default, but that translates to venv also being broken by default. I haven't worked out what the actual steps needed to fix it are (but I do know that installing "python3-pip" isn't enough the way it is on Fedora - we reconcile ensurepip with our security management policies by having a Recommends dep from python3 to python3-pip, and then patching ensurepip to use rewheel to inject a copy of the system pip into freshly created virtual environments. At least in theory, Debian should be able to do something similar with dirtbike, but whether or not they actually will would be a question for Matthias Klose as the Debian Python maintainer). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From stephanh42 at gmail.com Mon Nov 13 02:46:13 2017 From: stephanh42 at gmail.com (Stephan Houben) Date: Mon, 13 Nov 2017 08:46:13 +0100 Subject: [Python-ideas] venv *is* provided in the standard Python install on Debian/Ubuntu In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> <20171112193851.3c24085f@fsol> Message-ID: 2017-11-13 3:32 GMT+01:00 Nick Coghlan : > > So technically it's ensurepip that's broken by default, but that > translates to venv also being broken by default. > > I haven't worked out what the actual steps needed to fix it are > On Debian, ensurepip is in the python3.5-venv package. https://packages.debian.org/stretch/amd64/python3.5-venv/filelist Unfortunately there isn't (AFAIK) a meta-package which just says "stop fooling around and just give me everything from the python.org distribution, d*mmit." Stephan -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephanh42 at gmail.com Mon Nov 13 03:00:32 2017 From: stephanh42 at gmail.com (Stephan Houben) Date: Mon, 13 Nov 2017 09:00:32 +0100 Subject: [Python-ideas] venv *is* provided in the standard Python install on Debian/Ubuntu In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> <20171112193851.3c24085f@fsol> Message-ID: Hi all, Related to this text on https://docs.python.org/3/library/venv.html : ============ Note The pyvenv script has been deprecated as of Python 3.6 in favor of using python3 -m venv to help prevent any potential confusion as to which Python interpreter a virtual environment will be based on. ============ It's clearer than the text below to which I originally referred. However, this text has also problems in that it is too unix-specific. In particular: * Most seriously, it refers to "python3" which doesn't work with the python.org Windows installer. * Less seriously, it refers to "pyenv" as a "script" which is unix jargon and moreover technically incorrect on Windows. (Also, needlessly specific, it should just be "the pyenv command", how it is implemented is irrelevant for this section). Stephan 2017-11-13 0:32 GMT+01:00 Chris Angelico : > On Mon, Nov 13, 2017 at 10:29 AM, Nick Coghlan wrote: > > On 13 November 2017 at 07:11, Chris Angelico wrote: > >> On Mon, Nov 13, 2017 at 6:24 AM, Stephan Houben > wrote: > >>> Hi Antoine, > >>> > >>> The venv module is included, > >>> however the pyvenv script is in a separate package > >>> python3.5-venv . > >>> > >>> By the way, I was totally confused by the following text form the doc. > >>> > >>> https://docs.python.org/3/library/venv.html > >>> > >>> ======== > >>> Deprecated since version 3.6: pyvenv was the recommended tool for > creating > >>> virtual environments for Python 3.3 and 3.4, and is deprecated in > Python > >>> 3.6. > >>> > >>> Changed in version 3.5: The use of venv is now recommended for creating > >>> virtual environments. > >>> > >>> ======== > >> > >> Not sure where you're reading that. I'm seeing: > >> > >> """ > >> Note > >> The pyvenv script has been deprecated as of Python 3.6 in favor of > >> using python3 -m venv to help prevent any potential confusion as to > >> which Python interpreter a virtual environment will be based on. > >> """ > >> > >> I think that's pretty clear. "python3 -m venv env" is the standard and > >> recommended way to spin up a virtual environment. > > > > It's further down in the page, under > > https://docs.python.org/3/library/venv.html#creating- > virtual-environments > > > > I think the deprecation notice for pyvenv should just be deleted, > > since it renders like the *module* is deprecated. > > Ah, I see it now, thanks. > > Agreed; or maybe downgrade it to a parenthetical comment. Focus on > "this is how to do the obvious thing", and only as an afterthought > mention "it used to be done differently" in case someone greps for > pyvenv. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Nov 13 03:14:11 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 13 Nov 2017 18:14:11 +1000 Subject: [Python-ideas] venv *is* provided in the standard Python install on Debian/Ubuntu In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> <20171112193851.3c24085f@fsol> Message-ID: On 13 November 2017 at 17:46, Stephan Houben wrote: > 2017-11-13 3:32 GMT+01:00 Nick Coghlan : >> So technically it's ensurepip that's broken by default, but that >> translates to venv also being broken by default. >> >> I haven't worked out what the actual steps needed to fix it are > > On Debian, ensurepip is in the python3.5-venv package. > > https://packages.debian.org/stretch/amd64/python3.5-venv/filelist Thanks! I've tweaked the ncoghlan/debian-python Dockerfile to install that in addition to the base Python package (as well as finally enabling automatic rebuilds of that image whenever the base Debian image updates). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Mon Nov 13 05:05:47 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 13 Nov 2017 11:05:47 +0100 Subject: [Python-ideas] venv *is* provided in the standard Python install on Debian/Ubuntu References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> <20171112193851.3c24085f@fsol> Message-ID: <20171113110547.224f8c19@fsol> On Mon, 13 Nov 2017 12:32:06 +1000 Nick Coghlan wrote: > On 13 November 2017 at 04:38, Antoine Pitrou wrote: > > On Sun, 12 Nov 2017 12:20:45 +0000 > > Paul Moore wrote: > >> > >> > Well, not exactly. Do you do python -m venv, or py -x.x -m venv or > >> > pythonx -m venv ? Wait, it's not installed by default on debian. > >> > >> Seriously? Debian don't provide venv in the standard Python install? > >> That's just broken. > > > > Frankly, I don't know where the current discussion comes from, but on > > two recent Debian and Ubuntu setups, I get: > > > > $ dpkg -S /usr/lib/python3.5/venv/__init__.py > > libpython3.5-stdlib:amd64: /usr/lib/python3.5/venv/__init__.py > > The discussion comes from the fact that even though the code is > present it doesn't actually work: [...] Wow. I had forgotten Debian could be such a user-hostile distribution. I'm not sure what the reason would be to use it as a basis for a training course in Python programming, then (other than the teacher having their own ideological preferences). Regards Antoine. From encukou at gmail.com Mon Nov 13 05:17:54 2017 From: encukou at gmail.com (Petr Viktorin) Date: Mon, 13 Nov 2017 11:17:54 +0100 Subject: [Python-ideas] venv *is* provided in the standard Python install on Debian/Ubuntu In-Reply-To: <20171113110547.224f8c19@fsol> References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> <20171112193851.3c24085f@fsol> <20171113110547.224f8c19@fsol> Message-ID: <7cb85345-af33-d88b-d4c6-f9714705a373@gmail.com> On 11/13/2017 11:05 AM, Antoine Pitrou wrote: > On Mon, 13 Nov 2017 12:32:06 +1000 > Nick Coghlan wrote: >> On 13 November 2017 at 04:38, Antoine Pitrou wrote: >>> On Sun, 12 Nov 2017 12:20:45 +0000 >>> Paul Moore wrote: >>>> >>>>> Well, not exactly. Do you do python -m venv, or py -x.x -m venv or >>>>> pythonx -m venv ? Wait, it's not installed by default on debian. >>>> >>>> Seriously? Debian don't provide venv in the standard Python install? >>>> That's just broken. >>> >>> Frankly, I don't know where the current discussion comes from, but on >>> two recent Debian and Ubuntu setups, I get: >>> >>> $ dpkg -S /usr/lib/python3.5/venv/__init__.py >>> libpython3.5-stdlib:amd64: /usr/lib/python3.5/venv/__init__.py >> >> The discussion comes from the fact that even though the code is >> present it doesn't actually work: > [...] > > Wow. I had forgotten Debian could be such a user-hostile > distribution. I'm not sure what the reason would be to use it as a > basis for a training course in Python programming, then (other than the > teacher having their own ideological preferences). For us, it's the *student's* preference. I believe it's better to let students use the machine and environment they're used to, even if it means extra trouble for the instructors. So, we get a healthy mix of Windows, Mac, Debian, Fedora, and sometimes some surprises. From solipsis at pitrou.net Mon Nov 13 05:43:20 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 13 Nov 2017 11:43:20 +0100 Subject: [Python-ideas] venv *is* provided in the standard Python install on Debian/Ubuntu References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> <20171112193851.3c24085f@fsol> <20171113110547.224f8c19@fsol> <7cb85345-af33-d88b-d4c6-f9714705a373@gmail.com> Message-ID: <20171113114320.7b43d676@fsol> On Mon, 13 Nov 2017 11:17:54 +0100 Petr Viktorin wrote: > > > > Wow. I had forgotten Debian could be such a user-hostile > > distribution. I'm not sure what the reason would be to use it as a > > basis for a training course in Python programming, then (other than the > > teacher having their own ideological preferences). > > For us, it's the *student's* preference. I believe it's better to let > students use the machine and environment they're used to, even if it > means extra trouble for the instructors. > So, we get a healthy mix of Windows, Mac, Debian, Fedora, and sometimes > some surprises. In that case the student must be ready to deal with the perils of their own preferences. Regards Antoine. From ncoghlan at gmail.com Mon Nov 13 05:53:04 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 13 Nov 2017 20:53:04 +1000 Subject: [Python-ideas] venv *is* provided in the standard Python install on Debian/Ubuntu In-Reply-To: <20171113114320.7b43d676@fsol> References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> <20171112193851.3c24085f@fsol> <20171113110547.224f8c19@fsol> <7cb85345-af33-d88b-d4c6-f9714705a373@gmail.com> <20171113114320.7b43d676@fsol> Message-ID: On 13 November 2017 at 20:43, Antoine Pitrou wrote: > On Mon, 13 Nov 2017 11:17:54 +0100 > Petr Viktorin wrote: >> > >> > Wow. I had forgotten Debian could be such a user-hostile >> > distribution. I'm not sure what the reason would be to use it as a >> > basis for a training course in Python programming, then (other than the >> > teacher having their own ideological preferences). >> >> For us, it's the *student's* preference. I believe it's better to let >> students use the machine and environment they're used to, even if it >> means extra trouble for the instructors. >> So, we get a healthy mix of Windows, Mac, Debian, Fedora, and sometimes >> some surprises. > > In that case the student must be ready to deal with the perils of their > own preferences. It's also currently missing from https://packaging.python.org/guides/installing-using-linux-tools/, but we can amend that now that we know what the required fix is. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From chris.barker at noaa.gov Mon Nov 13 13:30:43 2017 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 13 Nov 2017 10:30:43 -0800 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> <5A087DE7.4050306@brenbarn.net> Message-ID: On Sun, Nov 12, 2017 at 3:18 PM, Nick Coghlan wrote: > P.S. As a user, it's also genuinely irritating to have to always type > the "python -m " prefix when inside an active virtual environment, as > in that case, there isn't any ambiguity about which environment pip > should be manipulating or which version of Python it should be using. > and not just there :-) I decided a year or two ago to recommend "python -m pip install" to my beginning students, and put it all my course documentation. I've done this for a while now. but last week, when I introduced pytest -- I said, if you haven't' installed pytest yet, do it now -- and then proceeded to type: pip install pytest In my terminal, projecting in a huge font for all the class to see. It's just too ingrained and easy. And the truth is, it actually works MOST of the time. And if it doesn't work right, you are either a semi-sophisticated user that needs multiple pythons accessible from your shell, or your system is mis-configured. Pain though it is, it's probably better to help newbies configure their systems correctly than to try to get everyone to invoke pip differently. In fact, I have a little excercise that my students do as teh very first thing in class: https://uwpce-pythoncert.github.io/PythonCertDevel/supplemental/HowToRunAPythonFile.html#making-sure-you-are-set-up-correctly If you don't want to go read that: I give them this code: (not as a downloadable python file) #!/usr/bin/env python import sysprint("This is my first python program") version = sys.version_info if version.major == 3: if version.minor != 6: print("You should be running version 3.6") else: print("You are running python3.6 -- all good!")else: print("You need to run Python 3!") print("This is version: {}.{}".format(version.major, version.minor)) And I have them put it in a file, save it and run it however they want to run it. I then know that they: Know how to create a new python file, put some code in it and run that code. and They are running the version of Python I want them to run. (though to be fair, I had one student struggling for weeks with "strange" errors, because his PyCharm was setup to run python2. He probably started using PyCharm after he ran that script some other way...) I suppose I should do something similar to check that they have pip properly configured also, but to some extent, that comes up the first time I have them pip install something -- which is in the first class anyway :-) -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Mon Nov 13 13:57:08 2017 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 13 Nov 2017 10:57:08 -0800 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> <5A087DE7.4050306@brenbarn.net> Message-ID: This has gotten to be a big thread, and it's a pretty intractable problem, but I think there are a few fairly small things that could be done to at least make it a bit easier: 1) Add python2.exe and python3.exe files to the Windows installers -- am I insane or did Windows used to have that? I really think it did -- maybe got removed when py.exe was added. 1a) alternatively, we could add a "py" executable to the standard linux builds, so there would be THAT one way to do it. But I think that's a "BAD IDEA" -- the whole "py" thing is not widely know or used, it's not going to show up in package install instructions for a LONG time, (actualy we could do both anyway) Then "python2 -m pip install" would work everywhere (only with new installations, but at least with newbies, that's a bit more likely ...) 2) Make adding to the PATH in Windows the default. I think there are really three user groups: - newbies starting from scratch -- they want it on the PATH - newbies with whatever left over cruft from previous installations on their systems -- they want it at the FRONT of their PATH. They SHOULD uninstall all the cruft, but if they don't this will still work with as few surprises a possible. - not-newbies with a previous version of python they need to continue using. They can uncheck the box, or use py.exe 3) Make --user be be automatic for pip install. Not actually the default, but pip could do a user install if you don't have the permissions for a non-user install. This means folks might accidentally install in user mode because they forgot to type sudo -- but that would be a mostly-sysadmin/sophisticated user problem. And maybe have an environment variable of configuration key for "prefer admin install". If tha was set, pip would only install in user mode if specifically asked to. I'm can't imagine a case where a user would have admin permissions, but want a user install (except people following bad practices!) Except for the pip change, these would be easy to implement and backward compatible. So why not? *NOTE:* even if nothing changes with any of this we need to get py.exe better documented and advertised -- it doesn't show up in: https://docs.python.org/3/faq/windows.html#id2 for instance. In fact, I knew about py.exe (from this discussion), and was writing up notes about how to run a Python file (without access to a Windows box) , and it took a LONG time to find ANY documentation of it (adding "py" to a google search about something python does not get far...). We can do that better, but frankly this may be a lesson on why we can't rely on anything "new" to help solve this problem, when maybe we could make the "old way" work better and more cross platform. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Mon Nov 13 14:10:12 2017 From: barry at python.org (Barry Warsaw) Date: Mon, 13 Nov 2017 14:10:12 -0500 Subject: [Python-ideas] A proliferation of (un-)Pythonically programmatic pragmas Message-ID: I love many of the ancillary tools that help improve the quality of my Python code, including flake8, coverage, and mypy. Each of these usually produces some great diagnostics, but none of them are perfect, so they also produce false positives that have to be suppressed. Each of these tools supports "pragmas", i.e. comments you can add to a line of code to suppress a warning. E.g. with flake8 I can say: import readline, rlcompleter # noqa: F401 to suppress the flake8 warnings about unused imports. These modules work by side-effect, which of course a static analyzer can't know. Similarly, when mypy complains about a line it's confused on, I can add a comment to suppress the useless warning: try: from pathlib import Path, PurePath except ImportError: from pathlib2 import Path, PurePath # type: ignore And when I want to boost coverage, but I know that some lines aren't covered under some versions of Python, I can do something like: self.send_error(HTTPStatus.NOT_FOUND) # pragma: nocover These are all well and good until I have to *combine* suppressions. E.g. in the pathlib2 pragma to mypy, I also get a flake8 warning and I've tried just about every combination of pragma comment I can think of, but I've not been able to make both tools happy. I've resorted to refactoring the code into an entire module for flake8 to ignore and added: # flake8: noqa to suppress all warnings in the file. I actually have hit on a few places where I need to suppress warnings from all three tools on the same line, and I basically can't do it. The specifics aren't as important as the general use case: multiple tools competing for the same valuable real-estate. I have no ideas how to improve the situation, and of course any solution would involve some coordination between all of these tools, but it's beginning to feel like a losing battle. Is there a better way? Cheers, -Barry From kulakov.ilya at gmail.com Mon Nov 13 14:32:45 2017 From: kulakov.ilya at gmail.com (Ilya Kulakov) Date: Mon, 13 Nov 2017 11:32:45 -0800 Subject: [Python-ideas] Complete typing.Union with the rest of set-like functions Message-ID: I needed to declare a type that would mean "Any but None" and didn't find how. Current implementation allows to expand set of allowed types, not narrow it. Perhaps typing needs the rest of set operators in addition to Union? Best Regards, Ilya Kulakov -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Mon Nov 13 14:52:14 2017 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Mon, 13 Nov 2017 13:52:14 -0600 Subject: [Python-ideas] A proliferation of (un-)Pythonically programmatic pragmas In-Reply-To: References: Message-ID: Off the top of my head there's: - Offer a standard pragma-parser in the standard library's ast module that allows multiple pragmas per line. This is kind of the reason there's the weird ast vs typed_ast discrepancy right now, too. - Create a custom pragma syntax that doesn't use comments. Maybe something like: import stuff @[noqa: 'F401', type: 'ignore'] (Except nicer, because that's ugly as crud.) On Mon, Nov 13, 2017 at 1:10 PM, Barry Warsaw wrote: > I love many of the ancillary tools that help improve the quality of my > Python code, including flake8, coverage, and mypy. Each of these > usually produces some great diagnostics, but none of them are perfect, > so they also produce false positives that have to be suppressed. > > Each of these tools supports "pragmas", i.e. comments you can add to a > line of code to suppress a warning. E.g. with flake8 I can say: > > import readline, rlcompleter # noqa: F401 > > to suppress the flake8 warnings about unused imports. These modules > work by side-effect, which of course a static analyzer can't know. > > Similarly, when mypy complains about a line it's confused on, I can add > a comment to suppress the useless warning: > > try: > from pathlib import Path, PurePath > except ImportError: > from pathlib2 import Path, PurePath # type: ignore > > And when I want to boost coverage, but I know that some lines aren't > covered under some versions of Python, I can do something like: > > self.send_error(HTTPStatus.NOT_FOUND) # pragma: nocover > > These are all well and good until I have to *combine* suppressions. > E.g. in the pathlib2 pragma to mypy, I also get a flake8 warning and > I've tried just about every combination of pragma comment I can think > of, but I've not been able to make both tools happy. I've resorted to > refactoring the code into an entire module for flake8 to ignore and added: > > # flake8: noqa > > to suppress all warnings in the file. I actually have hit on a few > places where I need to suppress warnings from all three tools on the > same line, and I basically can't do it. > > The specifics aren't as important as the general use case: multiple > tools competing for the same valuable real-estate. > > I have no ideas how to improve the situation, and of course any solution > would involve some coordination between all of these tools, but it's > beginning to feel like a losing battle. Is there a better way? > > Cheers, > -Barry > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- Ryan (????) Yoko Shimomura, ryo (supercell/EGOIST), Hiroyuki Sawano >> everyone else https://refi64.com/ From levkivskyi at gmail.com Mon Nov 13 15:01:35 2017 From: levkivskyi at gmail.com (Ivan Levkivskyi) Date: Mon, 13 Nov 2017 21:01:35 +0100 Subject: [Python-ideas] Complete typing.Union with the rest of set-like functions In-Reply-To: References: Message-ID: Intersection has already been proposed, see https://github.com/python/typing/issues/213 But it is not yet implemented. You can show your use cases on the typing issue tracker, maybe they can be covered by Intersection (which will be most probably added at some point). -- Ivan On 13 November 2017 at 20:32, Ilya Kulakov wrote: > I needed to declare a type that would mean "Any but None" and didn't find > how. > > Current implementation allows to expand set of allowed types, not narrow > it. > Perhaps typing needs the rest of set operators in addition to Union? > > > Best Regards, > Ilya Kulakov > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephanh42 at gmail.com Mon Nov 13 15:28:56 2017 From: stephanh42 at gmail.com (Stephan Houben) Date: Mon, 13 Nov 2017 21:28:56 +0100 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> <5A087DE7.4050306@brenbarn.net> Message-ID: Hi Chris, +1 to all three of these proposals! Stephan 2017-11-13 19:57 GMT+01:00 Chris Barker : > This has gotten to be a big thread, and it's a pretty intractable problem, > but I think there are a few fairly small things that could be done to at > least make it a bit easier: > > 1) Add python2.exe and python3.exe files to the Windows installers -- am I > insane or did Windows used to have that? I really think it did -- maybe got > removed when py.exe was added. > 1a) alternatively, we could add a "py" executable to the standard linux > builds, so there would be THAT one way to do it. But I think that's a "BAD > IDEA" -- the whole "py" thing is not widely know or used, it's not going to > show up in package install instructions for a LONG time, (actualy we could > do both anyway) > > Then "python2 -m pip install" would work everywhere (only with new > installations, but at least with newbies, that's a bit more likely ...) > > > 2) Make adding to the PATH in Windows the default. I think there are > really three user groups: > > - newbies starting from scratch -- they want it on the PATH > > - newbies with whatever left over cruft from previous installations on > their systems -- they want it at the FRONT of their PATH. They SHOULD > uninstall all the cruft, but if they don't this will still work with as few > surprises a possible. > > - not-newbies with a previous version of python they need to continue > using. They can uncheck the box, or use py.exe > > > 3) Make --user be be automatic for pip install. Not actually the default, > but pip could do a user install if you don't have the permissions for a > non-user install. > > This means folks might accidentally install in user mode because they > forgot to type sudo -- but that would be a mostly-sysadmin/sophisticated > user problem. And maybe have an environment variable of configuration key > for "prefer admin install". If tha was set, pip would only install in user > mode if specifically asked to. I'm can't imagine a case where a user would > have admin permissions, but want a user install (except people following > bad practices!) > > Except for the pip change, these would be easy to implement and backward > compatible. So why not? > > > *NOTE:* even if nothing changes with any of this we need to get py.exe > better documented and advertised -- it doesn't show up in: > > https://docs.python.org/3/faq/windows.html#id2 > > for instance. > > In fact, I knew about py.exe (from this discussion), and was writing up > notes about how to run a Python file (without access to a Windows box) , > and it took a LONG time to find ANY documentation of it (adding "py" to a > google search about something python does not get far...). > > We can do that better, but frankly this may be a lesson on why we can't > rely on anything "new" to help solve this problem, when maybe we could make > the "old way" work better and more cross platform. > > -Chris > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Mon Nov 13 15:43:42 2017 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 13 Nov 2017 20:43:42 +0000 Subject: [Python-ideas] A proliferation of (un-)Pythonically programmatic pragmas In-Reply-To: References: Message-ID: On 2017-11-13 19:10, Barry Warsaw wrote: > I love many of the ancillary tools that help improve the quality of my > Python code, including flake8, coverage, and mypy. Each of these > usually produces some great diagnostics, but none of them are perfect, > so they also produce false positives that have to be suppressed. > > Each of these tools supports "pragmas", i.e. comments you can add to a > line of code to suppress a warning. E.g. with flake8 I can say: > > import readline, rlcompleter # noqa: F401 > > to suppress the flake8 warnings about unused imports. These modules > work by side-effect, which of course a static analyzer can't know. > > Similarly, when mypy complains about a line it's confused on, I can add > a comment to suppress the useless warning: > > try: > from pathlib import Path, PurePath > except ImportError: > from pathlib2 import Path, PurePath # type: ignore > > And when I want to boost coverage, but I know that some lines aren't > covered under some versions of Python, I can do something like: > > self.send_error(HTTPStatus.NOT_FOUND) # pragma: nocover > > These are all well and good until I have to *combine* suppressions. > E.g. in the pathlib2 pragma to mypy, I also get a flake8 warning and > I've tried just about every combination of pragma comment I can think > of, but I've not been able to make both tools happy. I've resorted to > refactoring the code into an entire module for flake8 to ignore and added: > > # flake8: noqa > > to suppress all warnings in the file. I actually have hit on a few > places where I need to suppress warnings from all three tools on the > same line, and I basically can't do it. > > The specifics aren't as important as the general use case: multiple > tools competing for the same valuable real-estate. > > I have no ideas how to improve the situation, and of course any solution > would involve some coordination between all of these tools, but it's > beginning to feel like a losing battle. Is there a better way? > I suppose that you could have suggest to them that they follow a convention such as: 1. There can be multiple pragmas in a comment, separated by semicolons: if you don't recognise it, skip past the semicolon. 2. A pragma can be prefixed with the name of the tool, e.g. "# flake8.noqa: F401": if there's a prefix, but it's not yours, skip past the semicolon. From p.f.moore at gmail.com Mon Nov 13 16:52:58 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 13 Nov 2017 21:52:58 +0000 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> <5A087DE7.4050306@brenbarn.net> Message-ID: On 13 November 2017 at 18:57, Chris Barker wrote: > This has gotten to be a big thread, and it's a pretty intractable problem, > but I think there are a few fairly small things that could be done to at > least make it a bit easier: In principle, I agree with the ideas here, but there are some practical issues that make them somewhat less straightforward than we might like. > 1) Add python2.exe and python3.exe files to the Windows installers -- am I > insane or did Windows used to have that? I really think it did -- maybe got > removed when py.exe was added. I don't think Windows ever had python2.exe/python3.exe, but I could be wrong. The only possible way we'd get a python2.exe is by adding it to future releases of Python 2.7, and we shouldn't be recommending Python 2 for new users anyway. And I'm strongly -1 on promoting "python3.exe" as the canonical way of getting Python 3. The python3 command is a result of the Unix issues around switching to Python 3 as the default, and we shouldn't perpetuate that approach on Windows, where it's unneeded. Having said that, I don't object to including versioned executables as a convenience (if it is one). I just dislike promoting the use of them via standard documentation. > 1a) alternatively, we could add a "py" executable to the standard linux > builds, so there would be THAT one way to do it. But I think that's a "BAD > IDEA" -- the whole "py" thing is not widely know or used, it's not going to > show up in package install instructions for a LONG time, (actualy we could > do both anyway) I think that getting a "py" launcher on Unix is a lost cause at this stage. It would be nice, and maybe if the original PEP had proposed a cross-platform py command, it might have worked, but that's history now, and I think the py launcher will probably always be a Windows-only thing. > Then "python2 -m pip install" would work everywhere (only with new > installations, but at least with newbies, that's a bit more likely ...) No newcomer should *ever* be getting told to use "python2 -m pip install". Maybe "python3 -m pip install", but never Python 2 (outside of specialised environments where training people in an out of date version is required). And of course not all Unix distributions come with Python 3 installed (Mac OS being an obvious example, and I think Nick mentioned CentOS and RHEL) so python3 won't work everywhere either... > 2) Make adding to the PATH in Windows the default. I think there are really > three user groups: > > - newbies starting from scratch -- they want it on the PATH > > - newbies with whatever left over cruft from previous installations on > their systems -- they want it at the FRONT of their PATH. They SHOULD > uninstall all the cruft, but if they don't this will still work with as few > surprises a possible. > > - not-newbies with a previous version of python they need to continue > using. They can uncheck the box, or use py.exe Unfortunately, adding Python to the *front* of PATH is not possible. It might be the best option, but it simply cannot be done. A per-user install of Python (the default in Python 3.5+) does not have the privileges to add items to the front of the system PATH, so the best we can do is add it to the front of the user PATH. But the system PATH always goes ahead of the user PATH, and system installs (for example, default installs of Python 2.x or 3.4 or earlier) will come before that. (Making a system install of Python be the default won't work either, as then pip won't work from a normal prompt and we have the same issues on Windows that Unix users have that results in people doing "sudo pip install" with all the issues that brings). So for (1) I agree. For (2) it simply isn't possible. For (3) you're getting beyond newcomer so sure, "manually handle it" is a reasonable option. There's also the problem that file associations (i.e., what does double clicking on a .py/.pyw file do) don't follow the same rules, as they go through the launcher. The only really sensible "do what I expect" situation is a single Python 3 installation on the user's machine. For that case, adding Python to PATH (it doesn't matter where as there's only one) is a sensible option. The downside is that optimising for that case makes the behaviour for other (more experienced) users worse. But it's not unreasonable to make that trade-off. > 3) Make --user be be automatic for pip install. Not actually the default, > but pip could do a user install if you don't have the permissions for a > non-user install. The problem here is that the user scripts directory isn't on PATH. We could put it on, but again we hit the "goes after the system PATH" problem (on Windows). > This means folks might accidentally install in user mode because they forgot > to type sudo -- but that would be a mostly-sysadmin/sophisticated user > problem. And maybe have an environment variable of configuration key for > "prefer admin install". If tha was set, pip would only install in user mode > if specifically asked to. I'm can't imagine a case where a user would have > admin permissions, but want a user install (except people following bad > practices!) The various options here are under discussion on the pip tracker. Suffice it to say, it's not a clear-cut change, but it would ideally be something we do. Nobody really objects to it, but there are significant backward compatibility and behaviour change issues to address first. > Except for the pip change, these would be easy to implement and backward > compatible. So why not? See above. But in principle, if the issues can be solved in a way we're all happy with, I agree with the principle that "download and install Python" should result in an environment where python myscript.py pip install something_or_other "just work" if the user previously had no Python on their PC. Anything less results in a worse beginner experience for Python than for other languages (which usually don't support having multiple versions installed at once, and often require admin installs, so bypass most of the problems Python has to deal with). > NOTE: even if nothing changes with any of this we need to get py.exe better > documented and advertised -- it doesn't show up in: > > https://docs.python.org/3/faq/windows.html#id2 > > for instance. It has a whole section at https://docs.python.org/3/using/windows.html#python-launcher-for-windows. And https://docs.python.org/3/using/windows.html discusses how to handle PATH, so that's clearly documented too, even if (a) no-one reads the docs, and (b) the defaults may not be the best choices. Agreed that https://docs.python.org/3/faq/windows.html seems pretty out of date, and could do with an update. I'm sure patches would be gratefully accepted, if only to replace the out of date information with pointers to the correct places in the main documentation. Ultimately, though, most newcomers don't read the docs. They muddle through, ask colleagues who may or may not know any better than they do, read out of date articles on Stack Overflow and watch videos on Youtube. The *only* way of stopping them doing that is to make everything "just work" for them. And when one person can tell them to download Python from python.org, another says "use Anaconda" and a third suggests ActiveState, and they try all three while trying to work out what they need, sometimes we just have to do the best we can :-) Paul From p.f.moore at gmail.com Mon Nov 13 16:58:56 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 13 Nov 2017 21:58:56 +0000 Subject: [Python-ideas] A proliferation of (un-)Pythonically programmatic pragmas In-Reply-To: References: Message-ID: On 13 November 2017 at 20:43, MRAB wrote: > On 2017-11-13 19:10, Barry Warsaw wrote: >> The specifics aren't as important as the general use case: multiple >> tools competing for the same valuable real-estate. >> >> I have no ideas how to improve the situation, and of course any solution >> would involve some coordination between all of these tools, but it's >> beginning to feel like a losing battle. Is there a better way? >> > I suppose that you could have suggest to them that they follow a convention > such as: > > 1. There can be multiple pragmas in a comment, separated by semicolons: if > you don't recognise it, skip past the semicolon. > > 2. A pragma can be prefixed with the name of the tool, e.g. "# flake8.noqa: > F401": if there's a prefix, but it's not yours, skip past the semicolon. An informational PEP defining a common convention for pragma-style comments could standardise things. I'd suggest starting a discussion (somewhere?) with the development teams for the relevant projects (flake8, mypy, coverage...) with the intention of developing such a PEP that they could all support. Paul From brett at python.org Mon Nov 13 17:18:13 2017 From: brett at python.org (Brett Cannon) Date: Mon, 13 Nov 2017 22:18:13 +0000 Subject: [Python-ideas] A proliferation of (un-)Pythonically programmatic pragmas In-Reply-To: References: Message-ID: On Mon, Nov 13, 2017, 13:59 Paul Moore, wrote: > On 13 November 2017 at 20:43, MRAB wrote: > > On 2017-11-13 19:10, Barry Warsaw wrote: > >> The specifics aren't as important as the general use case: multiple > >> tools competing for the same valuable real-estate. > >> > >> I have no ideas how to improve the situation, and of course any solution > >> would involve some coordination between all of these tools, but it's > >> beginning to feel like a losing battle. Is there a better way? > >> > > I suppose that you could have suggest to them that they follow a > convention > > such as: > > > > 1. There can be multiple pragmas in a comment, separated by semicolons: > if > > you don't recognise it, skip past the semicolon. > > > > 2. A pragma can be prefixed with the name of the tool, e.g. "# > flake8.noqa: > > F401": if there's a prefix, but it's not yours, skip past the semicolon. > > An informational PEP defining a common convention for pragma-style > comments could standardise things. I'd suggest starting a discussion > (somewhere?) with the development teams for the relevant projects > (flake8, mypy, coverage...) with the intention of developing such a > PEP that they could all support. > And possibly the easiest way to reach them is on the pyqa-dev mailing list. -brett > Paul > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg at krypto.org Mon Nov 13 18:31:13 2017 From: greg at krypto.org (Gregory P. Smith) Date: Mon, 13 Nov 2017 23:31:13 +0000 Subject: [Python-ideas] A proliferation of (un-)Pythonically programmatic pragmas In-Reply-To: References: Message-ID: fwiw, we're going to need the tool name in any pragma anyways so the existing thing that should be common is: # tool-name: meaningfultoken It seems like the only convention that makes sense to me. When I saw your flake8 example of "# noqa: F401" I wanted to rip my eyes out. Because it didn't mention the tool name *and* it used a numeric code. Tool authors: use descriptive names! Otherwise it is meaningless to anyone reading it. ex: import rlcompleter # pylint: disable=unused-import Nobody will ever question _what_ that means. -gps On Mon, Nov 13, 2017 at 2:19 PM Brett Cannon wrote: > > > On Mon, Nov 13, 2017, 13:59 Paul Moore, wrote: > >> On 13 November 2017 at 20:43, MRAB wrote: >> > On 2017-11-13 19:10, Barry Warsaw wrote: >> >> The specifics aren't as important as the general use case: multiple >> >> tools competing for the same valuable real-estate. >> >> >> >> I have no ideas how to improve the situation, and of course any >> solution >> >> would involve some coordination between all of these tools, but it's >> >> beginning to feel like a losing battle. Is there a better way? >> >> >> > I suppose that you could have suggest to them that they follow a >> convention >> > such as: >> > >> > 1. There can be multiple pragmas in a comment, separated by semicolons: >> if >> > you don't recognise it, skip past the semicolon. >> > >> > 2. A pragma can be prefixed with the name of the tool, e.g. "# >> flake8.noqa: >> > F401": if there's a prefix, but it's not yours, skip past the semicolon. >> >> An informational PEP defining a common convention for pragma-style >> comments could standardise things. I'd suggest starting a discussion >> (somewhere?) with the development teams for the relevant projects >> (flake8, mypy, coverage...) with the intention of developing such a >> PEP that they could all support. >> > > And possibly the easiest way to reach them is on the pyqa-dev mailing list. > > -brett > > >> Paul >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Mon Nov 13 18:37:05 2017 From: barry at python.org (Barry Warsaw) Date: Mon, 13 Nov 2017 18:37:05 -0500 Subject: [Python-ideas] A proliferation of (un-)Pythonically programmatic pragmas In-Reply-To: References: Message-ID: Brett Cannon wrote: > And possibly the easiest way to reach them is on the pyqa-dev mailing list. What's that? I can't find it on python.org, Gmane, or the Googles. -Barry From guido at python.org Mon Nov 13 18:41:09 2017 From: guido at python.org (Guido van Rossum) Date: Mon, 13 Nov 2017 15:41:09 -0800 Subject: [Python-ideas] PEP 560 (second post) In-Reply-To: References: Message-ID: On Fri, Nov 10, 2017 at 8:54 AM, Ivan Levkivskyi wrote: > On 10 November 2017 at 17:43, Guido van Rossum wrote: > >> There seem to be some action items from this thread that I haven't seen >> reflected in the PEP source code yet. >> [...snip...] >> Then the next step I propose is a PR with a full implementation. After >> that I'll likely approve the PEP (or we'll have minor feedback based on >> trying the implementation). >> > > Yes, sorry, I wanted to make updates to the PEP and reference > implementation, > but last two weeks were very busy. > Hopefully, I will work on it this weekend. > Thanks, I am happy now with the PEP, except for one detail: maybe `__mro_entry__` should always return a tuple and then maybe renamed to `__mro_entries__`. (See debate at https://github.com/python/peps/pull/460#issuecomment-343969528 .) Other than that I think we just need to satisfy a process nit: let's post the final PEP (after that issue is resolved) to python-dev. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon Nov 13 18:48:31 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 14 Nov 2017 10:48:31 +1100 Subject: [Python-ideas] A proliferation of (un-)Pythonically programmatic pragmas In-Reply-To: References: Message-ID: <20171113234830.GK19802@ando.pearwood.info> On Mon, Nov 13, 2017 at 06:37:05PM -0500, Barry Warsaw wrote: > Brett Cannon wrote: > > > And possibly the easiest way to reach them is on the pyqa-dev mailing list. > > What's that? I can't find it on python.org, Gmane, or the Googles. Brett may have meant https://mail.python.org/mailman/listinfo/code-quality -- Steve From tjreedy at udel.edu Mon Nov 13 19:17:26 2017 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 13 Nov 2017 19:17:26 -0500 Subject: [Python-ideas] A proliferation of (un-)Pythonically programmatic pragmas In-Reply-To: References: Message-ID: On 11/13/2017 4:58 PM, Paul Moore wrote: > On 13 November 2017 at 20:43, MRAB > such as: >> >> 1. There can be multiple pragmas in a comment, separated by semicolons: if >> you don't recognise it, skip past the semicolon. >> >> 2. A pragma can be prefixed with the name of the tool, e.g. "# flake8.noqa: >> F401": if there's a prefix, but it's not yours, skip past the semicolon. > > An informational PEP defining a common convention for pragma-style > comments could standardise things. I'd suggest starting a discussion > (somewhere?) with the development teams for the relevant projects > (flake8, mypy, coverage...) with the intention of developing such a > PEP that they could all support. This seems like a good subject for an informational PEP. It could and I think should include an implementation of the convention as a generator function with parameter 'tool' that yields (option, value) pairs for that tool. Docstring directives might be considered for points to copy or not copy. An example is # doctest: +ELLIPSIS, +NORMALIZE_WHITESPACE '.' instead of ': ' to separate the tool prefix seems better for multiple tools, except that the latter works better for omitting the tool prefix when there are multiple options. For binary options, '+' and '-' prefixes are much more compact than '=True' and '=False' suffixes. This will be more important with multiple directives in one comment. -- Terry Jan Reedy From brett at python.org Mon Nov 13 19:27:15 2017 From: brett at python.org (Brett Cannon) Date: Tue, 14 Nov 2017 00:27:15 +0000 Subject: [Python-ideas] A proliferation of (un-)Pythonically programmatic pragmas In-Reply-To: <20171113234830.GK19802@ando.pearwood.info> References: <20171113234830.GK19802@ando.pearwood.info> Message-ID: On Mon, Nov 13, 2017, 15:55 Steven D'Aprano, wrote: > On Mon, Nov 13, 2017 at 06:37:05PM -0500, Barry Warsaw wrote: > > Brett Cannon wrote: > > > > > And possibly the easiest way to reach them is on the pyqa-dev mailing > list. > > > > What's that? I can't find it on python.org, Gmane, or the Googles. > > Brett may have meant > > https://mail.python.org/mailman/listinfo/code-quality Steve's right. http://meta.pycqa.org/en/latest/ -brett > > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ned at nedbatchelder.com Mon Nov 13 19:50:25 2017 From: ned at nedbatchelder.com (Ned Batchelder) Date: Mon, 13 Nov 2017 19:50:25 -0500 Subject: [Python-ideas] A proliferation of (un-)Pythonically programmatic pragmas In-Reply-To: References: Message-ID: On 11/13/17 2:10 PM, Barry Warsaw wrote: > I love many of the ancillary tools that help improve the quality of my > Python code, including flake8, coverage, and mypy. Each of these > usually produces some great diagnostics, but none of them are perfect, > so they also produce false positives that have to be suppressed. > > Each of these tools supports "pragmas", i.e. comments you can add to a > line of code to suppress a warning. E.g. with flake8 I can say: > > import readline, rlcompleter # noqa: F401 > > to suppress the flake8 warnings about unused imports. These modules > work by side-effect, which of course a static analyzer can't know. > > Similarly, when mypy complains about a line it's confused on, I can add > a comment to suppress the useless warning: > > try: > from pathlib import Path, PurePath > except ImportError: > from pathlib2 import Path, PurePath # type: ignore > > And when I want to boost coverage, but I know that some lines aren't > covered under some versions of Python, I can do something like: > > self.send_error(HTTPStatus.NOT_FOUND) # pragma: nocover > > These are all well and good until I have to *combine* suppressions. > E.g. in the pathlib2 pragma to mypy, I also get a flake8 warning and > I've tried just about every combination of pragma comment I can think > of, but I've not been able to make both tools happy. I've resorted to > refactoring the code into an entire module for flake8 to ignore and added: > > # flake8: noqa > > to suppress all warnings in the file. I actually have hit on a few > places where I need to suppress warnings from all three tools on the > same line, and I basically can't do it. > > The specifics aren't as important as the general use case: multiple > tools competing for the same valuable real-estate. > > I have no ideas how to improve the situation, and of course any solution > would involve some coordination between all of these tools, but it's > beginning to feel like a losing battle. Is there a better way? > > Coverage.py has no fixed syntax for its pragmas.? The default is "# pragma: no cover", but you can configure any regex you like, and it's matched against the entire line, not just the comment.? This makes it possible to exclude lines based on the source, or to change the syntax of the pragma. If a PEP is written with a common syntax, I can add it to coverage.py's default. --Ned. From brett at python.org Mon Nov 13 20:31:06 2017 From: brett at python.org (Brett Cannon) Date: Tue, 14 Nov 2017 01:31:06 +0000 Subject: [Python-ideas] venv *is* provided in the standard Python install on Debian/Ubuntu In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> <20171112193851.3c24085f@fsol> Message-ID: On Mon, Nov 13, 2017, 00:01 Stephan Houben, wrote: > Hi all, > > Related to this text on https://docs.python.org/3/library/venv.html : > > ============ > > Note > The pyvenv script has been deprecated as of Python 3.6 in favor of > using python3 -m venv to help prevent any potential confusion as to > which Python interpreter a virtual environment will be based on. > ============ > > It's clearer than the text below to which I originally referred. > > However, this text has also problems in that it is too unix-specific. > In particular: > * Most seriously, it refers to "python3" which doesn't work with the > python.org Windows installer. > It can, but it's opt-in. It's just one of those things that's easy to forget. * Less seriously, it refers to "pyenv" as a "script" which is unix jargon > and moreover technically > incorrect on Windows. (Also, needlessly specific, it should just be > "the pyenv command", > how it is implemented is irrelevant for this section). > I disagree with this as Python refers to .Py files that you execute directly as "scripts", so I don't think this requires clarification. Anyway, a pull request with suggested wording to address your concerns would be the best way to try and rectify the issue. -brett > Stephan > > 2017-11-13 0:32 GMT+01:00 Chris Angelico : > >> On Mon, Nov 13, 2017 at 10:29 AM, Nick Coghlan >> wrote: >> > On 13 November 2017 at 07:11, Chris Angelico wrote: >> >> On Mon, Nov 13, 2017 at 6:24 AM, Stephan Houben >> wrote: >> >>> Hi Antoine, >> >>> >> >>> The venv module is included, >> >>> however the pyvenv script is in a separate package >> >>> python3.5-venv . >> >>> >> >>> By the way, I was totally confused by the following text form the doc. >> >>> >> >>> https://docs.python.org/3/library/venv.html >> >>> >> >>> ======== >> >>> Deprecated since version 3.6: pyvenv was the recommended tool for >> creating >> >>> virtual environments for Python 3.3 and 3.4, and is deprecated in >> Python >> >>> 3.6. >> >>> >> >>> Changed in version 3.5: The use of venv is now recommended for >> creating >> >>> virtual environments. >> >>> >> >>> ======== >> >> >> >> Not sure where you're reading that. I'm seeing: >> >> >> >> """ >> >> Note >> >> The pyvenv script has been deprecated as of Python 3.6 in favor of >> >> using python3 -m venv to help prevent any potential confusion as to >> >> which Python interpreter a virtual environment will be based on. >> >> """ >> >> >> >> I think that's pretty clear. "python3 -m venv env" is the standard and >> >> recommended way to spin up a virtual environment. >> > >> > It's further down in the page, under >> > >> https://docs.python.org/3/library/venv.html#creating-virtual-environments >> > >> > I think the deprecation notice for pyvenv should just be deleted, >> > since it renders like the *module* is deprecated. >> >> Ah, I see it now, thanks. >> >> Agreed; or maybe downgrade it to a parenthetical comment. Focus on >> "this is how to do the obvious thing", and only as an afterthought >> mention "it used to be done differently" in case someone greps for >> pyvenv. >> >> ChrisA >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Mon Nov 13 20:37:50 2017 From: barry at python.org (Barry Warsaw) Date: Mon, 13 Nov 2017 20:37:50 -0500 Subject: [Python-ideas] A proliferation of (un-)Pythonically programmatic pragmas In-Reply-To: References: Message-ID: Gregory P. Smith wrote: > fwiw, we're going to need the tool name in any pragma anyways so the > existing thing that should be common is: > > # tool-name: meaningfultoken > > It seems like the only convention that makes sense to me. One of the things that bother me about end-line comments is that this is going to blow up line length limits. I think this could work if such pragma comments could apply to the following line, and multiline pragmas would be acceptable. Then you could have something like: # flake8: disable=unused-import # mypy: alias=pathlib2.Path # coverage: ignore=when>py2.7 > When I saw your flake8 example of "# noqa: F401" I wanted to rip my eyes > out. Because it didn't mention the tool name *and* it used a numeric code. > Tool authors: use descriptive names! Otherwise it is meaningless to anyone > reading it. ex: Hah. One of the things I never imagined was that folks would throw around numeric PEP numbers and just expect everyone to have PEP 0 tattooed to the back of their eyelids. Quick, let's discuss PEP 3128! Cheers, -Barry From barry at python.org Mon Nov 13 20:40:07 2017 From: barry at python.org (Barry Warsaw) Date: Mon, 13 Nov 2017 20:40:07 -0500 Subject: [Python-ideas] A proliferation of (un-)Pythonically programmatic pragmas In-Reply-To: <20171113234830.GK19802@ando.pearwood.info> References: <20171113234830.GK19802@ando.pearwood.info> Message-ID: Steven D'Aprano wrote: > > Brett may have meant > > https://mail.python.org/mailman/listinfo/code-quality That definitely makes more sense. :) -Barry From njs at pobox.com Mon Nov 13 20:51:42 2017 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 13 Nov 2017 17:51:42 -0800 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: <5A087DE7.4050306@brenbarn.net> References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> <5A087DE7.4050306@brenbarn.net> Message-ID: On Sun, Nov 12, 2017 at 8:59 AM, Brendan Barnwell wrote: > On 2017-11-12 05:18, Nick Coghlan wrote: >> >> * the `pip install` option really is nicer looking than `python -m pip >> install`, and it only has actual problems in the presence of multiple >> Python versions and when upgrading pip itself on Windows (plus: lots >> of third party guides recommend it, as do pypi.org project pages) > > Is there any *advantage* to using `pip install` instead of `python > -m install`? If not, could we at least change everything under Python/pip > control (e.g., pip documentation) to never recommend `pip` and always > recommend `python -m pip` instead, and encourage all third-party > documentation to always use `python -m pip` and never use `pip`? Obviously > this isn't a full solution, but in the end there's no way we change external > third-party documentation, which will always eventually become outdated. > Absent that, it seems worthwhile to regularize existing official > documentation. Can we instead make it so that 'pip' and 'python -m pip' *are* actually equivalent? I know there are all kinds of pathological things that can happen, but it seems like we can drive the frequency of this error down more. What if instead of installing a standard entry point, the pip executable was installed as #!/bin/sh exec python -m pip "$@" on Unix-likes, and a pip.bat with the equivalent contents on Windows? (Bonus: maybe this would fix the problem with upgrading pip on Windows?) -n -- Nathaniel J. Smith -- https://vorpus.org From ncoghlan at gmail.com Mon Nov 13 21:14:50 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 14 Nov 2017 12:14:50 +1000 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> <5A087DE7.4050306@brenbarn.net> Message-ID: On 14 November 2017 at 07:52, Paul Moore wrote: > On 13 November 2017 at 18:57, Chris Barker wrote: >> This has gotten to be a big thread, and it's a pretty intractable problem, >> but I think there are a few fairly small things that could be done to at >> least make it a bit easier: > > In principle, I agree with the ideas here, but there are some > practical issues that make them somewhat less straightforward than we > might like. > >> 1) Add python2.exe and python3.exe files to the Windows installers -- am I >> insane or did Windows used to have that? I really think it did -- maybe got >> removed when py.exe was added. > > I don't think Windows ever had python2.exe/python3.exe, but I could be wrong. Not that I'm aware of in the python.org installers, and I don't think ActivePython does either. I'm less sure about Enthought or Anaconda (since I've never used either of them on Windows). > The only possible way we'd get a python2.exe is by adding it to future > releases of Python 2.7, and we shouldn't be recommending Python 2 for > new users anyway. And I'm strongly -1 on promoting "python3.exe" as > the canonical way of getting Python 3. The python3 command is a result > of the Unix issues around switching to Python 3 as the default, and we > shouldn't perpetuate that approach on Windows, where it's unneeded. Right, we actually want to get to a state where "python" as a symlink to "python3" is a supported system configuration, but we need to go through a period of it not linking to anything at all first (to ensure that anyone that currently needs Python 2 has a chance to either switch to explicitly calling python2 instead, or else update their software to be Python 3 compatible). While the specifics are out of date (due to changes we've made in the past few weeks while working on the F27 Modular Server release), https://fedora-python.readthedocs.io/en/latest/plans/default-python-module/ sketches out a possible approach to getting Fedora there by the end of 2018 (in Fedora 29, as part of the switch to CPython 3.7). The other reason we don't want to permanently entrench `python3` as a preferred command is that we assume that Python 4.x will eventually happen in some form or another, and we'd like to get to a point where `pythonX` is how you request a particular *old* version of Python, with the unqualified `python` typically meaning the latest available version. > Having said that, I don't object to including versioned executables as > a convenience (if it is one). I just dislike promoting the use of them > via standard documentation. > >> 1a) alternatively, we could add a "py" executable to the standard linux >> builds, so there would be THAT one way to do it. But I think that's a "BAD >> IDEA" -- the whole "py" thing is not widely know or used, it's not going to >> show up in package install instructions for a LONG time, (actualy we could >> do both anyway) > > I think that getting a "py" launcher on Unix is a lost cause at this > stage. It would be nice, and maybe if the original PEP had proposed a > cross-platform py command, it might have worked, but that's history > now, and I think the py launcher will probably always be a > Windows-only thing. Yeah, myself, Barry Warsaw, Matthias Klose, and a number of other folks on linux-sig have poked at this idea multiple times since Geoff Thomas first posted about https://github.com/geofft/pythonmux, but our conclusion each time has been that it wouldn't help enough to justify the effort involved in implementing and promoting it. After 20+ years of usage in the Linux ecosystem, `/usr/bin/python` and `/usr/bin/env python` are simply too entrenched in both people's habits and existing code. Hence the approach the mainstream distros have ended up taking: break references to `python` in our default installs, offer a straightforward way to get that back as a reference to Python 2.7, and then start trying to figure out a supportable mechanism that will allow system operators to opt in to having it point to Python 3.x instead. Ultimately, this isn't that different from what Arch did back in 2011, just much later in the 3.x adoption cycle, and with a simpler and more obvious forced breakage (a missing command), rather than the potentially cryptic errors emitted when attempting to run Python 2 code on Python 3. A couple of other points worth noting here: * https://github.com/python/redistributor-guide/issues/1 which links to https://github.com/python/peps/pull/315/files, which was a draft replacement for PEP 394 that we never actually merged. We know external visibility into these distro-level discussions is currently poor, and doing something about that is on the todo list, we(/I) just haven't gotten to it yet * even on Windows, the main intended purpose of the `py` launcher was to more reliably handle file associations by introducing an equivalent to POSIX shebang line processing. The interactive command line use case was more of an added bonus than anything else. >> Then "python2 -m pip install" would work everywhere (only with new >> installations, but at least with newbies, that's a bit more likely ...) > > No newcomer should *ever* be getting told to use "python2 -m pip > install". Maybe "python3 -m pip install", but never Python 2 (outside > of specialised environments where training people in an out of date > version is required). And of course not all Unix distributions come > with Python 3 installed (Mac OS being an obvious example, and I think > Nick mentioned CentOS and RHEL) so python3 won't work everywhere > either... Right, getting a regular Fedora-style python3 install on RHEL/CentOS currently requires enabling the IUS community repos, as per the last option in https://packaging.python.org/guides/installing-using-linux-tools/#centos-rhel We do offer python3-by-default as part of OpenShift, though, where the base container images auto-activate the Python 3 Software Collection, such that `python` refers to Python 3.x by default, just as it does inside a Python level virtual environment: https://github.com/sclorg/s2i-python-container > The only really sensible "do what I expect" situation is a single > Python 3 installation on the user's machine. For that case, adding > Python to PATH (it doesn't matter where as there's only one) is a > sensible option. The downside is that optimising for that case makes > the behaviour for other (more experienced) users worse. But it's not > unreasonable to make that trade-off. Yep, I think add CPython to the user PATH by default will still be helpful: 1. If there's no system Python, then it should "just work"(tm) 2. If there is a system Python, and the user has admin privileges, then "Take these entries from your user PATH and add them to the system PATH" is a better trouble-shooting step 3. Otherwise, they're in the same situation as a Linux user wanting to run a non-default Python, and the solution is similar (i.e. learn to dynamically manipulate the environment, perhaps with the aid of a tool like conda) 4. The Windows installers are one of the cases where python.org *does* get to quickly alter typical new user behaviour, since we control the download links (Related: if anyone wants to tackle a potentially interesting web UX design problem, right now, Python 3.6.3 and 2.7.14 are presented as "peer" downloads on python.org, with no obvious guidance on choosing between them. This should probably be separated out, such that folks that already know they need 2.7 have no trouble finding it, but 3.6 is clearly presented as the default choice for folks that don't already know which version they want) >> 3) Make --user be be automatic for pip install. Not actually the default, >> but pip could do a user install if you don't have the permissions for a >> non-user install. > > The problem here is that the user scripts directory isn't on PATH. We > could put it on, but again we hit the "goes after the system PATH" > problem (on Windows). We should still optimise the defaults for the desired system configuration (i.e. user-mode installs as the cross-platform default outside a venv), and then work on platform-specific troubleshooting guides for the common ways that things can go wrong. > But in principle, if the issues can be solved in a way > we're all happy with, I agree with the principle that "download and > install Python" should result in an environment where > > python myscript.py > pip install something_or_other > > "just work" if the user previously had no Python on their PC. Anything > less results in a worse beginner experience for Python than for other > languages (which usually don't support having multiple versions > installed at once, and often require admin installs, so bypass most of > the problems Python has to deal with). As per https://github.com/pypa/python-packaging-user-guide/issues/396#issuecomment-343739110 and Chris's example of "Analyse your Python environment" as a student's first script, we may be able to get to a point where we can say: 1. Try these commands. If they work, excellent, your Python environment is correctly set up, and there's nothing further to be done. 2. If they don't, then either: - follow [platform specific troubleshooting guide]; or - follow [platform specific venv bootstrapping guide] With any luck, a Python installation troubleshooting guide with concrete steps to try may also prove more popular as a third party link target than installation instructions that assume you're starting with a clean system :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Mon Nov 13 21:47:18 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 14 Nov 2017 12:47:18 +1000 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> <5A087DE7.4050306@brenbarn.net> Message-ID: On 14 November 2017 at 11:51, Nathaniel Smith wrote: > What if instead of installing a standard entry point, the pip > executable was installed as > > #!/bin/sh > exec python -m pip "$@" > > on Unix-likes It would technically be enough to make the shebang line `#!/usr/bin/env python` so the interpreter used was picked up from the environment, rather than being preconfigured at install time. However, the problem is that you don't know for certain that that python will actually have `pip` installed, so it might just fail with a cryptic error instead. However, `pip` could potentially be updated with a `checkenv` subcommand that complains if `sys.executable` and `shutil.which('python')` don't match (and could presumably include other checks as well). > and a pip.bat with the equivalent contents on Windows? > (Bonus: maybe this would fix the problem with upgrading pip on > Windows?) Depending on how the batch file was written, I think the answer to that is "maybe": https://stackoverflow.com/questions/2888976/how-to-make-bat-file-delete-it-self-after-completion/20333152#20333152 Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Mon Nov 13 21:49:56 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 14 Nov 2017 12:49:56 +1000 Subject: [Python-ideas] PEP 560 (second post) In-Reply-To: References: Message-ID: On 14 November 2017 at 09:41, Guido van Rossum wrote: > On Fri, Nov 10, 2017 at 8:54 AM, Ivan Levkivskyi > wrote: >> >> On 10 November 2017 at 17:43, Guido van Rossum wrote: >>> >>> There seem to be some action items from this thread that I haven't seen >>> reflected in the PEP source code yet. >>> [...snip...] >>> Then the next step I propose is a PR with a full implementation. After >>> that I'll likely approve the PEP (or we'll have minor feedback based on >>> trying the implementation). >> >> >> Yes, sorry, I wanted to make updates to the PEP and reference >> implementation, >> but last two weeks were very busy. >> Hopefully, I will work on it this weekend. > > Thanks, I am happy now with the PEP, except for one detail: maybe > `__mro_entry__` should always return a tuple and then maybe renamed to > `__mro_entries__`. (See debate at > https://github.com/python/peps/pull/460#issuecomment-343969528 .) I like that - very nice refinement. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From njs at pobox.com Mon Nov 13 22:08:17 2017 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 13 Nov 2017 19:08:17 -0800 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> <5A087DE7.4050306@brenbarn.net> Message-ID: On Nov 13, 2017 6:47 PM, "Nick Coghlan" wrote: On 14 November 2017 at 11:51, Nathaniel Smith wrote: > What if instead of installing a standard entry point, the pip > executable was installed as > > #!/bin/sh > exec python -m pip "$@" > > on Unix-likes It would technically be enough to make the shebang line `#!/usr/bin/env python` so the interpreter used was picked up from the environment, rather than being preconfigured at install time. However, the problem is that you don't know for certain that that python will actually have `pip` installed, so it might just fail with a cryptic error instead. This would still be a massive improvement over the status quo, which in this situation would present a perfect simulacrum of downloading and installing the package you asked for, except then when you start python the import still fails. I did think of another issue: when installing into a virtualenv, we probably want to keep the current system, so explicit/path/bin/pip continues to work as expected. However, `pip` could potentially be updated with a `checkenv` subcommand that complains if `sys.executable` and `shutil.which('python')` don't match (and could presumably include other checks as well). Unfortunately by the time you know to run this command to have already understood the problem and how to fix it :-). That said, this is probably cheap enough we could do it automatically at startup. Does pip know whether it's pip, pip3, etc. that invoked it? I guess sys.argv[0] should tell us? > and a pip.bat with the equivalent contents on Windows? > (Bonus: maybe this would fix the problem with upgrading pip on > Windows?) Depending on how the batch file was written, I think the answer to that is "maybe": https://stackoverflow.com/questions/2888976/how-to-make-bat- file-delete-it-self-after-completion/20333152#20333152 Sigh. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Nov 13 22:56:21 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 14 Nov 2017 13:56:21 +1000 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> <5A087DE7.4050306@brenbarn.net> Message-ID: On 14 November 2017 at 13:08, Nathaniel Smith wrote: > On Nov 13, 2017 6:47 PM, "Nick Coghlan" wrote: > > On 14 November 2017 at 11:51, Nathaniel Smith wrote: > > What if instead of installing a standard entry point, the pip > > executable was installed as > > > > #!/bin/sh > > exec python -m pip "$@" > > > > on Unix-likes > > It would technically be enough to make the shebang line > `#!/usr/bin/env python` so the interpreter used was picked up from the > environment, rather than being preconfigured at install time. However, > the problem is that you don't know for certain that that python will > actually have `pip` installed, so it might just fail with a cryptic > error instead. > > > This would still be a massive improvement over the status quo, which in > this situation would present a perfect simulacrum of downloading and > installing the package you asked for, except then when you start python the > import still fails. > > I did think of another issue: when installing into a virtualenv, we > probably want to keep the current system, so explicit/path/bin/pip > continues to work as expected. > My essential concern is that I don't think we should be opening ourselves up to *even more* platform dependent behaviours in this area. Instead, if we decide that we would like to request that pip to start doing something different when: * `sys.executable` and `shutil.which('python')` are inconsistent with each other * `sys.argv[0]` doesn't end with `site-packages/pip/__main__.py` and isn't `shutil.which('pip')` then we'd be better off putting that logic in pip's Python code, rather than in the platform dependent script wrappers where we can't control the error messages presented when our expectations aren't met. I do like the idea of making that distinction in principle though, as it's a better one than "inside a virtual environment or not?". While Python level venvs are *a* way to obtain the desired pip/python consistency, they're not the only one: - Arch Linux does it by default - Docker's Python 3 images do it by default (Alpine Linux may even do it by default in general) - Windows installs do it by default (if PATH is configured correctly) - conda env does it - Software Collections do it (including in the Python 3 Docker images for RHEL & CentOS) - *nix environment modules do it - system admins may set up per-user install profiles to do it - etc... Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From desmoulinmichel at gmail.com Tue Nov 14 01:39:52 2017 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Tue, 14 Nov 2017 07:39:52 +0100 Subject: [Python-ideas] venv *is* provided in the standard Python install on Debian/Ubuntu In-Reply-To: <20171112193851.3c24085f@fsol> References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> <20171112193851.3c24085f@fsol> Message-ID: Le 12/11/2017 ? 19:38, Antoine Pitrou a ?crit?: > On Sun, 12 Nov 2017 12:20:45 +0000 > Paul Moore wrote: >> >>> Well, not exactly. Do you do python -m venv, or py -x.x -m venv or >>> pythonx -m venv ? Wait, it's not installed by default on debian. >> >> Seriously? Debian don't provide venv in the standard Python install? >> That's just broken. > > Frankly, I don't know where the current discussion comes from, but on > two recent Debian and Ubuntu setups, I get: > > $ dpkg -S /usr/lib/python3.5/venv/__init__.py > libpython3.5-stdlib:amd64: /usr/lib/python3.5/venv/__init__.py > > > Which, for the uninitiated, means "the venv module is provided by the > Debian/Ubuntu package named libpython3.5-stdlib". That package is, in > turn, a dependency of the "python3.5" package. > Screenshot from yesterday morning. On Ubuntu 16.04, LTS of most popular Linux distro based on debian: we can see venv is clearly missing. On the bright side, it does give you an error message. We should have that for pip as well ! Unfortunatly, the message gives you a wrong advice, as you had to run the: sudo apt-get install python3.6-venv command and not: sudo apt-get install python-venv And my students were stucks. And this was a traning with professional adults java devs. Imagine total beginers... -------------- next part -------------- A non-text attachment was scrubbed... Name: IMG_20171113_161409.jpg Type: image/jpeg Size: 1217278 bytes Desc: not available URL: From desmoulinmichel at gmail.com Tue Nov 14 01:45:09 2017 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Tue, 14 Nov 2017 07:45:09 +0100 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> Message-ID: Le 12/11/2017 ? 19:02, Stephan Houben a ?crit?: > > 2017-11-12 13:20 GMT+01:00 Paul Moore >: > > On 12 November 2017 at 06:19, Michel Desmoulin > > wrote: > > > Well, not exactly. Do you do python -m venv, or py -x.x -m venv or > > pythonx -m venv ? Wait, it's not installed by default on debian. > > Seriously? Debian don't provide venv in the standard Python install? > That's just broken. > > > I think that it is wrong to think of Debian's "python3" package > as purporting to provide the "standard Python install". > > This is really more properly though of as the Python *runtime* > environment, i.e. the minimum stuff you need to run programs > written in Python. > I think it's wrong to have a different setup on different plateforms. Python is a portable language. It's also a language you use for quick scripts and from a lot of non professional devs. Unifying the python experience and making sure it's easy for beginners to get started is a very important goal IMO. Python is actually quite good at it. But we can make it even better. From desmoulinmichel at gmail.com Tue Nov 14 01:47:25 2017 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Tue, 14 Nov 2017 07:47:25 +0100 Subject: [Python-ideas] Looking for input to help with the pip situation Message-ID: <76a18ed7-1a0f-5a8e-b7f6-69fc48a764aa@gmail.com> General summary so far ####################### This debate has been very civil so far and quite productive, so thanks to everyone involved. Several issues and solution proposals have been mentioned, so I will make a summary here. Issue 1: running Python is a different command different OS when several versions are installed ================================================================================================== Proposal A: ----------- Suffix Python executable on Windows like on Unix, so that people will type pythonX.X if they want a specify version. Pros: easy and discoverable. Cons: you need a lot of stuff in the system path. Proposal B: ------------ Provide the "py" command on Unix, so that people will type "py -x.x" Pros: no more issues with system path. Just add this command, and register python installations to be listed by "py". Cons: very few people know about "py", even on windows. This would need lots of communication. Harder to change Mac and majors Linux distros setup than just the next windows installer. Issue 2: there is no one and only one obvious way to do venv / pip =================================================================== Because pip and venv are tied to a Python version, there are numerous ways to call them. Just for pip, you have "pip", "py -x.x -m pip", "pythonx.x -m pip" and "pipx". Proposal A: ----------- Decide on one form, and promote it everywhere. Pros: simple. Cons: long and tedious process, with a huge inertia before all 3rd party to it. Would need a "PEP 8 for Python command". Requires "issue 1" to be solved before. Proposal B: ------------ Provide a pipenv-like behavior pipenv is a tool created by the author of "requests" that automatically "do the right thing" by default. The first time you install a package, it creates a virtualenv for the current directory (but don't store it in the current directory). It will then install all packages in there. "pipenv shell" starts a shell in the virtualenv. Dependencies are automatically dumped in a requirement files separating dev and prod. Pros: bypass the multiple installers issue by prompting you what version you wish the first time it creates a virtualenv. Solve the problem of admin rights and system path. Cons: lots of work. Either a new tool or breaking compat with pip. If debian didn't install pip by default, this may not help. Issue 3: pip and virtualenv are not available everywhere =================================================================== Proposal A: ------------ Make a mock "pip" commands for when it's not installed. venv already does that in some cases (see attachements), alhought it does give you the wrong command so it should be fixed. The gist of it: if the users try to run pip install while it's missing, it prints a warning explaining it's not installed and the command you can run to solve the problem plus a link to more information in the documentation. Pros: low friction to implement. Cons: requires to put up a hook somewhere so that each linux distro can input "the proper commands to use" for their system when they package python. More a workaround than a solution. Proposal B: ------------ Make pip and venv part of the standard and request debian that they provide it. Pros: straight forward. Cons: holy war in the making. ................. Can I have +1 and -1 from each of you on each point ? Should I start a new thread for each of them ? Favorite personnal combination ================================ 1 - So far, I like "providing py on unix" better, but I think it's too hard to do. So provide suffix on windows seems more achievable. So +1 for having pythonX.X on windows. 2 - Decide on one form, and promote it everywhere is again the simplest IMO. I love pipenv but the strings to pull to get something like this means it probably would not happen. If we stick to suffix for issue 1, this mean we can promote: pythonX.X -m pip pythonX.X -m venv Everywhere. In the docs. In the tutorials. In teaching classes. This will help a lot to remove all confusions and make the Python start up experience way easier. 3 - getting angry debian devs is probably not wise. The "mock" pip/venv command also plays very well with the 2 other solutions. It means we can promote everywhere: pythonX.X -m cmd It will work most of the time. And when it doesn't, give clear contextual steps on From rosuav at gmail.com Tue Nov 14 02:38:36 2017 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 14 Nov 2017 18:38:36 +1100 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: <76a18ed7-1a0f-5a8e-b7f6-69fc48a764aa@gmail.com> References: <76a18ed7-1a0f-5a8e-b7f6-69fc48a764aa@gmail.com> Message-ID: On Tue, Nov 14, 2017 at 5:47 PM, Michel Desmoulin wrote: > Proposal B: > ------------ > > Make pip and venv part of the standard and request debian that they > provide it. > > Pros: straight forward. > > Cons: holy war in the making. If this would really cause such back-lash, how about: Make pip and venv part of the standard, but permit them to be omitted from "python3" as long as there is a "python3-full" package that will drag them in. Several other packages are split up like that. Pros: Only slightly less straight-forward, and hopefully incites only a small war. Cons: What does "full" really mean? Does it require a GUI subsystem, for instance (as a dep of tkinter)? Might just shift the problem. OTOH, the sorts of people who run non-graphical Linuxes are usually going to be comfortable installing python3-pip explicitly, so maybe it's doable. ChrisA From rosuav at gmail.com Tue Nov 14 02:40:27 2017 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 14 Nov 2017 18:40:27 +1100 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <76a18ed7-1a0f-5a8e-b7f6-69fc48a764aa@gmail.com> Message-ID: On Tue, Nov 14, 2017 at 6:38 PM, Chris Angelico wrote: > On Tue, Nov 14, 2017 at 5:47 PM, Michel Desmoulin > wrote: >> Proposal B: >> ------------ >> >> Make pip and venv part of the standard and request debian that they >> provide it. >> >> Pros: straight forward. >> >> Cons: holy war in the making. > > If this would really cause such back-lash, how about: Make pip and > venv part of the standard, but permit them to be omitted from > "python3" as long as there is a "python3-full" package that will drag > them in. Several other packages are split up like that. > > Pros: Only slightly less straight-forward, and hopefully incites only > a small war. > > Cons: What does "full" really mean? Does it require a GUI subsystem, > for instance (as a dep of tkinter)? Might just shift the problem. > OTOH, the sorts of people who run non-graphical Linuxes are usually > going to be comfortable installing python3-pip explicitly, so maybe > it's doable. Oh, forgot to give a clear opinion: Even if the compromise isn't viable, I am still +1 on auto-installing pip. There's already a python3-minimal package for people who want "just the interpreter, please" installs. ChrisA From storchaka at gmail.com Tue Nov 14 03:24:19 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 14 Nov 2017 10:24:19 +0200 Subject: [Python-ideas] PEP 562 In-Reply-To: References: Message-ID: 10.09.17 21:48, Ivan Levkivskyi ????: > ? # lib.py > > ? from warnings import warn > > ? deprecated_names = ["old_function", ...] > > ? def _deprecated_old_function(arg, other): > ? ? ? ... > > ? def __getattr__(name): > ? ? ? if name in deprecated_names: > ? ? ? ? ? warn(f"{name} is deprecated", DeprecationWarning) > ? ? ? ? ? return globals()[f"_deprecated_{name}"] > ? ? ? raise AttributeError(f"module {__name__} has no attribute {name}") > > ? # main.py > > ? from lib import old_function ?# Works, but emits the warning I think the PEP should provide better examples, because they will be copied in third-party code. For keeping all functionality (in particularly be pickleable) the deprecated function should preserve its name. def old_function(arg, other): ... _deprecated_old_function = old_function del old_function or def _deprecated_old_function(arg, other): ... _deprecated_old_function.__name__ = 'old_function' _deprecated_old_function.__qualname__ = 'old_function' (I prefer the former variant.) I'm wondering if it is worth to provide a special helper that will rename the deprecated function and create the corresponding __getattr__ function. It could be possible to create helpers that implement module-level properties too. From p.f.moore at gmail.com Tue Nov 14 03:56:51 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 14 Nov 2017 08:56:51 +0000 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> <5A087DE7.4050306@brenbarn.net> Message-ID: On 14 November 2017 at 03:08, Nathaniel Smith wrote: > On Nov 13, 2017 6:47 PM, "Nick Coghlan" wrote: >> and a pip.bat with the equivalent contents on Windows? >> (Bonus: maybe this would fix the problem with upgrading pip on >> Windows?) > > Depending on how the batch file was written, I think the answer to > that is "maybe": > https://stackoverflow.com/questions/2888976/how-to-make-bat-file-delete-it-self-after-completion/20333152#20333152 > > > Sigh. Batch files are not suitable for this task. The wrappers have to be executables. See http://paul-moores-notes.readthedocs.io/en/latest/wrappers.html for a detailed analysis I did some time ago. Paul From stephanh42 at gmail.com Tue Nov 14 04:06:08 2017 From: stephanh42 at gmail.com (Stephan Houben) Date: Tue, 14 Nov 2017 10:06:08 +0100 Subject: [Python-ideas] venv *is* provided in the standard Python install on Debian/Ubuntu In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> <20171112193851.3c24085f@fsol> Message-ID: Hi Brett, The current documentation *cannot* be fixed, since fixing it would entail adding an initial two-page essay on "how to start Python on various platforms/systems" (it is really NOT as simple as Windows=python, Linux=python3) and such a PR will certainly by rejected. In my opinion, the only alternatives are 1. either harmonize the invocation of python across platforms (and *then* adapt the docs to follow). Which was pretty much the whole topic of this thread so far. 2. or just use "python" consistently across all docs (since that is the *only* command which is at least consistent among python.org installers), and add weasel-wording to "consult documentation of third-party installers" 3. or leave the docs broken for at least some people some of the time. Stephan 2017-11-14 2:31 GMT+01:00 Brett Cannon : > > > On Mon, Nov 13, 2017, 00:01 Stephan Houben, wrote: > >> Hi all, >> >> Related to this text on https://docs.python.org/3/library/venv.html : >> >> ============ >> >> Note >> The pyvenv script has been deprecated as of Python 3.6 in favor of >> using python3 -m venv to help prevent any potential confusion as to >> which Python interpreter a virtual environment will be based on. >> ============ >> >> It's clearer than the text below to which I originally referred. >> >> However, this text has also problems in that it is too unix-specific. >> In particular: >> * Most seriously, it refers to "python3" which doesn't work with the >> python.org Windows installer. >> > > It can, but it's opt-in. It's just one of those things that's easy to > forget. > > * Less seriously, it refers to "pyenv" as a "script" which is unix jargon >> and moreover technically >> incorrect on Windows. (Also, needlessly specific, it should just be >> "the pyenv command", >> how it is implemented is irrelevant for this section). >> > > I disagree with this as Python refers to .Py files that you execute > directly as "scripts", so I don't think this requires clarification. > > > Anyway, a pull request with suggested wording to address your concerns > would be the best way to try and rectify the issue. > > -brett > > > >> Stephan >> >> 2017-11-13 0:32 GMT+01:00 Chris Angelico : >> >>> On Mon, Nov 13, 2017 at 10:29 AM, Nick Coghlan >>> wrote: >>> > On 13 November 2017 at 07:11, Chris Angelico wrote: >>> >> On Mon, Nov 13, 2017 at 6:24 AM, Stephan Houben >>> wrote: >>> >>> Hi Antoine, >>> >>> >>> >>> The venv module is included, >>> >>> however the pyvenv script is in a separate package >>> >>> python3.5-venv . >>> >>> >>> >>> By the way, I was totally confused by the following text form the >>> doc. >>> >>> >>> >>> https://docs.python.org/3/library/venv.html >>> >>> >>> >>> ======== >>> >>> Deprecated since version 3.6: pyvenv was the recommended tool for >>> creating >>> >>> virtual environments for Python 3.3 and 3.4, and is deprecated in >>> Python >>> >>> 3.6. >>> >>> >>> >>> Changed in version 3.5: The use of venv is now recommended for >>> creating >>> >>> virtual environments. >>> >>> >>> >>> ======== >>> >> >>> >> Not sure where you're reading that. I'm seeing: >>> >> >>> >> """ >>> >> Note >>> >> The pyvenv script has been deprecated as of Python 3.6 in favor of >>> >> using python3 -m venv to help prevent any potential confusion as to >>> >> which Python interpreter a virtual environment will be based on. >>> >> """ >>> >> >>> >> I think that's pretty clear. "python3 -m venv env" is the standard and >>> >> recommended way to spin up a virtual environment. >>> > >>> > It's further down in the page, under >>> > https://docs.python.org/3/library/venv.html#creating- >>> virtual-environments >>> > >>> > I think the deprecation notice for pyvenv should just be deleted, >>> > since it renders like the *module* is deprecated. >>> >>> Ah, I see it now, thanks. >>> >>> Agreed; or maybe downgrade it to a parenthetical comment. Focus on >>> "this is how to do the obvious thing", and only as an afterthought >>> mention "it used to be done differently" in case someone greps for >>> pyvenv. >>> >>> ChrisA >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >>> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Tue Nov 14 05:02:24 2017 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 14 Nov 2017 02:02:24 -0800 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> <5A087DE7.4050306@brenbarn.net> Message-ID: On Tue, Nov 14, 2017 at 12:56 AM, Paul Moore wrote: > On 14 November 2017 at 03:08, Nathaniel Smith wrote: >> On Nov 13, 2017 6:47 PM, "Nick Coghlan" wrote: > >>> and a pip.bat with the equivalent contents on Windows? >>> (Bonus: maybe this would fix the problem with upgrading pip on >>> Windows?) >> >> Depending on how the batch file was written, I think the answer to >> that is "maybe": >> https://stackoverflow.com/questions/2888976/how-to-make-bat-file-delete-it-self-after-completion/20333152#20333152 >> >> >> Sigh. > > Batch files are not suitable for this task. The wrappers have to be > executables. See > http://paul-moores-notes.readthedocs.io/en/latest/wrappers.html for a > detailed analysis I did some time ago. Ah, interesting. My reason for suggesting it in the first place because I was hoping to avoid paying the process spawn overhead twice, but it sounds like this specific trick is misguided all around :-). -n -- Nathaniel J. Smith -- https://vorpus.org From ncoghlan at gmail.com Tue Nov 14 05:07:56 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 14 Nov 2017 20:07:56 +1000 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: <76a18ed7-1a0f-5a8e-b7f6-69fc48a764aa@gmail.com> References: <76a18ed7-1a0f-5a8e-b7f6-69fc48a764aa@gmail.com> Message-ID: On 14 November 2017 at 16:47, Michel Desmoulin wrote: > General summary so far > ####################### > > > This debate has been very civil so far and quite productive, so thanks > to everyone involved. > > Several issues and solution proposals have been mentioned, so I will > make a summary here. > > Issue 1: running Python is a different command different OS when several > versions are installed > ============================================================ > ====================================== > > Proposal A: > ----------- > > Suffix Python executable on Windows like on Unix, so that people will > type pythonX.X if they want a specify version. > > Pros: easy and discoverable. > > Cons: you need a lot of stuff in the system path. > Con: we hope to have the problem resolved on the Linux distro side such that "python" typically means "python" by the time community support for Python 2 ends in 2020. Since Windows has gone the better part of two decades without version Python commands, adding them because we're impatient with the pace of change at the Linux distro level doesn't really make sense (especially when Linux holds such a small fraction of the non-phone client device market). Proposal B: > ------------ > > Provide the "py" command on Unix, so that people will type "py -x.x" > > Pros: no more issues with system path. Just add this command, and > register python installations to be listed by "py". > > Cons: very few people know about "py", even on windows. This would need > lots of communication. Harder to change Mac and majors Linux distros > setup than just the next windows installer. > Proposal C: Make sure 'python' just works on a default Window user install, and treat any system where that doesn't work by default as a special case that needs troubleshooting. Pro: a solid troubleshooting guide will be useful regardless Con: mainstream Linux distros will still be a consistent exception for at least the next couple of years (and probably longer for the LTS variants) > Issue 2: there is no one and only one obvious way to do venv / pip > =================================================================== > > Because pip and venv are tied to a Python version, there are numerous > ways to call them. > > Just for pip, you have "pip", "py -x.x -m pip", "pythonx.x -m pip" and > "pipx". > > Proposal A: > ----------- > > Decide on one form, and promote it everywhere. > > > Pros: simple. > > Cons: long and tedious process, with a huge inertia before all 3rd party > to it. Would need a "PEP 8 for Python command". Requires "issue 1" to be > solved before. > This downside doesn't apply as long as the form we standardise on is "pip install module". That already works in a lot of contexts, and the cases where it doesn't work are typically ones where "python -m pip" may also be a problem. Given a solid troubleshooting guide for issue 1, we may also be able to add some implicit checks to pip that provide alerts for problematic configurations, like checking: >>> user_scheme = "posix_user" # or "nt_user" or "osx_framework_user" >>> sysconfig.get_path("scripts", scheme="posix_user") in os.getenv("PATH").split(os.pathsep) True The latter check returning false is a sign that the `PATH` configuration might be a problem. > Proposal B: > ------------ > > Provide a pipenv-like behavior > > pipenv is a tool created by the author of "requests" that automatically > "do the right thing" by default. The first time you install a package, > it creates a virtualenv for the current directory (but don't store it in > the current directory). It will then install all packages in there. > "pipenv shell" starts a shell in the virtualenv. Dependencies are > automatically dumped in a requirement files separating dev and prod. > > Pros: bypass the multiple installers issue by prompting you what version > you wish the first time it creates a virtualenv. Solve the problem of > admin rights and system path. > > Cons: lots of work. Either a new tool or breaking compat with pip. If > debian didn't install pip by default, this may not help. > This is essentially the path that conda takes, and it works well for folks that are willing to adopt conda's specific approach to managing your Python runtimes. > Issue 3: pip and virtualenv are not available everywhere > =================================================================== > > Proposal B: > ------------ > > Make pip and venv part of the standard and request debian that they > provide it. > > Pros: straight forward. > > Cons: holy war in the making. > I don't think it's *that* bad. Debian does have support for weak dependencies, so the resolution here may be as simple as asking Matthias to add a Recommends from python3 to python3-venv so that ensurepip is present by default on typical workstation configurations, but can still be easily omitted from server configurations. > Favorite personnal combination > ================================ > > 1 - So far, I like "providing py on unix" better, but I think it's too > hard to do. So provide suffix on windows seems more achievable. > > So +1 for having pythonX.X on windows. > I think this is actually similar to the situation with 'py' on *nix systems: with Windows having never had these before, adding and promoting them would be at best ineffective, and at worst outright confusing (especially if "python" starts meaning Python 3 by default on Linux as well within the next few years). > 2 - Decide on one form, and promote it everywhere is again the simplest > IMO. I love pipenv but the strings to pull to get something like this > means it probably would not happen. > We've been evaluating a potential pipenv tutorial for the Python Packaging User Guide, and while we're likely going to add it soon, it will be as a new "application dependency management" tutorial alongside the existing ones, rather than as a direct replacement for the pip-based "package installation" tutorial: https://github.com/pypa/python-packaging-user-guide/issues/394#issuecomment-343343760 > If we stick to suffix for issue 1, this mean we can promote: > > pythonX.X -m pip > pythonX.X -m venv > > Everywhere. In the docs. In the tutorials. In teaching classes. > > This will help a lot to remove all confusions and make the Python start > up experience way easier. > Except it won't, because now you can't seamlessly update your class instructions to new versions of Python - you'll have to go through and change all the "X.Y" references to refer to the right version. Writing cross-platform scripts will also get even harder, as you wouldn't be able to say "I just need a version of Python, I don't really care which one" any more. Python feature releases are already disruptive enough due to filesystem layout changes and file format changes - we don't need to make them worse by encouraging people to embed the exact target version in their documentation and helper scripts. > 3 - getting angry debian devs is probably not wise. The "mock" pip/venv > command also plays very well with the 2 other solutions. > While I was a bit snarky about "python -m venv" being broken by default earlier in the thread, *please* don't take that as meaning that we can't collaborate constructively with the Debian folks. We can, and do, but the friction between language level package management and system level package management is one with a *long* history that we're all working towards resolving together (See [1] & [2] for write-ups of a couple of linux.conf.au talks I've given about the problem), as is the question of how best to handle the `/usr/bin/python` symlink. > It means we can promote everywhere: > > pythonX.X -m cmd > > It will work most of the time. The combination of: pip install package python -m venv already works in most cases, *except* apparently the critical one of "New Python user on Windows downloads the python.org installer and clicks through all the buttons without changing any settings". So I think the main near term step forward would be to convince Steve Dower (as the Windows installer maintainer) to change that default behaviour yet again for 3.7, and then work towards coming up with a solid environment troubleshooting guide to include on packaging.python.org. Cheers, Nick. [1] https://lwn.net/Articles/580399/ [2] https://lwn.net/Articles/711906/ -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From gadgetsteve at live.co.uk Mon Nov 13 14:33:14 2017 From: gadgetsteve at live.co.uk (Steve Barnes) Date: Mon, 13 Nov 2017 19:33:14 +0000 Subject: [Python-ideas] A proliferation of (un-)Pythonically programmatic pragmas In-Reply-To: References: Message-ID: On 13/11/2017 19:10, Barry Warsaw wrote: > I love many of the ancillary tools that help improve the quality of my > Python code, including flake8, coverage, and mypy. Each of these > usually produces some great diagnostics, but none of them are perfect, > so they also produce false positives that have to be suppressed. > > Each of these tools supports "pragmas", i.e. comments you can add to a > line of code to suppress a warning. E.g. with flake8 I can say: > > import readline, rlcompleter # noqa: F401 > > to suppress the flake8 warnings about unused imports. These modules > work by side-effect, which of course a static analyzer can't know. > > Similarly, when mypy complains about a line it's confused on, I can add > a comment to suppress the useless warning: > > try: > from pathlib import Path, PurePath > except ImportError: > from pathlib2 import Path, PurePath # type: ignore > > And when I want to boost coverage, but I know that some lines aren't > covered under some versions of Python, I can do something like: > > self.send_error(HTTPStatus.NOT_FOUND) # pragma: nocover > > These are all well and good until I have to *combine* suppressions. > E.g. in the pathlib2 pragma to mypy, I also get a flake8 warning and > I've tried just about every combination of pragma comment I can think > of, but I've not been able to make both tools happy. I've resorted to > refactoring the code into an entire module for flake8 to ignore and added: > > # flake8: noqa > > to suppress all warnings in the file. I actually have hit on a few > places where I need to suppress warnings from all three tools on the > same line, and I basically can't do it. > > The specifics aren't as important as the general use case: multiple > tools competing for the same valuable real-estate. > > I have no ideas how to improve the situation, and of course any solution > would involve some coordination between all of these tools, but it's > beginning to feel like a losing battle. Is there a better way? > > Cheers, > -Barry > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org One thing that I have come across with some other toolchains static analysers it to have either or both of: - Suppression comments on a line of their own suppress that specific error on the next non-comment line - Block suppressions that are turned off and then on again. Either of the above could work nicely. -- Steve (Gadget) Barnes Any opinions in this message are my personal opinions and do not reflect those of my employer. --- This email has been checked for viruses by AVG. http://www.avg.com From solipsis at pitrou.net Tue Nov 14 07:28:20 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 14 Nov 2017 13:28:20 +0100 Subject: [Python-ideas] Looking for input to help with the pip situation References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> <5A087DE7.4050306@brenbarn.net> Message-ID: <20171114132820.6cbc67bf@fsol> On Tue, 14 Nov 2017 12:14:50 +1000 Nick Coghlan wrote: > > > > I don't think Windows ever had python2.exe/python3.exe, but I could be wrong. > > Not that I'm aware of in the python.org installers, and I don't think > ActivePython does either. I'm less sure about Enthought or Anaconda > (since I've never used either of them on Windows). Looking at my Windows VM... * from a Miniconda install (equivalent, I think, to a bare-bones Anaconda with the minimal package set required for a function "python" and "conda"): c:\>where python C:\Miniconda3\python.exe c:\>where python3 INFO: Could not find files for the given pattern(s). * from a user-created conda environment: c:\>activate da36 (da36) c:\>where python C:\Miniconda3\envs\da36\python.exe C:\Miniconda3\python.exe (da36) c:\>where python3 INFO: Could not find files for the given pattern(s). (no clue about Enthought, sorry) > Yeah, myself, Barry Warsaw, Matthias Klose, and a number of other > folks on linux-sig have poked at this idea multiple times since Geoff > Thomas first posted about https://github.com/geofft/pythonmux, but our > conclusion each time has been that it wouldn't help enough to justify > the effort involved in implementing and promoting it. After 20+ years > of usage in the Linux ecosystem, `/usr/bin/python` and `/usr/bin/env > python` are simply too entrenched in both people's habits and existing > code. +1. As a matter of fact, Anaconda also doesn't seem to expose "py" on Windows: (da36) c:\>py 'py' is not recognized as an internal or external command, operable program or batch file. > >> 3) Make --user be be automatic for pip install. Not actually the default, > >> but pip could do a user install if you don't have the permissions for a > >> non-user install. > > > > The problem here is that the user scripts directory isn't on PATH. We > > could put it on, but again we hit the "goes after the system PATH" > > problem (on Windows). > > We should still optimise the defaults for the desired system > configuration (i.e. user-mode installs as the cross-platform default > outside a venv), and then work on platform-specific troubleshooting > guides for the common ways that things can go wrong. Hmm... I liked --user at some point, but if you start using it a lot it becomes problematic, for example if you want to install two different scripts/applications with incompatible dependencies. That said, it *is* much better than system-wide installs of user packages :-) Regards Antoine. From solipsis at pitrou.net Tue Nov 14 07:33:16 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 14 Nov 2017 13:33:16 +0100 Subject: [Python-ideas] Looking for input to help with the pip situation References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> Message-ID: <20171114133316.732d3a62@fsol> On Tue, 14 Nov 2017 07:45:09 +0100 Michel Desmoulin wrote: > > I think it's wrong to have a different setup on different plateforms. > Python is a portable language. It's also a language you use for quick > scripts and from a lot of non professional devs. > > Unifying the python experience and making sure it's easy for beginners > to get started is a very important goal IMO. > > Python is actually quite good at it. But we can make it even better. If the problem has to do with how Debian decides to repackage upstream software, I don't think we (python-dev) really can make it better, though. Regards Antoine. From solipsis at pitrou.net Tue Nov 14 07:36:24 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 14 Nov 2017 13:36:24 +0100 Subject: [Python-ideas] Looking for input to help with the pip situation References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> <5A087DE7.4050306@brenbarn.net> <20171114132820.6cbc67bf@fsol> Message-ID: <20171114133624.4ca8db67@fsol> On Tue, 14 Nov 2017 13:28:20 +0100 Antoine Pitrou wrote: > On Tue, 14 Nov 2017 12:14:50 +1000 > Nick Coghlan wrote: > > > > > > I don't think Windows ever had python2.exe/python3.exe, but I could be wrong. > > > > Not that I'm aware of in the python.org installers, and I don't think > > ActivePython does either. I'm less sure about Enthought or Anaconda > > (since I've never used either of them on Windows). > > Looking at my Windows VM... > > * from a Miniconda install (equivalent, I think, to a bare-bones > Anaconda with the minimal package set required for a function > "python" and "conda"): > > c:\>where python > C:\Miniconda3\python.exe > > c:\>where python3 > INFO: Could not find files for the given pattern(s). > > * from a user-created conda environment: > > c:\>activate da36 > > (da36) c:\>where python > C:\Miniconda3\envs\da36\python.exe > C:\Miniconda3\python.exe > > (da36) c:\>where python3 > INFO: Could not find files for the given pattern(s). I forgot to mention that those "python.exe" are all Python 3.6 executables (or hardlinks thereof). Regards Antoine. From p.f.moore at gmail.com Tue Nov 14 08:06:41 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 14 Nov 2017 13:06:41 +0000 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> <5A087DE7.4050306@brenbarn.net> Message-ID: On 14 November 2017 at 10:02, Nathaniel Smith wrote: > On Tue, Nov 14, 2017 at 12:56 AM, Paul Moore wrote: >> On 14 November 2017 at 03:08, Nathaniel Smith wrote: >>> On Nov 13, 2017 6:47 PM, "Nick Coghlan" wrote: >> >>>> and a pip.bat with the equivalent contents on Windows? >>>> (Bonus: maybe this would fix the problem with upgrading pip on >>>> Windows?) >>> >>> Depending on how the batch file was written, I think the answer to >>> that is "maybe": >>> https://stackoverflow.com/questions/2888976/how-to-make-bat-file-delete-it-self-after-completion/20333152#20333152 >>> >>> >>> Sigh. >> >> Batch files are not suitable for this task. The wrappers have to be >> executables. See >> http://paul-moores-notes.readthedocs.io/en/latest/wrappers.html for a >> detailed analysis I did some time ago. > > Ah, interesting. My reason for suggesting it in the first place > because I was hoping to avoid paying the process spawn overhead twice, > but it sounds like this specific trick is misguided all around :-). It doesn't even save the process overhead if you're running Powershell as your main shell, or running pip from a terminal window in your editor/IDE, ... I really wish Windows *did* provide a usable "shell script" implementation (of any form) but sadly it simply doesn't. Paul From toddrjen at gmail.com Tue Nov 14 09:58:40 2017 From: toddrjen at gmail.com (Todd) Date: Tue, 14 Nov 2017 09:58:40 -0500 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <76a18ed7-1a0f-5a8e-b7f6-69fc48a764aa@gmail.com> Message-ID: On Nov 14, 2017 02:39, "Chris Angelico" wrote: On Tue, Nov 14, 2017 at 5:47 PM, Michel Desmoulin wrote: > Proposal B: > ------------ > > Make pip and venv part of the standard and request debian that they > provide it. > > Pros: straight forward. > > Cons: holy war in the making. Cons: What does "full" really mean? Does it require a GUI subsystem, for instance (as a dep of tkinter)? Might just shift the problem. The solution a lot of other Linux distributions use is to have a "python(2|3)-base" package that pulls in just the minimal runtime environment and have the regular "python(2|3)" package gives the full upstream python install. Things like tk bindings and pip tend to be separate packages the normal install depends on. "Minimal", if I understand it correctly, is the python(2|3) executable and any module not needing external dependencies, and "full" is everything upstream provides. -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephanh42 at gmail.com Tue Nov 14 10:23:11 2017 From: stephanh42 at gmail.com (Stephan Houben) Date: Tue, 14 Nov 2017 16:23:11 +0100 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <76a18ed7-1a0f-5a8e-b7f6-69fc48a764aa@gmail.com> Message-ID: Hi Nick, 2017-11-14 11:07 GMT+01:00 Nick Coghlan : > On 14 November 2017 at 16:47, Michel Desmoulin > wrote: > >> Proposal A: >> ----------- >> >> Suffix Python executable on Windows like on Unix, so that people will >> type pythonX.X if they want a specify version. >> >> Pros: easy and discoverable. >> >> Cons: you need a lot of stuff in the system path. >> > > Con: we hope to have the problem resolved on the Linux distro side such > that "python" typically means "python" by the time community support for > Python 2 ends in 2020. Since Windows has gone the better part of two > decades without version Python commands, adding them because we're > impatient with the pace of change at the Linux distro level doesn't really > make sense (especially when Linux holds such a small fraction of the > non-phone client device market). > Perhaps I could sell you on the idea of a Windows "python3" executable, not as the New Official Way to do things, but rather as a pragmatic measure to make code from those Linux weirdos ;-) more likely to work on Windows. I would like to compare it with the strings/bytes-for-pathnames issue: the official recommendation is to use strings everywhere, problems with round-tripping arbitrary not-valid-UTF8 filenames on POSIX have been solved now. Still, POSIX people continued to use bytes, so a pragmatic change was to make so that bytes now also work reliably as pathnames under Windows. Similarly, even when all Linux distributions have switched to python==python3, people will probably still read and write tutorials with python3 in it, and perhaps we should accommodate that. Otherwise I am +1 on your proposal C: if anything this thread has made it clear that there is so much variety in third-party Python installers, not just Linux distributions, but also things like Anaconda, that it seems unreasonable to require that the official Python documentation covers all those situations. Stephan -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Nov 14 11:32:41 2017 From: guido at python.org (Guido van Rossum) Date: Tue, 14 Nov 2017 08:32:41 -0800 Subject: [Python-ideas] PEP 562 In-Reply-To: References: Message-ID: On Tue, Nov 14, 2017 at 12:24 AM, Serhiy Storchaka wrote: > 10.09.17 21:48, Ivan Levkivskyi ????: > >> # lib.py >> >> from warnings import warn >> >> deprecated_names = ["old_function", ...] >> >> def _deprecated_old_function(arg, other): >> ... >> >> def __getattr__(name): >> if name in deprecated_names: >> warn(f"{name} is deprecated", DeprecationWarning) >> return globals()[f"_deprecated_{name}"] >> raise AttributeError(f"module {__name__} has no attribute {name}") >> >> # main.py >> >> from lib import old_function # Works, but emits the warning >> > > I think the PEP should provide better examples, because they will be > copied in third-party code. > > For keeping all functionality (in particularly be pickleable) the > deprecated function should preserve its name. > > def old_function(arg, other): > ... > _deprecated_old_function = old_function > del old_function > > or > > def _deprecated_old_function(arg, other): > ... > _deprecated_old_function.__name__ = 'old_function' > _deprecated_old_function.__qualname__ = 'old_function' > > (I prefer the former variant.) > However the latter could be done by a decorator. > I'm wondering if it is worth to provide a special helper that will rename > the deprecated function and create the corresponding __getattr__ function. > > It could be possible to create helpers that implement module-level > properties too. > OTOH not everyone cares about whether their functions are picklable. Maybe a discussion of how to make deprecated functions picklable could be in an Appendix? And another example of how to implement more functional lazy loading. Or even more complete examples could be maintained as a separate GitHub repo (like Ethan did for PEP 561). -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From nas-python-ideas at arctrix.com Tue Nov 14 15:34:07 2017 From: nas-python-ideas at arctrix.com (Neil Schemenauer) Date: Tue, 14 Nov 2017 14:34:07 -0600 Subject: [Python-ideas] Modules as global namespaces rather than dicts Message-ID: <20171114203407.dnw3ghecyz3qsow3@python.ca> This is an idea I have been playing with and seems to hold some promise. I think we should use a module instance as the standard global namespace rather than directly using its dict. I have a prototype version of CPython that does this, not working 100% yet though. Major changes: - In the frameobject, remove f_builtins and f_globals, add f_namespace that refers to the module. Make f_globals and f_builtins into properties. - Change ceval to use f_namespace, rather than carrying around globals and builtins. Change functions that take globals as a dict so they also accept a module object. - Change the module object to keep track of the builtins for it. - Change funcobject (e.g. PyFunction_NewWithQualName) to accept 'globals' as a module object - When given a dict and we now expect a module object, create an anonymous module to wrap it. This part is a bit tricky due to reference cycles and speed requirements. However, my hope is that if the internals of CPython can be made to pass modules instead of dicts around, these anonymous modules will become a rare case and so won't matter if they are a little slower. So, what is the purpose of all this trouble? - I believe quite a lot of Python internals can be simpler. For example, importlib is complicated by the fact that a dict is passed around when most of the logic would prefer to have the module. Grubbing in sys.modules to lookup the module object is ugly. The expression "exec(code, module)" is elegant to me. - I believe it will be possible to make global variable access work similar to fast locals. I.e. have each module contain a indexed list of global names, and use an array lookup rather than a dict lookup in the normal case. For backwards compatibility, my idea is to keep a flag on the module object noting if fast global behavior is okay. If someone grabs the module dict, e.g. using module.__dict__ , vars() or frame.f_globals then we clear the flag and revert to the slow dict behavior. Also, if a piece of code is executed in a different namespace, we have to revert to the slow case. That does not seem so hard to do. - I want to have properties for module globals and have them work from within the module. I.e. changing something from a global variable to a property should not require changes to other module code. I.e. LOAD_GLOBAL should trigger a property get. Regards, Neil From chris.barker at noaa.gov Tue Nov 14 19:48:40 2017 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Tue, 14 Nov 2017 16:48:40 -0800 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> <5A087DE7.4050306@brenbarn.net> Message-ID: <-8049199204044509553@unknownmsgid> On Nov 13, 2017, at 1:53 PM, Paul Moore wrote: In principle, I agree with the ideas here, but there are some practical issues that make them somewhat less straightforward than we might like. And practically beats principally ;-) But yeah. 1) Add python2.exe and python3.exe files to the Windows installers The only possible way we'd get a python2.exe is by adding it to future releases of Python 2.7, and we shouldn't be recommending Python 2 for new users anyway. No, but it?s going to be in use for a while yet (2020, yes?). And no, new users should not install py3, but new users may work in an org that is using py2. And I'm strongly -1 on promoting "python3.exe" as the canonical way of getting Python 3. So what is canonical? Not the py executable, for sure. *maybe* we are at a point where we expect py3 to be the default on most systems, but we DO need a way to tell people how to get py3 if: Typing python gets them py2. And it?s not an option to remove py2 or re-configure their systems to change the default ( see your point below about Windows PATH priorities) The python3 command is a result of the Unix issues around switching to Python 3 as the default, and we shouldn't perpetuate that approach on Windows, where it's unneeded. But the exact same issues are there in windows (except the shebang line) bit it?s pretty weird that py.exe respects ?python3?, when there is no python3.exe on the system... And having multiple systems work the same way IS a laudable goal. Having said that, I don't object to including versioned executables as a convenience (if it is one). I just dislike promoting the use of them via standard documentation. Totally agree. 1a) alternatively, we could add a "py" executable to the standard linux builds, so there would be THAT one way to do it. But I think that's a "BAD IDEA" -- I think that getting a "py" launcher on Unix is a lost cause at this stage. Fine ? I never liked the idea :-) Then "python2 -m pip install" would work everywhere (only with new installations, but at least with newbies, that's a bit more likely ...) No newcomer should *ever* be getting told to use "python2 -m pip install". Maybe "python3 -m pip install", Exactly ? that was supposed to be an example. Both should work, but probably: Python -m pip install would be recommended, with ?you can add 2 or 3 if it doesn?t do the right thing. And of course not all Unix distributions come with Python 3 installed Exactly ? which is why even newbies that install just python 3 end up with both on their system. (Mac OS being an obvious example, and I think Nick mentioned CentOS and RHEL) so python3 won't work everywhere either... But it will give you an error, rather that python 2. That?s exactly what we want! 2) Make adding to the PATH in Windows the default. ... Unfortunately, adding Python to the *front* of PATH is not possible. Well, darn. But we can still make it the default and it will sometimes work. And all the more reason we need a ?python3? executable. There's also the problem that file associations (i.e., what does double clicking on a .py/.pyw file do) don't follow the same rules, as they go through the launcher. I don?t see anything we can do with that. But I don?t recommend users use that anyway. The only really sensible "do what I expect" situation is a single Python 3 installation on the user's machine. Sure ? but we can?t control that. All we can do is mitigate the damage. For that case, adding Python to PATH (it doesn't matter where as there's only one) is a sensible option. The downside is that optimising for that case makes the behaviour for other (more experienced) users worse. But it's not unreasonable to make that trade-off. Agreed. And not much worse. 3) Make --user be be automatic for pip install. Not actually the default, but pip could do a user install if you don't have the permissions for a non-user install. The problem here is that the user scripts directory isn't on PATH. How does that have anything to do with it? Either they can launch pip or they can?t (the above are about that) this is about what should happen when they do launch pip. The various options here are under discussion on the pip tracker. I?ll try to go there to comment. but there are significant backward compatibility and behaviour change issues to address first. Always the challenge! python myscript.py pip install something_or_other "just work" if the user previously had no Python on their PC. And as many other configurations as possible :-) I think the changes we should make should emphasize two things that perhaps haven?t been given enough priority in the past: 1) things should work as much the same as possible across platforms. 2) making it easier for newbies is more important that for more experienced folks. It has a whole section at https://docs.python.org/3/using/windows.html#python-launcher-for-windows. What I can tell you is that I knew what I was looking for, and didn?t find in more than a few google searches :-( Ultimately, though, most newcomers don't read the docs. They muddle through, ask colleagues who may or may not know any better than they do, read out of date articles on Stack Overflow ... Given this, in fact, any changes to documentation and recommended best practices won?t filter through ?till after py2?s end of life ... :-( So really, if we can change behavior to better match what is already documented ? we should. Too late to change best practices. -CHB -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Tue Nov 14 20:12:10 2017 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 14 Nov 2017 17:12:10 -0800 Subject: [Python-ideas] A proliferation of (un-)Pythonically programmatic pragmas In-Reply-To: References: Message-ID: On Mon, Nov 13, 2017 at 5:37 PM, Barry Warsaw wrote: > One of the things that bother me about end-line comments is that this is > going to blow up line length limits. Is that a problem? the linters can ignore lines to long if they are only too long due to one of these :-) > I think this could work if such > pragma comments could apply to the following line, and multiline pragmas > would be acceptable. Then you could have something like: > > # flake8: disable=unused-import > # mypy: alias=pathlib2.Path > # coverage: ignore=when>py2.7 > I'd much rather have the tools all look at the entire comment for their particular pragma. and then, even if you wanted to put it on the previous line, you would't have the line-length issue: # flake8: disable=unused-import mypy: alias=pathlib2.Path coverage: ignore=when>py2.7 -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Tue Nov 14 21:14:57 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 15 Nov 2017 13:14:57 +1100 Subject: [Python-ideas] A proliferation of (un-)Pythonically programmatic pragmas In-Reply-To: References: Message-ID: <20171115021457.GM19802@ando.pearwood.info> On Tue, Nov 14, 2017 at 05:12:10PM -0800, Chris Barker wrote: > On Mon, Nov 13, 2017 at 5:37 PM, Barry Warsaw wrote: > > > One of the things that bother me about end-line comments is that this is > > going to blow up line length limits. > > > Is that a problem? the linters can ignore lines to long if they are only > too long due to one of these :-) Yes, it is a problem. See below. > > I think this could work if such > > pragma comments could apply to the following line, and multiline pragmas > > would be acceptable. Then you could have something like: > > > > # flake8: disable=unused-import > > # mypy: alias=pathlib2.Path > > # coverage: ignore=when>py2.7 +1 to this. > I'd much rather have the tools all look at the entire comment for their > particular pragma. I have no objection to tooling supporting that as well. > and then, even if you wanted to put it on the previous line, you would't > have the line-length issue: > > # flake8: disable=unused-import mypy: alias=pathlib2.Path coverage: > ignore=when>py2.7 And the irony is that this shows exactly why it is still a problem. Code gets emailed, and lines get wrapped, breaking the too-long line. Even today, we're not entirely paperless, and code gets printed, either trunctating or wrapping too long lines. And of course, code gets read, including comments, and too long lines are, well, too long. -- Steven From ncoghlan at gmail.com Wed Nov 15 01:44:34 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 15 Nov 2017 16:44:34 +1000 Subject: [Python-ideas] Modules as global namespaces rather than dicts In-Reply-To: <20171114203407.dnw3ghecyz3qsow3@python.ca> References: <20171114203407.dnw3ghecyz3qsow3@python.ca> Message-ID: On 15 November 2017 at 06:34, Neil Schemenauer wrote: > So, what is the purpose of all this trouble? > > - I believe quite a lot of Python internals can be simpler. For > example, importlib is complicated by the fact that a dict is > passed around when most of the logic would prefer to have the > module. Grubbing in sys.modules to lookup the module object is > ugly. The expression "exec(code, module)" is elegant to me. > I like the idea in principle, but highlighting a particular backwards compatibility pain point: one of the complications in importlib is that we promise to keep the "sys.modules[__name__] = some_other_object" idiom working. That means the need to do that check exists regardless of whether importlib is passing the module itself around, or the module's dict. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Nov 15 01:56:16 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 15 Nov 2017 16:56:16 +1000 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <76a18ed7-1a0f-5a8e-b7f6-69fc48a764aa@gmail.com> Message-ID: On 15 November 2017 at 01:23, Stephan Houben wrote: > Hi Nick, > > 2017-11-14 11:07 GMT+01:00 Nick Coghlan : > >> On 14 November 2017 at 16:47, Michel Desmoulin > > wrote: >> >>> Proposal A: >>> ----------- >>> >>> Suffix Python executable on Windows like on Unix, so that people will >>> type pythonX.X if they want a specify version. >>> >>> Pros: easy and discoverable. >>> >>> Cons: you need a lot of stuff in the system path. >>> >> >> Con: we hope to have the problem resolved on the Linux distro side such >> that "python" typically means "python" by the time community support for >> Python 2 ends in 2020. Since Windows has gone the better part of two >> decades without version Python commands, adding them because we're >> impatient with the pace of change at the Linux distro level doesn't really >> make sense (especially when Linux holds such a small fraction of the >> non-phone client device market). >> > > Perhaps I could sell you on the idea of a Windows "python3" executable, > not as the New Official Way to do things, > but rather as a pragmatic measure to make code from those Linux weirdos > ;-) > Aye, I'm not opposed to adding pythonX and pythonX.Y binaries or symlinks to the Windows installers as a way to make Linux-assuming tutorials slightly more likely to work for Windows users. I'm only opposed to promoting that as the new preferred way of launching Python from the Windows command line, if for no other reason than if the release after Python 3.9 actually does end up being Python 4.0, then we only have around 5 more years of "the latest Python" and "the latest Python 3.x" being the same thing :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From gadgetsteve at live.co.uk Wed Nov 15 01:13:11 2017 From: gadgetsteve at live.co.uk (Steve Barnes) Date: Wed, 15 Nov 2017 06:13:11 +0000 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> Message-ID: >>> We still have to deal with the fact that basically every Unix >>> environment is "advanced" in the above sense (the python2/python3 >>> split). I don't have a solution for that (other than "upgrade to >>> Windows" ;-)). >> >> Provide the "py" command on linux and mac. And make it the default >> recommended way in the documentation. > > +1 from me. > How about update pip so that, from the command line it can accept pretty much the same syntax as the py launcher: - "python -m pip operation ..." all use the same python as is running the pip module. - "pip -X[.Y][-32|-64] operation ..." tries to find a python matching -X[.Y][-32|-64] and if it succeeds executes "python -m pip operation ..." with that python, (if it doesn't find a matching python is should fail with a sensible error message and possibly list the valid python specifiers). "pip operation ..." searches the available python contexts and, if there is only one, (or in a venv), calls "python -m pip operation ..." using it but, if there is more than one candidate, (or in not in a venv), tells the user that they have more than one python and need to specify which, listing the options. "pip -0" searches the available pythons and lists the available contexts. Ideally the list contexts operations would differentiate between user space and system contexts and display them suitably based on the current user privileges. This would: - maintain the current behaviour as much as possible, - not break all of the existing examples in the wild & literature - avoid trying to introduce a new launcher command on Mac/Linux, - propagate reasonably quickly, as people are already fairly used to being prompted to update pip, - Automatically propagate to the default pip as long as it was back ported to at least 2.7, possibly further. - Allow users who do need to support multiple python installations to do so, (I am one of those). - Does not require updates to python itself. -- Steve (Gadget) Barnes Any opinions in this message are my personal opinions and do not reflect those of my employer. --- This email has been checked for viruses by AVG. http://www.avg.com From ncoghlan at gmail.com Wed Nov 15 03:22:12 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 15 Nov 2017 18:22:12 +1000 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> Message-ID: On 15 November 2017 at 16:13, Steve Barnes wrote: > - "pip -X[.Y][-32|-64] operation ..." tries to find a python matching > -X[.Y][-32|-64] and if it succeeds executes "python -m pip operation > ..." with that python, (if it doesn't find a matching python is should > fail with a sensible error message and possibly list the valid python > specifiers). > This is a genuinely interesting option, especially as `pipenv` has already implemented a feature somewhat akin to this: https://docs.pipenv.org/basics.html#specifying-versions-of-python `pipenv` also allows `pipenv --two` and `pipenv --three` when setting up your initial virtual environment. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From contact at brice.xyz Wed Nov 15 04:00:52 2017 From: contact at brice.xyz (Brice Parent) Date: Wed, 15 Nov 2017 10:00:52 +0100 Subject: [Python-ideas] A proliferation of (un-)Pythonically programmatic pragmas In-Reply-To: References: Message-ID: <24d1bfc9-aca3-2633-8967-68379b54ca47@brice.xyz> I think, if it belongs to a python file, it has to follow the same rules as the rest of the file: - if we follow the pep8, less-than-80 chars remains the rule, whether we are in a comment or not - if the first of 3 following codes works, all three should (the 3 comments refer to the line of code): some_function(foo, bar)? # coverage: ignore=when>py2.7, alias=pathlib2.Path # coverage: ignore=when>py2.7, alias=pathlib2.Path some_function(foo, bar) # coverage: ignore=when>py2.7 # alias=pathlib2.Path some_function(foo, bar) Also, having these pragmas instructions should not prevent classic commenting in any ways, or make them harder to write or understand, so those should all be valid: # doing something important here some_function(foo, bar)? # coverage: ignore=when>py2.7, alias=pathlib2.Path # coverage: ignore=when>py2.7, alias=pathlib2.Path some_function(foo, bar)# doing something important here # coverage: ignore=when>py2.7 # alias=pathlib2.Path # doing something important here some_function(foo, bar) or (what's better for readability?) # doing something important here # coverage: ignore=when>py2.7 # alias=pathlib2.Path some_function(foo, bar) Also, why not adopt the same syntax that is already quite commonly used for todos in many languages, with "@"? # @todo: write tests for this function! @coverage: ignore=when>py2.7 @alias=pathlib2.Path some_function(foo, bar) -Brice Le 15/11/17 ? 02:12, Chris Barker a ?crit?: > On Mon, Nov 13, 2017 at 5:37 PM, Barry Warsaw > wrote: > > One of the things that bother me about end-line comments is that > this is > going to blow up line length limits. > > > Is that a problem? the linters can ignore lines to long if they are > only too long due to one of these :-) > > I think this could work if such > pragma comments could apply to the following line, and multiline > pragmas > would be acceptable.? Then you could have something like: > > ? ? # flake8: disable=unused-import > ? ? # mypy: alias=pathlib2.Path > ? ? # coverage: ignore=when>py2.7 > > > I'd much rather have the tools all look at the entire comment for > their particular pragma. > > and then, even if you wanted to put it on the previous line, you > would't have the line-length issue: > > # flake8: disable=unused-import mypy: alias=pathlib2.Path coverage: > ignore=when>py2.7 > > -CHB > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R ? ? ? ? ? ?(206) 526-6959?? voice > 7600 Sand Point Way NE ??(206) 526-6329?? fax > Seattle, WA ?98115 ? ? ??(206) 526-6317?? main reception > > Chris.Barker at noaa.gov > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From k7hoven at gmail.com Wed Nov 15 04:43:19 2017 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Wed, 15 Nov 2017 11:43:19 +0200 Subject: [Python-ideas] Modules as global namespaces rather than dicts In-Reply-To: References: <20171114203407.dnw3ghecyz3qsow3@python.ca> Message-ID: On Wed, Nov 15, 2017 at 8:44 AM, Nick Coghlan wrote: > > I like the idea in principle, but highlighting a particular backwards > compatibility pain point: one of the complications in importlib is that we > promise to keep the "sys.modules[__name__] = some_other_object" idiom > working. That means the need to do that check exists regardless of whether > importlib is passing the module itself around, or the module's dict. > > > Another point, perhaps more difficult to address: Would for instance globals() then return a module instead of a dict/mapping?? ???Koos? -- + Koos Zevenhoven + http://twitter.com/k7hoven + -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Wed Nov 15 04:51:42 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 15 Nov 2017 09:51:42 +0000 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> Message-ID: On 15 November 2017 at 08:22, Nick Coghlan wrote: > On 15 November 2017 at 16:13, Steve Barnes wrote: >> >> - "pip -X[.Y][-32|-64] operation ..." tries to find a python matching >> -X[.Y][-32|-64] and if it succeeds executes "python -m pip operation >> ..." with that python, (if it doesn't find a matching python is should >> fail with a sensible error message and possibly list the valid python >> specifiers). > > > This is a genuinely interesting option, especially as `pipenv` has already > implemented a feature somewhat akin to this: > https://docs.pipenv.org/basics.html#specifying-versions-of-python > > `pipenv` also allows `pipenv --two` and `pipenv --three` when setting up > your initial virtual environment. This is an interesting idea for *any* tool that's about "working with Python environments" as opposed to "writing Python code". So pip, virtualenv, tox, pipenv, ... Many of these tools are variously reinventing "tell me which Python environment to work on" options. Having a common way to do this would be really useful. I'm not sure how likely it would be for pip to be able to use it (pip introspects sys.executable to get site-packages, etc), but it's certainly a possibility. Having a standardised library/wrapper that handles the "select a Python environment" process would be a huge plus. There's https://github.com/zooba/pep514tools which is a start on handling the Windows registry scanning logic, and building on that to include Unix and anything else we needed would be great. Paul From ncoghlan at gmail.com Wed Nov 15 05:30:27 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 15 Nov 2017 20:30:27 +1000 Subject: [Python-ideas] A proliferation of (un-)Pythonically programmatic pragmas In-Reply-To: References: <20171113234830.GK19802@ando.pearwood.info> Message-ID: On 14 November 2017 at 10:27, Brett Cannon wrote: > > On Mon, Nov 13, 2017, 15:55 Steven D'Aprano, wrote: > >> On Mon, Nov 13, 2017 at 06:37:05PM -0500, Barry Warsaw wrote: >> > Brett Cannon wrote: >> > >> > > And possibly the easiest way to reach them is on the pyqa-dev mailing >> list. >> > >> > What's that? I can't find it on python.org, Gmane, or the Googles. >> >> Brett may have meant >> >> https://mail.python.org/mailman/listinfo/code-quality > > > Steve's right. http://meta.pycqa.org/en/latest/ > I think that will be the right way to go about it, as this seems similar to a lot of the problems that arise in the packaging interoperability space: what really matters is what tool developers support in practice, so documenting a proposed convention without a clear implementation plan across common tools wouldn't really achieve a great deal. This means that any new convention ideas need to be discussed and supported by the tool developers, with python-dev & python-ideas really only getting involved if some standard library adjustments are needed (e.g. updating doctest to abide by any proposed conventions), or if the convention is going to be captured as an informational PEP, rather than solely as a recommended comment parsing library. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Wed Nov 15 05:31:06 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 15 Nov 2017 12:31:06 +0200 Subject: [Python-ideas] Modules as global namespaces rather than dicts In-Reply-To: <20171114203407.dnw3ghecyz3qsow3@python.ca> References: <20171114203407.dnw3ghecyz3qsow3@python.ca> Message-ID: 14.11.17 22:34, Neil Schemenauer ????: > This is an idea I have been playing with and seems to hold some > promise. I think we should use a module instance as the standard > global namespace rather than directly using its dict. I have a > prototype version of CPython that does this, not working 100% yet > though. Wouldn't this harm a performance? Especially after implementing PEP 562. From ncoghlan at gmail.com Wed Nov 15 05:50:20 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 15 Nov 2017 20:50:20 +1000 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> Message-ID: On 15 November 2017 at 19:51, Paul Moore wrote: > On 15 November 2017 at 08:22, Nick Coghlan wrote: > > On 15 November 2017 at 16:13, Steve Barnes > wrote: > >> > >> - "pip -X[.Y][-32|-64] operation ..." tries to find a python matching > >> -X[.Y][-32|-64] and if it succeeds executes "python -m pip operation > >> ..." with that python, (if it doesn't find a matching python is should > >> fail with a sensible error message and possibly list the valid python > >> specifiers). > > > > > > This is a genuinely interesting option, especially as `pipenv` has > already > > implemented a feature somewhat akin to this: > > https://docs.pipenv.org/basics.html#specifying-versions-of-python > > > > `pipenv` also allows `pipenv --two` and `pipenv --three` when setting up > > your initial virtual environment. > > This is an interesting idea for *any* tool that's about "working with > Python environments" as opposed to "writing Python code". So pip, > virtualenv, tox, pipenv, ... Many of these tools are variously > reinventing "tell me which Python environment to work on" options. > Having a common way to do this would be really useful. I'm not sure > how likely it would be for pip to be able to use it (pip introspects > sys.executable to get site-packages, etc), but it's certainly a > possibility. > > Having a standardised library/wrapper that handles the "select a > Python environment" process would be a huge plus. There's > https://github.com/zooba/pep514tools which is a start on handling the > Windows registry scanning logic, and building on that to include Unix > and anything else we needed would be great. > While I suspect pep514tools itself may not end up being the right place, I filed https://github.com/zooba/pep514tools/issues/2 to start discussing that idea further. For other platforms, just naively scanning `PATH` is often good enough, so it's mainly Windows where discovery needs to be a bit more aware of platform specific details (i.e. PEP 514's registry entries). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From k7hoven at gmail.com Wed Nov 15 06:41:17 2017 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Wed, 15 Nov 2017 13:41:17 +0200 Subject: [Python-ideas] Terminology of types / typing [was: PEP 560 (second post)] Message-ID: Here are some thoughts??maybe even a proposal??for type-related terminology, because clear terminology makes discussion and reasoning easier, and helps avoid errors. (And related to the PEP 560 thread, the question of what should go into class attributes like __bases__). Terminology regarding types is confusing enough, and of all the terminology I've seen, I like these most: * concrete type: something that concretely implements data storage or functionality. Usually this is a normal class. * abstract type: an assumption or set of assumptions made about an instance. While both kinds can (and probably should) have a corresponding runtime representation, there's not that much that we can currently assume about the second kind??abstract types. They could be almost anything. ?Anyway, regarding PEP 560, in my speculated naming, the __bases__ attribute of a class would contain both concrete and abstract bases. Those with any concrete method implementations should go in the mro. But then there's the problem that "abstract base classes" may contain both concrete and abstract methods. What do we call such a "type"? Maybe we have both "concrete" and "strictly concrete" types. Perhaps we also have both "abstract" and "strictly abstract" types. An ABC with some concrete default implementations might then be both a concrete type and an abstract type. Note that in the above bullet point "definition" of concrete type, I intentionally left out the requirement that the type can be instantiated. The other two bullet points are: * strictly concrete type: a concrete type that is not abstract??it concretely implements everything that it represents / describes. This is almost always a normal class, so it might be also known as "class". * strictly abstract type: an abstract type that is not concrete??it does not implement any functionality or storage. ?There might be a way to improve terminology from this, but I feel that what I sketched here is usable but still not very ambiguous. ???Koos? -- + Koos Zevenhoven + http://twitter.com/k7hoven + -------------- next part -------------- An HTML attachment was scrubbed... URL: From k7hoven at gmail.com Wed Nov 15 06:45:07 2017 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Wed, 15 Nov 2017 13:45:07 +0200 Subject: [Python-ideas] PEP 560 (second post) In-Reply-To: References: Message-ID: On Tue, Nov 14, 2017 at 4:49 AM, Nick Coghlan wrote: > On 14 November 2017 at 09:41, Guido van Rossum wrote: > > > > Thanks, I am happy now with the PEP, except for one detail: maybe > > `__mro_entry__` should always return a tuple and then maybe renamed to > > `__mro_entries__`. (See debate at > > https://github.com/python/peps/pull/460#issuecomment-343969528 .) > > I like that - very nice refinement. > > ?I hope the order in which multiple __mro_entries__ will appear in the mro will be documented clearly, regardless of how obvious it might feel.? It might take a while, before anyone notices that something weird happens because they did it the wrong way around. ?Out of curiosity, what kind of cases would benefit from __mro__entries__ being able to return two ?or more ? entries? Also, I'm still wondering about __bases__ and __orig_bases__. Could we call these __concrete_bases__ and __bases__ instead (respectively)? For an explanation of why I think this might be a good idea, see this new thread: https://mail.python.org/pipermail/python-ideas/2017-November/047896.html ???Koos ? -- + Koos Zevenhoven + http://twitter.com/k7hoven + -------------- next part -------------- An HTML attachment was scrubbed... URL: From k7hoven at gmail.com Wed Nov 15 06:59:22 2017 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Wed, 15 Nov 2017 13:59:22 +0200 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> Message-ID: On Nov 12, 2017 14:21, "Paul Moore" wrote: [....] > Virtualenvs are a hard tool to use for beginners. Agreed. Genuine beginners just install Python, then use it. If they install extra packages, they want them to be available to all of their scripts. Agreed. >From personal experience, ?I do have to say that for my day-to-day normal python *use*, I try to use a single environment (the default miniconda one) for everything. Most of the time, that's enough. I suppose that makes me a genuine beginner :-). Being able to conveniently installing a package into the current environment from within any interactive interpreter would definitely improve the beginner experience, I would say. ??Koos [..] -------------- next part -------------- An HTML attachment was scrubbed... URL: From desmoulinmichel at gmail.com Wed Nov 15 07:46:33 2017 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Wed, 15 Nov 2017 13:46:33 +0100 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> <5A087DE7.4050306@brenbarn.net> Message-ID: Le 13/11/2017 ? 19:57, Chris Barker a ?crit?: > This has gotten to be a big thread, and it's a pretty intractable > problem, but I think there are a few fairly small things that could be > done to at least make it a bit easier: > > 1) Add python2.exe and python3.exe files to the Windows installers -- am > I insane or did Windows used to have that? I really think it did -- > maybe got removed when py.exe was added. More than that. Add pythonX.X. You may want to run Python3.5 and not Python3.6. > ? 1a) alternatively, we could add a "py" executable to the standard > linux builds, so there would be THAT one way to do it. But I think > that's a "BAD IDEA" -- the whole "py" thing is not widely know or used, > it's not going to show up in package install instructions for a LONG > time, (actualy we could do both anyway) It would work better and be cleaner, but is unlikely to happen. 1) is simpler and have a chance to happen. > Then "python2 -m pip install" would work everywhere (only with new > installations, but at least with newbies, that's a bit more likely ...) > > Yeah, if 1) happens, we should promote -m for pip. > 2) Make adding to the PATH in Windows the default. I think there are > really three user groups: > > ? ?- newbies starting from scratch -- they want it on the PATH > > ? ?- newbies with whatever left over cruft from previous installations > on their systems -- they want it at the FRONT of their PATH. They > SHOULD? uninstall all the cruft, but if they don't this will still work > with as few surprises a possible. > > ? ?- not-newbies with a previous version of python they need to continue > using. They can uncheck the box, or use py.exe Agreed. But if an existing python exist in the path, there should be a warning. > > > 3) Make --user be be automatic for pip install. Not actually the > default, but pip could do a user install if you don't have the > permissions for a non-user install. Breaking compat ? Not sure people will accept. ===== Should I do a PEP with a summary of all the stuff we discussed ? From ncoghlan at gmail.com Wed Nov 15 09:17:25 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 16 Nov 2017 00:17:25 +1000 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> <5A087DE7.4050306@brenbarn.net> Message-ID: On 15 November 2017 at 22:46, Michel Desmoulin wrote: > Le 13/11/2017 ? 19:57, Chris Barker a ?crit : > > 3) Make --user be be automatic for pip install. Not actually the > > default, but pip could do a user install if you don't have the > > permissions for a non-user install. > > Breaking compat ? Not sure people will accept. > This isn't actually a Python level change - it's a pip UX level one, and already on their todo list: https://github.com/pypa/pip/issues/1668 I believe the next concrete step that can be taken there is to actually add an explicit `--global` flag, so I've belatedly broken that out as its own issue: https://github.com/pypa/pip/issues/4865 ===== > > Should I do a PEP with a summary of all the stuff we discussed ? > I think a Windows-specific PEP covering adding PATH updates back to the default installer behaviour, and adding pythonX and pythonX.Y commands would be useful (and Guido would presumably delegate resolving that to Steve Dower as the Windows installer maintainer). The one thing I'd ask is that any such PEP *not* advocate for promoting ther variants as the preferred way of invoking Python on Windows - rather, they should be positioned as a way of making online instructions written for Linux more likely to "just work" for folks on Windows (similar to the utf-8 encoding changes in https://www.python.org/dev/peps/pep-0529/) Instead, the focus should be on ensuring the "python -m pip install" and "pip install" both work after clicking through the installer without changing any settings, and devising a troubleshooting guide to help folks that are familiar with computers and Python, but perhaps not with Windows, guide folks to a properly working environment. The update to the Windows installer would then start offering benefits as soon as Python 3.7 becomes the default download, while the troubleshooting guide would be beneficial as soon as folks start learning about it's existence. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From k7hoven at gmail.com Wed Nov 15 11:54:25 2017 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Wed, 15 Nov 2017 18:54:25 +0200 Subject: [Python-ideas] Terminology of types / typing [was: PEP 560 (second post)] In-Reply-To: References: Message-ID: On Wed, Nov 15, 2017 at 1:41 PM, Koos Zevenhoven wrote: [..] > What do we call such a "type"? Maybe we have both "concrete" and "strictly > concrete" types. Perhaps we also have both "abstract" and "strictly > abstract" types. An ABC with some concrete default implementations might > then be both a concrete type and an abstract type. > > Note that in the above bullet point "definition" of concrete type, I > intentionally left out the requirement that the type can be instantiated. > > The other two bullet points are: > > * strictly concrete type: a concrete type that is not abstract??it > concretely implements everything that it represents / describes. This is > almost always a normal class, so it might be also known as "class". > > * strictly abstract type: an abstract type that is not concrete??it does > not implement any functionality or storage. > > ?There might be a way to improve terminology from this, but I feel that > what I sketched here is usable but still not very ambiguous. > > ?Let me rephrase that last sentence: I think this terminology is more clear. And here's some additional clarification: I expect that the possibility of a type being both concrete and abstract may sound strange. In some ways it is indeed strange, but this overlap of concepts definitely exists already, we just need to categorize and define the concepts clearly, but without introducing too many concepts whose relations to each other are messy. This might also seem strange if you are not used to how "strict" is often used in mathematics and related sciences. Essentially, it's synonymous to "proper"?. For example, "strict subset" and "proper subset" of a set both refer to a subset that is not the set itself. Any set is both a superset and subset of itself (in non-strict terms). Also "pure" might sometimes refer to something similar. So in some sense it means excluding the "gray area". Often that "gray area" is kept part of the non-strict/improper concept for convenience. ? ??Koos ? -- + Koos Zevenhoven + http://twitter.com/k7hoven + -------------- next part -------------- An HTML attachment was scrubbed... URL: From nas-python-ideas at arctrix.com Wed Nov 15 12:28:36 2017 From: nas-python-ideas at arctrix.com (Neil Schemenauer) Date: Wed, 15 Nov 2017 11:28:36 -0600 Subject: [Python-ideas] Modules as global namespaces rather than dicts In-Reply-To: References: <20171114203407.dnw3ghecyz3qsow3@python.ca> Message-ID: <20171115172836.thj5saap7bzcolp2@python.ca> On 2017-11-15, Koos Zevenhoven wrote: > Another point, perhaps more difficult to address: Would for instance > globals() then return a module instead of a dict/mapping?? For compatibility, it would definitely have to return a dict. As a result, calling globals() would cause the "fast globals" flag to be cleared. My hope is that getting a reference to the module dict from user code is the exception rather than the norm. I.e. most modules would never have globals() called within their code. If it is too common, the flag will be cleared on most modules and we won't gain much speed. From levkivskyi at gmail.com Wed Nov 15 13:41:02 2017 From: levkivskyi at gmail.com (Ivan Levkivskyi) Date: Wed, 15 Nov 2017 19:41:02 +0100 Subject: [Python-ideas] Terminology of types / typing [was: PEP 560 (second post)] In-Reply-To: References: Message-ID: At some point it was proposed to distinguish two things: types (static) and classes (runtime). I don't think we need more fine grained terminology here. -- Ivan On 15 November 2017 at 17:54, Koos Zevenhoven wrote: > On Wed, Nov 15, 2017 at 1:41 PM, Koos Zevenhoven > wrote: > [..] > >> What do we call such a "type"? Maybe we have both "concrete" and >> "strictly concrete" types. Perhaps we also have both "abstract" and >> "strictly abstract" types. An ABC with some concrete default >> implementations might then be both a concrete type and an abstract type. >> >> Note that in the above bullet point "definition" of concrete type, I >> intentionally left out the requirement that the type can be instantiated. >> >> The other two bullet points are: >> >> * strictly concrete type: a concrete type that is not abstract??it >> concretely implements everything that it represents / describes. This is >> almost always a normal class, so it might be also known as "class". >> >> * strictly abstract type: an abstract type that is not concrete??it does >> not implement any functionality or storage. >> >> ?There might be a way to improve terminology from this, but I feel that >> what I sketched here is usable but still not very ambiguous. >> >> > ?Let me rephrase that last sentence: I think this terminology is more > clear. > > And here's some additional clarification: > > I expect that the possibility of a type being both concrete and abstract > may sound strange. In some ways it is indeed strange, but this overlap of > concepts definitely exists already, we just need to categorize and define > the concepts clearly, but without introducing too many concepts whose > relations to each other are messy. > > This might also seem strange if you are not used to how "strict" is often > used in mathematics and related sciences. Essentially, it's synonymous to > "proper"?. For example, "strict subset" and "proper subset" of a set both > refer to a subset that is not the set itself. Any set is both a superset > and subset of itself (in non-strict terms). Also "pure" might sometimes > refer to something similar. > > So in some sense it means excluding the "gray area". Often that "gray > area" is kept part of the non-strict/improper concept for convenience. > ? > ??Koos > ? > > -- > + Koos Zevenhoven + http://twitter.com/k7hoven + > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve.dower at python.org Wed Nov 15 14:07:31 2017 From: steve.dower at python.org (Steve Dower) Date: Wed, 15 Nov 2017 11:07:31 -0800 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> <5A087DE7.4050306@brenbarn.net> Message-ID: <8d5574dc-b614-ccfe-3ce8-f1ba86508179@python.org> On 15Nov2017 0617, Nick Coghlan wrote: > On 15 November 2017 at 22:46, Michel Desmoulin > > wrote: > Should I do a PEP with a summary of all the stuff we discussed ? > I think a Windows-specific PEP covering adding PATH updates back to the > default installer behaviour, and adding pythonX and pythonX.Y commands > would be useful (and Guido would presumably delegate resolving that to > Steve Dower as the Windows installer maintainer). If you write such a PEP, please also research and write up the issues with modifying PATH on Windows (they're largely scattered throughout bugs.p.o and earlier discussions on python-dev). Once you realise the tradeoff involved in modifying these global settings, you'll either come around to my point of view or be volunteering to take *all* the support questions when they come in :) > The one thing I'd ask is that any such PEP *not* advocate for promoting > ther variants as the preferred way of invoking Python on Windows - > rather, they should be positioned as a way of making online instructions > written for Linux more likely to "just work" for folks on Windows > (similar to the utf-8 encoding changes in > https://www.python.org/dev/peps/pep-0529/) > > Instead, the focus should be on ensuring the "python -m pip install" and > "pip install" both work after clicking through the installer without > changing any settings, and devising a troubleshooting guide to help > folks that are familiar with computers and Python, but perhaps not with > Windows, guide folks to a properly working environment. My preferred solution for this is to rename "py.exe" to "python.exe" (or rather, make a copy of it with the new name), and extend (or more likely, rewrite) the launcher such that: * if argv[0] == "py.exe", use PEP 514 company/tag resolution to find and launch Python based on first command line argument * if argv[0] == "python.exe", find the matching PythonCore/ install (where tag may be a partial match - e.g. "python3.exe" finds the latest PythonCore/3.x) * else, if argv[0] == ".exe, find the matching PythonCore/ install and launch "-m " With the launcher behaving like this, we can make as many hard links as we want in its install directory (it only gets installed once, so only needs one PATH entry, and this is C:\Windows for admin installs): * python.exe * python2.exe * python3.exe * python3.6.exe * pip.exe * pip2.exe * pip3.exe As well as allowing e.g. "py.exe -anaconda36-64 ..." to reliably locate and run non-Python.org installs. It needs to be fully specced out, obviously, and we may want to move the all-users install to its own directory to reduce clutter, but part of the reason behind PEP 514 was to enable this sort of launcher. It could even extend to "you don't have this version right now, want to download and install it?" And finally it should be fairly obvious that this doesn't have to be a core Python tool. It has no reliance on anything in core (that isn't already specified in a PEP) and could be written totally independently. I've tried (weakly) to get work time allocated to this in the past, and if it's genuinely not going to get done unless I do it then I'll try again. Cheers, Steve From zachary.ware+pydev at gmail.com Wed Nov 15 14:29:10 2017 From: zachary.ware+pydev at gmail.com (Zachary Ware) Date: Wed, 15 Nov 2017 13:29:10 -0600 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: <8d5574dc-b614-ccfe-3ce8-f1ba86508179@python.org> References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> <5A087DE7.4050306@brenbarn.net> <8d5574dc-b614-ccfe-3ce8-f1ba86508179@python.org> Message-ID: On Wed, Nov 15, 2017 at 1:07 PM, Steve Dower wrote: > My preferred solution for this is to rename "py.exe" to "python.exe" (or > rather, make a copy of it with the new name), and extend (or more likely, > rewrite) the launcher such that: > > * if argv[0] == "py.exe", use PEP 514 company/tag resolution to find and > launch Python based on first command line argument > * if argv[0] == "python.exe", find the matching > PythonCore/ install (where tag may be a partial match - e.g. > "python3.exe" finds the latest PythonCore/3.x) > * else, if argv[0] == ".exe, find the matching > PythonCore/ install and launch "-m " > > With the launcher behaving like this, we can make as many hard links as we > want in its install directory (it only gets installed once, so only needs > one PATH entry, and this is C:\Windows for admin installs): > * python.exe > * python2.exe > * python3.exe > * python3.6.exe > * pip.exe > * pip2.exe > * pip3.exe I haven't been following this thread closely, but this sounds lovely. I'm not terribly keen on cluttering up C:\Windows with this, but that's a minor issue. -- Zach From p.f.moore at gmail.com Wed Nov 15 14:51:18 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 15 Nov 2017 19:51:18 +0000 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> <5A087DE7.4050306@brenbarn.net> <8d5574dc-b614-ccfe-3ce8-f1ba86508179@python.org> Message-ID: On 15 November 2017 at 19:29, Zachary Ware wrote: > On Wed, Nov 15, 2017 at 1:07 PM, Steve Dower wrote: >> My preferred solution for this is to rename "py.exe" to "python.exe" (or >> rather, make a copy of it with the new name), and extend (or more likely, >> rewrite) the launcher such that: >> >> * if argv[0] == "py.exe", use PEP 514 company/tag resolution to find and >> launch Python based on first command line argument >> * if argv[0] == "python.exe", find the matching >> PythonCore/ install (where tag may be a partial match - e.g. >> "python3.exe" finds the latest PythonCore/3.x) >> * else, if argv[0] == ".exe, find the matching >> PythonCore/ install and launch "-m " >> >> With the launcher behaving like this, we can make as many hard links as we >> want in its install directory (it only gets installed once, so only needs >> one PATH entry, and this is C:\Windows for admin installs): >> * python.exe >> * python2.exe >> * python3.exe >> * python3.6.exe >> * pip.exe >> * pip2.exe >> * pip3.exe > > I haven't been following this thread closely, but this sounds lovely. > I'm not terribly keen on cluttering up C:\Windows with this, but > that's a minor issue. Agreed. That sounds sufficiently awesome that I'll try to find some of my essentially-non-existent coding time and largely-forgotten C skills to implement it. Remind me of that promise in 6 months when I've failed to report any progress :-) Paul From rk546394 at gmail.com Wed Nov 15 14:52:54 2017 From: rk546394 at gmail.com (Robert) Date: Wed, 15 Nov 2017 14:52:54 -0500 Subject: [Python-ideas] Greetings from a lurker Message-ID: Hi, I've been lurking for quite a time and just wanted to let you know I'm out here. Thanks for making Python the best! Robert Kaplan From mistersheik at gmail.com Wed Nov 15 16:49:03 2017 From: mistersheik at gmail.com (Neil Girdhar) Date: Wed, 15 Nov 2017 13:49:03 -0800 (PST) Subject: [Python-ideas] Consider (one day) adding an inheritance order class precedence mechanism Message-ID: Sometimes I get MRO failures when defining classes. For example, if R < E, C B < S, E S < C Z < B, R Then Z cannot be instantiated because C precedes E in B and E precedes C in R. The problem is that when the inheritance graph was topologically-sorted in B, the order was S, C, E. It could just as easily have been S, E, C. So, one solution is to add C explicitly to B's base class list, but this is annoying because B might not care about C (it's an implementation detail of S). It also means that if the hierarchy changes, a lot of these added extra base classes need to be fixed. I propose adding a magic member to classes: __precedes__ that is a list of classes. So, the fix for the above problem would be to define E as follows: class E: from whatever import C __precedes__ = [C] Then, when topologically-sorting (so-called linearizing) the ancestor classes, Python can try to ensure that E precedes C when defining B. Best, Neil -------------- next part -------------- An HTML attachment was scrubbed... URL: From k7hoven at gmail.com Wed Nov 15 17:30:34 2017 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Thu, 16 Nov 2017 00:30:34 +0200 Subject: [Python-ideas] Consider (one day) adding an inheritance order class precedence mechanism In-Reply-To: References: Message-ID: On Wed, Nov 15, 2017 at 11:49 PM, Neil Girdhar wrote: > Sometimes I get MRO failures when defining classes. For example, if > > R < E, C > B < S, E > S < C > Z < B, R > > Then Z cannot be instantiated because C precedes E in B and E precedes C > in R. The problem is that when the inheritance graph was > topologically-sorted in B, the order was S, C, E. It could just as easily > have been S, E, C. So, one solution is to add C explicitly to B's base > class list, but this is annoying because B might not care about C (it's an > implementation detail of S). It also means that if the hierarchy changes, > a lot of these added extra base classes need to be fixed. > > I propose adding a magic member to classes: > > __precedes__ that is a list of classes. So, the fix for the above problem > would be to define E as follows: > > class E: > from whatever import C > __precedes__ = [C] > > Then, when topologically-sorting (so-called linearizing) the ancestor > classes, Python can try to ensure that E precedes C when defining B. > > So it sounds like you are talking about the way that the siblings in the inheritance tree (the bases of each class) get "flattened" into the mro in a depth-first manner, and the relative order of siblings is not preserved. What would you think about not topologically sorting the inheritance tree as such, but sorting a graph that has additional edges according to the base lists of each class involved? In this case those edges would be E->C, S->E, B->R. Is this what your talking about, or do I misinterpret the problem? ??-- Koos -- + Koos Zevenhoven + http://twitter.com/k7hoven + -------------- next part -------------- An HTML attachment was scrubbed... URL: From k7hoven at gmail.com Wed Nov 15 17:45:38 2017 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Thu, 16 Nov 2017 00:45:38 +0200 Subject: [Python-ideas] Terminology of types / typing [was: PEP 560 (second post)] In-Reply-To: References: Message-ID: On Wed, Nov 15, 2017 at 8:41 PM, Ivan Levkivskyi wrote: > ?? > At some point it was proposed to distinguish two things: types (static) > and classes (runtime). > I don't think we need more fine grained terminology here. > > ?Yeah, well I was trying to wrap my head around this runtime vs static thing for quite some time, and it never felt quite right. I'm not sure that runtime vs static is a useful distinction at all. In the current situation of typing in python, I think it misses the point slightly. One might also say that, at static-analysis time, all types are static--including normal classes. But those correspond to concepts at runtime--abstract or concrete. -- Koos > -- > Ivan > > > > On 15 November 2017 at 17:54, Koos Zevenhoven wrote: > >> On Wed, Nov 15, 2017 at 1:41 PM, Koos Zevenhoven >> wrote: >> [..] >> >>> What do we call such a "type"? Maybe we have both "concrete" and >>> "strictly concrete" types. Perhaps we also have both "abstract" and >>> "strictly abstract" types. An ABC with some concrete default >>> implementations might then be both a concrete type and an abstract type. >>> >>> Note that in the above bullet point "definition" of concrete type, I >>> intentionally left out the requirement that the type can be instantiated. >>> >>> The other two bullet points are: >>> >>> * strictly concrete type: a concrete type that is not abstract??it >>> concretely implements everything that it represents / describes. This is >>> almost always a normal class, so it might be also known as "class". >>> >>> * strictly abstract type: an abstract type that is not concrete??it does >>> not implement any functionality or storage. >>> >>> ?There might be a way to improve terminology from this, but I feel that >>> what I sketched here is usable but still not very ambiguous. >>> >>> >> ?Let me rephrase that last sentence: I think this terminology is more >> clear. >> >> And here's some additional clarification: >> >> I expect that the possibility of a type being both concrete and abstract >> may sound strange. In some ways it is indeed strange, but this overlap of >> concepts definitely exists already, we just need to categorize and define >> the concepts clearly, but without introducing too many concepts whose >> relations to each other are messy. >> >> This might also seem strange if you are not used to how "strict" is often >> used in mathematics and related sciences. Essentially, it's synonymous to >> "proper"?. For example, "strict subset" and "proper subset" of a set both >> refer to a subset that is not the set itself. Any set is both a superset >> and subset of itself (in non-strict terms). Also "pure" might sometimes >> refer to something similar. >> >> So in some sense it means excluding the "gray area". Often that "gray >> area" is kept part of the non-strict/improper concept for convenience. >> ? >> ??Koos >> ? >> >> -- >> + Koos Zevenhoven + http://twitter.com/k7hoven + >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> > -- + Koos Zevenhoven + http://twitter.com/k7hoven + -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Wed Nov 15 19:18:41 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 16 Nov 2017 11:18:41 +1100 Subject: [Python-ideas] Consider (one day) adding an inheritance order class precedence mechanism In-Reply-To: References: Message-ID: <20171116001841.GN19802@ando.pearwood.info> On Wed, Nov 15, 2017 at 01:49:03PM -0800, Neil Girdhar wrote: > Sometimes I get MRO failures when defining classes. For example, if > > R < E, C > B < S, E > S < C > Z < B, R > > Then Z cannot be instantiated because C precedes E in B and E precedes C in > R. The problem is that when the inheritance graph was topologically-sorted > in B, the order was S, C, E. It could just as easily have been S, E, C. These are not equivalent: B < S, E B < E, S so your comment that it could "just have easily" been S, E, C is not generally true. Calling C.method(self) E.method(self) in that order is not, in general, the same as calling them in the opposite order. > So, one solution is to add C explicitly to B's base class list, but this is > annoying because B might not care about C (it's an implementation detail of > S). It also means that if the hierarchy changes, a lot of these added > extra base classes need to be fixed. > > I propose adding a magic member to classes: > > __precedes__ that is a list of classes. So, the fix for the above problem > would be to define E as follows: > > class E: > from whatever import C > __precedes__ = [C] As the author of E, why would I do that? I don't even know that B exists, and even if I did know about B, by adding __precedes__ I risk breaking classes X, Y and Z which expect C to precede E. > Then, when topologically-sorting (so-called linearizing) the ancestor > classes, Python can try to ensure that E precedes C when defining B. How? You're glossing over the most important detail: *how* should Python produce a consistant, monotonic, bug-free linearization of the superclasses given an arbitrary number of (possibly conflicting) __precedes__ constraints? If __precedes__ is a hard constraint, then you have to deal with inconsistent constraints *as well as* inconsistent inheritence orders. If __precedes__ is just a suggestion, rather than a hard constraint, then you have to deal with the cases where the actual MRO ignores the suggestion, leading to contradiction between what the source says and what the code actually does. In either case, we know have an even more complicated set of inheritence rules, with even more things that can go wrong. Getting the MRO for multiple inheritence right is hard enough already. Python's current MRO algorithm is the third, the first two were broken: http://python-history.blogspot.com.au/2010/06/method-resolution-order.html https://www.python.org/download/releases/2.3/mro/ Letting people mess with the MRO will surely lead to subtle and hard to diagnose bugs. The harsh truth is that sometimes you just do not have a consistent MRO for a certain set of superclasses. Python refuses to let you use such an inconsistent MRO, not because it is trying to be annoying, but because such inconsistent MROs are broken. If you know of an alternative MRO that gives correct results for the class hierarchy you give above, please share. -- Steve From mistersheik at gmail.com Wed Nov 15 22:28:44 2017 From: mistersheik at gmail.com (Neil Girdhar) Date: Wed, 15 Nov 2017 19:28:44 -0800 (PST) Subject: [Python-ideas] Consider (one day) adding an inheritance order class precedence mechanism In-Reply-To: References: Message-ID: <979c2edf-e94c-46ed-bacd-730686ea3999@googlegroups.com> On Wednesday, November 15, 2017 at 5:32:00 PM UTC-5, Koos Zevenhoven wrote: > > On Wed, Nov 15, 2017 at 11:49 PM, Neil Girdhar > wrote: > >> Sometimes I get MRO failures when defining classes. For example, if >> >> R < E, C >> B < S, E >> S < C >> Z < B, R >> >> Then Z cannot be instantiated because C precedes E in B and E precedes C >> in R. The problem is that when the inheritance graph was >> topologically-sorted in B, the order was S, C, E. It could just as easily >> have been S, E, C. So, one solution is to add C explicitly to B's base >> class list, but this is annoying because B might not care about C (it's an >> implementation detail of S). It also means that if the hierarchy changes, >> a lot of these added extra base classes need to be fixed. >> >> I propose adding a magic member to classes: >> >> __precedes__ that is a list of classes. So, the fix for the above >> problem would be to define E as follows: >> >> class E: >> from whatever import C >> __precedes__ = [C] >> >> Then, when topologically-sorting (so-called linearizing) the ancestor >> classes, Python can try to ensure that E precedes C when defining B. >> >> > So it sounds like you are talking about the way that the siblings in the > inheritance tree (the bases of each class) get "flattened" into the mro in > a depth-first manner, and the relative order of siblings is not preserved. > It is preserved, but there are insufficient constraints, which causes problems with future class definitions. > What would you think about not topologically sorting the inheritance tree > as such, but sorting a graph that has additional edges according to the > base lists of each class involved? In this case those edges would be E->C, > S->E, B->R. Is this what your talking about, or do I misinterpret the > problem? > That's already part of the topological sorting algorithm. You have to do that. I'm suggesting additional constraints. > > ??-- Koos > > > -- > + Koos Zevenhoven + http://twitter.com/k7hoven + > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Thu Nov 16 01:10:15 2017 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 16 Nov 2017 19:10:15 +1300 Subject: [Python-ideas] Consider (one day) adding an inheritance order class precedence mechanism In-Reply-To: <20171116001841.GN19802@ando.pearwood.info> References: <20171116001841.GN19802@ando.pearwood.info> Message-ID: <5A0D2BC7.3040209@canterbury.ac.nz> Steven D'Aprano wrote: > These are not equivalent: > > B < S, E > B < E, S Not in general, but in many cases they will be, e.g. if E and S have no method names in common. I think the OP is implying that his case is one of those. Maybe what's really wanted is a way to say "B inherits from S and E, but it doesn't care what order they go in". Then the MRO generating algorithm could in principle swap them if it would result in a consistent MRO. Or maybe the MRO generator could decide for itself if the order of two base classes can be swapped by inspecting their attributes to see if any of them clash? -- Greg From ncoghlan at gmail.com Thu Nov 16 01:49:44 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 16 Nov 2017 16:49:44 +1000 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> <5A087DE7.4050306@brenbarn.net> <8d5574dc-b614-ccfe-3ce8-f1ba86508179@python.org> Message-ID: On 16 November 2017 at 05:29, Zachary Ware wrote: > On Wed, Nov 15, 2017 at 1:07 PM, Steve Dower > wrote: > > My preferred solution for this is to rename "py.exe" to "python.exe" (or > > rather, make a copy of it with the new name), and extend (or more likely, > > rewrite) the launcher such that: > > > > * if argv[0] == "py.exe", use PEP 514 company/tag resolution to find and > > launch Python based on first command line argument > > * if argv[0] == "python.exe", find the matching > > PythonCore/ install (where tag may be a partial match - e.g. > > "python3.exe" finds the latest PythonCore/3.x) > > * else, if argv[0] == ".exe, find the matching > > PythonCore/ install and launch "-m " > > > > With the launcher behaving like this, we can make as many hard links as > we > > want in its install directory (it only gets installed once, so only needs > > one PATH entry, and this is C:\Windows for admin installs): > > * python.exe > > * python2.exe > > * python3.exe > > * python3.6.exe > > * pip.exe > > * pip2.exe > > * pip3.exe > > I haven't been following this thread closely, but this sounds lovely. > I'm not terribly keen on cluttering up C:\Windows with this, but > that's a minor issue. > I'd missed Steve's post before writing my last one. This sounds like a really nice technical solution to me, too, especially as it will handle Python 2 as well (even for Python 2 only systems, the launcher is available as an independently installable executable). Regardless of the underlying implementation details though, a PEP would be a helpful way of writing it up so we can make sure packaging.python.org and other resources properly account for it. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Thu Nov 16 04:05:53 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 16 Nov 2017 09:05:53 +0000 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> <5A087DE7.4050306@brenbarn.net> <8d5574dc-b614-ccfe-3ce8-f1ba86508179@python.org> Message-ID: On 16 November 2017 at 06:49, Nick Coghlan wrote: > On 16 November 2017 at 05:29, Zachary Ware > wrote: >> >> On Wed, Nov 15, 2017 at 1:07 PM, Steve Dower >> wrote: >> > My preferred solution for this is to rename "py.exe" to "python.exe" (or >> > rather, make a copy of it with the new name), and extend (or more >> > likely, >> > rewrite) the launcher such that: >> > >> > * if argv[0] == "py.exe", use PEP 514 company/tag resolution to find and >> > launch Python based on first command line argument >> > * if argv[0] == "python.exe", find the matching >> > PythonCore/ install (where tag may be a partial match - e.g. >> > "python3.exe" finds the latest PythonCore/3.x) >> > * else, if argv[0] == ".exe, find the matching >> > PythonCore/ install and launch "-m " >> > >> > With the launcher behaving like this, we can make as many hard links as >> > we >> > want in its install directory (it only gets installed once, so only >> > needs >> > one PATH entry, and this is C:\Windows for admin installs): >> > * python.exe >> > * python2.exe >> > * python3.exe >> > * python3.6.exe >> > * pip.exe >> > * pip2.exe >> > * pip3.exe >> >> I haven't been following this thread closely, but this sounds lovely. >> I'm not terribly keen on cluttering up C:\Windows with this, but >> that's a minor issue. > > > I'd missed Steve's post before writing my last one. This sounds like a > really nice technical solution to me, too, especially as it will handle > Python 2 as well (even for Python 2 only systems, the launcher is available > as an independently installable executable). > > Regardless of the underlying implementation details though, a PEP would be a > helpful way of writing it up so we can make sure packaging.python.org and > other resources properly account for it. OK, I'll add that to the list of things to look at. Paul From storchaka at gmail.com Thu Nov 16 05:23:07 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 16 Nov 2017 12:23:07 +0200 Subject: [Python-ideas] Ignorable whitespaces in the re.VERBOSE mode Message-ID: Currently the re module ignores only 6 ASCII whitespaces in the re.VERBOSE mode: U+0009 CHARACTER TABULATION U+000A LINE FEED U+000B LINE TABULATION U+000C FORM FEED U+000D CARRIAGE RETURN U+0020 SPACE Perl ignores characters that Unicode calls "Pattern White Space" in the /x mode. It ignores additional 5 non-ASCII characters. U+0085 NEXT LINE U+200E LEFT-TO-RIGHT MARK U+200F RIGHT-TO-LEFT MARK U+2028 LINE SEPARATOR U+2029 PARAGRAPH SEPARATOR The regex module just ignores characters for which str.isspace() returns True. It ignores additional 20 non-ASCII whitespace characters, including characters U+001C..001F whose classification as whitespaces is questionable, but doesn't ignore LEFT-TO-RIGHT MARK and RIGHT-TO-LEFT MARK. U+001C [FILE SEPARATOR] U+001D [GROUP SEPARATOR] U+001E [RECORD SEPARATOR] U+001F [UNIT SEPARATOR] U+00A0 NO-BREAK SPACE U+1680 OGHAM SPACE MARK U+2000 EN QUAD U+2001 EM QUAD U+2002 EN SPACE U+2003 EM SPACE U+2004 THREE-PER-EM SPACE U+2005 FOUR-PER-EM SPACE U+2006 SIX-PER-EM SPACE U+2007 FIGURE SPACE U+2008 PUNCTUATION SPACE U+2009 THIN SPACE U+200A HAIR SPACE U+202F NARROW NO-BREAK SPACE U+205F MEDIUM MATHEMATICAL SPACE U+3000 IDEOGRAPHIC SPACE Is it worth to extend the set of ignored whitespaces to "Pattern Whitespaces"? Would it add any benefit? Or add confusion? Should this depend on the re.ASCII mode? Should the byte b'\x85' be ignorable in verbose bytes patterns? And there is a similar question about the Python parser. If Python uses Unicode definition for identifier, shouldn't it accept non-ASCII "Pattern Whitespaces" as whitespaces? There will be technical problems with supporting this, but are there any benefits? https://perldoc.perl.org/perlre.html https://www.unicode.org/reports/tr31/tr31-4.html#Pattern_Syntax https://unicode.org/L2/L2005/05012r-pattern.html From p.f.moore at gmail.com Thu Nov 16 05:44:35 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 16 Nov 2017 10:44:35 +0000 Subject: [Python-ideas] Ignorable whitespaces in the re.VERBOSE mode In-Reply-To: References: Message-ID: My instinct is not to worry about it unless someone has actually hit the issue in practice and raised a bug. Paul On 16 November 2017 at 10:23, Serhiy Storchaka wrote: > Currently the re module ignores only 6 ASCII whitespaces in the re.VERBOSE > mode: > > U+0009 CHARACTER TABULATION > U+000A LINE FEED > U+000B LINE TABULATION > U+000C FORM FEED > U+000D CARRIAGE RETURN > U+0020 SPACE > > Perl ignores characters that Unicode calls "Pattern White Space" in the /x > mode. It ignores additional 5 non-ASCII characters. > > U+0085 NEXT LINE > U+200E LEFT-TO-RIGHT MARK > U+200F RIGHT-TO-LEFT MARK > U+2028 LINE SEPARATOR > U+2029 PARAGRAPH SEPARATOR > > The regex module just ignores characters for which str.isspace() returns > True. It ignores additional 20 non-ASCII whitespace characters, including > characters U+001C..001F whose classification as whitespaces is questionable, > but doesn't ignore LEFT-TO-RIGHT MARK and RIGHT-TO-LEFT MARK. > > U+001C [FILE SEPARATOR] > U+001D [GROUP SEPARATOR] > U+001E [RECORD SEPARATOR] > U+001F [UNIT SEPARATOR] > U+00A0 NO-BREAK SPACE > U+1680 OGHAM SPACE MARK > U+2000 EN QUAD > U+2001 EM QUAD > U+2002 EN SPACE > U+2003 EM SPACE > U+2004 THREE-PER-EM SPACE > U+2005 FOUR-PER-EM SPACE > U+2006 SIX-PER-EM SPACE > U+2007 FIGURE SPACE > U+2008 PUNCTUATION SPACE > U+2009 THIN SPACE > U+200A HAIR SPACE > U+202F NARROW NO-BREAK SPACE > U+205F MEDIUM MATHEMATICAL SPACE > U+3000 IDEOGRAPHIC SPACE > > Is it worth to extend the set of ignored whitespaces to "Pattern > Whitespaces"? Would it add any benefit? Or add confusion? Should this depend > on the re.ASCII mode? Should the byte b'\x85' be ignorable in verbose bytes > patterns? > > And there is a similar question about the Python parser. If Python uses > Unicode definition for identifier, shouldn't it accept non-ASCII "Pattern > Whitespaces" as whitespaces? There will be technical problems with > supporting this, but are there any benefits? > > > https://perldoc.perl.org/perlre.html > https://www.unicode.org/reports/tr31/tr31-4.html#Pattern_Syntax > https://unicode.org/L2/L2005/05012r-pattern.html > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From wes.turner at gmail.com Thu Nov 16 06:35:30 2017 From: wes.turner at gmail.com (Wes Turner) Date: Thu, 16 Nov 2017 06:35:30 -0500 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: <8d5574dc-b614-ccfe-3ce8-f1ba86508179@python.org> References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> <5A087DE7.4050306@brenbarn.net> <8d5574dc-b614-ccfe-3ce8-f1ba86508179@python.org> Message-ID: Would something like this be helpful? (`python -m pip` seems to work fine after `conda install pip` after a miniconda install) https://gist.github.com/westurner/80e81cd4bf9b79acb5989622f2621655 ```python #!/usr/bin/env python """ Something like this as e.g site.discover (for use as ``python -m site.discover``) could be helpful for explaining and diagnosing multiple pythons and pips """ import os import re from distutils.spawn import find_executable from subprocess import check_output, STDOUT import collections def discover(path=None, printsite=False, uniques=True): """ Find all ``python*`` and ``pip*`` executables and their version strings within ``os.environ['PATH']`` Kwargs: path (str): os.pathsep-delimited path str (defaults to ``os.environ['PATH']``) printsite (bool): call `` -m site`` with each found python binary uniques (bool): print uniques according to ``os.realpath`` at the end Returns: None """ if path is None: path = os.environ['PATH'] rgx = re.compile(r'(python|pip)\-?(\d?\.?\d?)$') uniques = collections.OrderedDict() for directory in path.split(os.pathsep): paths = sorted( ((match.group(0), match.group(2)) for match in (rgx.match(name) for name in os.listdir(directory)) if match), key=lambda x: x[::-1], reverse=True) print(u'# %s' % directory) for (name, ver) in paths: pathjoined = os.path.join(directory, name) binpath = find_executable(pathjoined, directory) realpath = os.path.realpath(binpath) if binpath is None: continue if os.path.exists(pathjoined): if os.path.islink(pathjoined): print(( u"%r is a symlink to %r, which doesn't exist" % (pathjoined, binpath))) # TODO: %r exists but isnt executable verstring = check_output((binpath, '-V'), stderr=STDOUT).rstrip() uniques.setdefault(realpath, collections.OrderedDict()) \ .setdefault(binpath, verstring) print(u'%-11r %-5r %r' % (name, ver, verstring)) if printsite and name.startswith('python'): sitestring = check_output((binpath, '-m', 'site')) lines = (u'> %s' % l for l in sitestring.splitlines()) for line in lines: print(line) # TODO: check_output((binpath, '-m', 'pip')) # TODO: AND/OR (?) only print uniques at the end if uniques: print('### UNIQUES') for path in uniques: print('## %s' % path) paths = uniques.get(path) for otherpath in sorted(paths): print(otherpath) print('') if __name__ == '__main__': discover() ``` On Wednesday, November 15, 2017, Steve Dower wrote: > On 15Nov2017 0617, Nick Coghlan wrote: > >> On 15 November 2017 at 22:46, Michel Desmoulin > > wrote: >> Should I do a PEP with a summary of all the stuff we discussed ? >> I think a Windows-specific PEP covering adding PATH updates back to the >> default installer behaviour, and adding pythonX and pythonX.Y commands >> would be useful (and Guido would presumably delegate resolving that to >> Steve Dower as the Windows installer maintainer). >> > > If you write such a PEP, please also research and write up the issues with > modifying PATH on Windows (they're largely scattered throughout bugs.p.o > and earlier discussions on python-dev). > > Once you realise the tradeoff involved in modifying these global settings, > you'll either come around to my point of view or be volunteering to take > *all* the support questions when they come in :) > > The one thing I'd ask is that any such PEP *not* advocate for promoting >> ther variants as the preferred way of invoking Python on Windows - rather, >> they should be positioned as a way of making online instructions written >> for Linux more likely to "just work" for folks on Windows (similar to the >> utf-8 encoding changes in https://www.python.org/dev/peps/pep-0529/) >> >> Instead, the focus should be on ensuring the "python -m pip install" and >> "pip install" both work after clicking through the installer without >> changing any settings, and devising a troubleshooting guide to help folks >> that are familiar with computers and Python, but perhaps not with Windows, >> guide folks to a properly working environment. >> > > My preferred solution for this is to rename "py.exe" to "python.exe" (or > rather, make a copy of it with the new name), and extend (or more likely, > rewrite) the launcher such that: > > * if argv[0] == "py.exe", use PEP 514 company/tag resolution to find and > launch Python based on first command line argument > * if argv[0] == "python.exe", find the matching > PythonCore/ install (where tag may be a partial match - e.g. > "python3.exe" finds the latest PythonCore/3.x) > * else, if argv[0] == ".exe, find the matching > PythonCore/ install and launch "-m " > > With the launcher behaving like this, we can make as many hard links as we > want in its install directory (it only gets installed once, so only needs > one PATH entry, and this is C:\Windows for admin installs): > * python.exe > * python2.exe > * python3.exe > * python3.6.exe > * pip.exe > * pip2.exe > * pip3.exe > > As well as allowing e.g. "py.exe -anaconda36-64 ..." to reliably locate > and run non-Python.org installs. > > It needs to be fully specced out, obviously, and we may want to move the > all-users install to its own directory to reduce clutter, but part of the > reason behind PEP 514 was to enable this sort of launcher. It could even > extend to "you don't have this version right now, want to download and > install it?" > > And finally it should be fairly obvious that this doesn't have to be a > core Python tool. It has no reliance on anything in core (that isn't > already specified in a PEP) and could be written totally independently. > I've tried (weakly) to get work time allocated to this in the past, and if > it's genuinely not going to get done unless I do it then I'll try again. > > Cheers, > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From k7hoven at gmail.com Thu Nov 16 07:06:29 2017 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Thu, 16 Nov 2017 14:06:29 +0200 Subject: [Python-ideas] Consider (one day) adding an inheritance order class precedence mechanism In-Reply-To: <979c2edf-e94c-46ed-bacd-730686ea3999@googlegroups.com> References: <979c2edf-e94c-46ed-bacd-730686ea3999@googlegroups.com> Message-ID: On Thu, Nov 16, 2017 at 5:28 AM, Neil Girdhar wrote: > On Wednesday, November 15, 2017 at 5:32:00 PM UTC-5, Koos Zevenhoven wrote: >> >> On Wed, Nov 15, 2017 at 11:49 PM, Neil Girdhar >> wrote: >> >>> Sometimes I get MRO failures when defining classes. For example, if >>> >>> R < E, C >>> B < S, E >>> S < C >>> Z < B, R >>> >>> Then Z cannot be instantiated because C precedes E in B and E precedes C >>> in R. The problem is that when the inheritance graph was >>> topologically-sorted in B, the order was S, C, E. It could just as easily >>> have been S, E, C. So, one solution is to add C explicitly to B's base >>> class list, but this is annoying because B might not care about C (it's an >>> implementation detail of S). It also means that if the hierarchy changes, >>> a lot of these added extra base classes need to be fixed. >>> >>> I propose adding a magic member to classes: >>> >>> __precedes__ that is a list of classes. So, the fix for the above >>> problem would be to define E as follows: >>> >>> class E: >>> from whatever import C >>> __precedes__ = [C] >>> >>> Then, when topologically-sorting (so-called linearizing) the ancestor >>> classes, Python can try to ensure that E precedes C when defining B. >>> >>> >> So it sounds like you are talking about the way that the siblings in the >> inheritance tree (the bases of each class) get "flattened" into the mro in >> a depth-first manner, and the relative order of siblings is not preserved. >> > > It is preserved, but there are insufficient constraints, which causes > problems with future class definitions. > > >> What would you think about not topologically sorting the inheritance tree >> as such, but sorting a graph that has additional edges according to the >> base lists of each class involved? In this case those edges would be E->C, >> S->E, B->R. Is this what your talking about, or do I misinterpret the >> problem? >> > > That's already part of the topological sorting algorithm. You have to do > that. I'm suggesting additional constraints. > > ??I'm talking about the initial graph to be sorted. Not some intermediate graph that may be used for describing steps in an algorithm. Anyway, the contraint given by the edge E->C is not y?et part of the algorithm, because if it was already there, you wouldn't need "__precedes__" to add that edge. But another thing that affects this is whether we consider multiple occurrences of a class in the inheritance tree as the same node in the inheritance graph or not. If yes, then the inheritance tree and the inheritance graph are two different things. ??Koos -- + Koos Zevenhoven + http://twitter.com/k7hoven + -------------- next part -------------- An HTML attachment was scrubbed... URL: From bunslow at gmail.com Thu Nov 16 08:56:21 2017 From: bunslow at gmail.com (bunslow) Date: Thu, 16 Nov 2017 07:56:21 -0600 Subject: [Python-ideas] Rewriting the "roundrobin" recipe in the itertools documentation Message-ID: For taking values alternately from a series of iterables, there's two primary functions: builtin.zip itertools.zip_longest zip of course stops when the shortest iterable ends. zip_longest is generally a useful substitute for when you don't want the zip behavior, but it fills extra values in the blanks rather than just ignoring a finished iterator and moving on with the rest. This latter most use case is at least somewhat common, according to this[1] StackOverflow question (and other duplicates), in addition to the existence of the `roundrobin` recipe[2] in the itertools docs. The recipe satisfies this use case, and its code is repeated in the StackOverflow answer. However, it is remarkably unpythonic, in my opinion, which is one thing when such is necessary to achieve a goal, but for this functionality, such is most definitely *not* necessary. I'll paste the code here for quick reference: def roundrobin(*iterables): "roundrobin('ABC', 'D', 'EF') --> A D E B F C" pending = len(iterables) nexts = cycle(iter(it).__next__ for it in iterables) while pending: try: for next in nexts: yield next() except StopIteration: pending -= 1 nexts = cycle(islice(nexts, pending)) Things that strike me as unpythonic: 1) requiring the total number of input iterables 2) making gratuitous use of `next`, 3) using a while loop in code dealing with iterables, 4) combining loops, exceptions, and composed itertools functions in non-obvious ways that make control flow difficult to determine Now, I get it, looking at the "roughly equivalent to" code for zip_longest in the docs, there doesn't seem to be much way around it for generally similar goals, and as I said above, unpythonic is fine when necessary (practicality beats purity), but in this case, for being a "recipe" in the itertools docs, it should *make use* of the zip_longest which already does all the unpythonic stuff for you (though honestly I'm not convinced either that the zip_longest code in the docs is the most possible pythonic-ness). Instead, the following recipe (which I also submitted to the StackOverflow question, and which is generally similar to several other later answers, all remarking that they believe it's more pythonic) is much cleaner and more suited to demonstrating the power of itertools to new developers than the mess of a "recipe" pasted above. def roundrobin(*iters): "roundrobin('ABC', 'D', 'EF') --> A D E B F C" # Perhaps "flat_zip_nofill" is a better name, or something similar sentinel = object() for tup in it.zip_longest(*iters, fillvalue=sentinel): yield from (x for x in tup if x is not sentinel) In particular, this is just an extremely thin wrapper around zip_longest, whose primary purpose is to eliminate the otherwise-mandatory "fillvalues" that zip_longest requires to produce uniform-length tuples. It's also an excellent example of how to make best pythonic use of iterables in general, and itertools in particular, and as such a much better implementation to be demonstrated in documentation. I would thus advocate that the former recipe is replaced with the latter recipe, being much more pythonic, understandable, and useful for helping new developers acquire the style of python. (Using the common linguistics analogy: a dictionary and grammar for a spoken language may be enough to communicate, but we rely on a large body of literature -- fiction, research, poetry, etc -- as children to get that special flavor and most expressive taste to the language. The stdlib is no Shakespeare, but it and its docs still form an important part of the formative literature of the Python language.) I realize at the end of the day this is a pretty trivial and ultimately meaningless nit to pick, but I've never contributed before and have a variety of similar minor pain points in the docs/stdlib, and I'm trying to gauge 1) how well this sort of minor QoL improvement is wanted, and 2) even if it is wanted, am I going about it the right way. If the answers to both of these questions are positive regarding this particular case, then I'll look into making a BPO issue and pull request on GitHub, which IIUC is the standard path for contributions. Thank you for your consideration. ~~~~ [1]: https://stackoverflow.com/questions/3678869/ pythonic-way-to-combine-two-lists-in-an-alternating-fashion/ [2]: https://docs.python.org/3/library/itertools.html#itertools-recipes -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Thu Nov 16 08:58:10 2017 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 16 Nov 2017 05:58:10 -0800 Subject: [Python-ideas] Consider (one day) adding an inheritance order class precedence mechanism In-Reply-To: <5A0D2BC7.3040209@canterbury.ac.nz> References: <20171116001841.GN19802@ando.pearwood.info> <5A0D2BC7.3040209@canterbury.ac.nz> Message-ID: <5A0D9972.8020905@stoneleaf.us> On 11/15/2017 10:10 PM, Greg Ewing wrote: > Steven D'Aprano wrote: >> These are not equivalent: >> >> B < S, E >> B < E, S > > Not in general, but in many cases they will be, e.g. if > E and S have no method names in common. I think the OP is > implying that his case is one of those. > > Maybe what's really wanted is a way to say "B inherits from > S and E, but it doesn't care what order they go in". Then > the MRO generating algorithm could in principle swap them > if it would result in a consistent MRO. > > Or maybe the MRO generator could decide for itself if the > order of two base classes can be swapped by inspecting > their attributes to see if any of them clash? If they don't have any clashing attributes, why does order matter? -- ~Ethan~ From mertz at gnosis.cx Thu Nov 16 10:08:56 2017 From: mertz at gnosis.cx (David Mertz) Date: Thu, 16 Nov 2017 07:08:56 -0800 Subject: [Python-ideas] Rewriting the "roundrobin" recipe in the itertools documentation In-Reply-To: References: Message-ID: I agree this is a much better recipe presented. Have you benchmarked the two on more realistically long iterators. E.g. a hundred iterators of millions of items where many terminate much earlier than others. I doubt the repeated 'is not' comparison makes much difference, but it would be good to see. On Nov 16, 2017 5:57 AM, "bunslow" wrote: > For taking values alternately from a series of iterables, there's two > primary functions: > > builtin.zip > itertools.zip_longest > > zip of course stops when the shortest iterable ends. zip_longest is > generally a useful substitute for when you don't want the zip behavior, but > it fills extra values in the blanks rather than just ignoring a finished > iterator and moving on with the rest. > > This latter most use case is at least somewhat common, according to > this[1] StackOverflow question (and other duplicates), in addition to the > existence of the `roundrobin` recipe[2] in the itertools docs. The recipe > satisfies this use case, and its code is repeated in the StackOverflow > answer. > > However, it is remarkably unpythonic, in my opinion, which is one thing > when such is necessary to achieve a goal, but for this functionality, such > is most definitely *not* necessary. I'll paste the code here for quick > reference: > > > def roundrobin(*iterables): > "roundrobin('ABC', 'D', 'EF') --> A D E B F C" > pending = len(iterables) > nexts = cycle(iter(it).__next__ for it in iterables) > while pending: > try: > for next in nexts: > yield next() > except StopIteration: > pending -= 1 > nexts = cycle(islice(nexts, pending)) > > > Things that strike me as unpythonic: 1) requiring the total number of > input iterables 2) making gratuitous use of `next`, 3) using a while loop > in code dealing with iterables, 4) combining loops, exceptions, and > composed itertools functions in non-obvious ways that make control flow > difficult to determine > > Now, I get it, looking at the "roughly equivalent to" code for zip_longest > in the docs, there doesn't seem to be much way around it for generally > similar goals, and as I said above, unpythonic is fine when necessary > (practicality beats purity), but in this case, for being a "recipe" in the > itertools docs, it should *make use* of the zip_longest which already does > all the unpythonic stuff for you (though honestly I'm not convinced either > that the zip_longest code in the docs is the most possible pythonic-ness). > Instead, the following recipe (which I also submitted to the StackOverflow > question, and which is generally similar to several other later answers, > all remarking that they believe it's more pythonic) is much cleaner and > more suited to demonstrating the power of itertools to new developers than > the mess of a "recipe" pasted above. > > > def roundrobin(*iters): > "roundrobin('ABC', 'D', 'EF') --> A D E B F C" > # Perhaps "flat_zip_nofill" is a better name, or something similar > sentinel = object() > for tup in it.zip_longest(*iters, fillvalue=sentinel): > yield from (x for x in tup if x is not sentinel) > > > In particular, this is just an extremely thin wrapper around zip_longest, > whose primary purpose is to eliminate the otherwise-mandatory "fillvalues" > that zip_longest requires to produce uniform-length tuples. It's also an > excellent example of how to make best pythonic use of iterables in general, > and itertools in particular, and as such a much better implementation to be > demonstrated in documentation. > > I would thus advocate that the former recipe is replaced with the latter > recipe, being much more pythonic, understandable, and useful for helping > new developers acquire the style of python. (Using the common linguistics > analogy: a dictionary and grammar for a spoken language may be enough to > communicate, but we rely on a large body of literature -- fiction, > research, poetry, etc -- as children to get that special flavor and most > expressive taste to the language. The stdlib is no Shakespeare, but it and > its docs still form an important part of the formative literature of the > Python language.) > > I realize at the end of the day this is a pretty trivial and ultimately > meaningless nit to pick, but I've never contributed before and have a > variety of similar minor pain points in the docs/stdlib, and I'm trying to > gauge 1) how well this sort of minor QoL improvement is wanted, and 2) even > if it is wanted, am I going about it the right way. If the answers to both > of these questions are positive regarding this particular case, then I'll > look into making a BPO issue and pull request on GitHub, which IIUC is the > standard path for contributions. > > Thank you for your consideration. > > ~~~~ > > [1]: https://stackoverflow.com/questions/3678869/pythonic- > way-to-combine-two-lists-in-an-alternating-fashion/ > > [2]: https://docs.python.org/3/library/itertools.html#itertools-recipes > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Thu Nov 16 12:21:51 2017 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 16 Nov 2017 17:21:51 +0000 Subject: [Python-ideas] Ignorable whitespaces in the re.VERBOSE mode In-Reply-To: References: Message-ID: <2b1a62e2-21bf-9521-f1d7-079107bf220c@mrabarnett.plus.com> On 2017-11-16 10:23, Serhiy Storchaka wrote: > Currently the re module ignores only 6 ASCII whitespaces in the > re.VERBOSE mode: > > U+0009 CHARACTER TABULATION > U+000A LINE FEED > U+000B LINE TABULATION > U+000C FORM FEED > U+000D CARRIAGE RETURN > U+0020 SPACE > > Perl ignores characters that Unicode calls "Pattern White Space" in the > /x mode. It ignores additional 5 non-ASCII characters. > > U+0085 NEXT LINE > U+200E LEFT-TO-RIGHT MARK > U+200F RIGHT-TO-LEFT MARK > U+2028 LINE SEPARATOR > U+2029 PARAGRAPH SEPARATOR > > The regex module just ignores characters for which str.isspace() returns > True. It ignores additional 20 non-ASCII whitespace characters, > including characters U+001C..001F whose classification as whitespaces is > questionable, but doesn't ignore LEFT-TO-RIGHT MARK and RIGHT-TO-LEFT MARK. > > U+001C [FILE SEPARATOR] > U+001D [GROUP SEPARATOR] > U+001E [RECORD SEPARATOR] > U+001F [UNIT SEPARATOR] > U+00A0 NO-BREAK SPACE > U+1680 OGHAM SPACE MARK > U+2000 EN QUAD > U+2001 EM QUAD > U+2002 EN SPACE > U+2003 EM SPACE > U+2004 THREE-PER-EM SPACE > U+2005 FOUR-PER-EM SPACE > U+2006 SIX-PER-EM SPACE > U+2007 FIGURE SPACE > U+2008 PUNCTUATION SPACE > U+2009 THIN SPACE > U+200A HAIR SPACE > U+202F NARROW NO-BREAK SPACE > U+205F MEDIUM MATHEMATICAL SPACE > U+3000 IDEOGRAPHIC SPACE > str.isspace appears to be Unicode "Whitespace" plus those 4 "questionable" codepoints. > Is it worth to extend the set of ignored whitespaces to "Pattern > Whitespaces"? Would it add any benefit? Or add confusion? Should this > depend on the re.ASCII mode? Should the byte b'\x85' be ignorable in > verbose bytes patterns? > > And there is a similar question about the Python parser. If Python uses > Unicode definition for identifier, shouldn't it accept non-ASCII > "Pattern Whitespaces" as whitespaces? There will be technical problems > with supporting this, but are there any benefits? > > > https://perldoc.perl.org/perlre.html > https://www.unicode.org/reports/tr31/tr31-4.html#Pattern_Syntax > https://unicode.org/L2/L2005/05012r-pattern.html > From guido at python.org Thu Nov 16 12:38:59 2017 From: guido at python.org (Guido van Rossum) Date: Thu, 16 Nov 2017 09:38:59 -0800 Subject: [Python-ideas] Ignorable whitespaces in the re.VERBOSE mode In-Reply-To: <2b1a62e2-21bf-9521-f1d7-079107bf220c@mrabarnett.plus.com> References: <2b1a62e2-21bf-9521-f1d7-079107bf220c@mrabarnett.plus.com> Message-ID: Who would benefit from changing this? Let's not change things just because we can, or because Perl 6 does it. On Thu, Nov 16, 2017 at 9:21 AM, MRAB wrote: > On 2017-11-16 10:23, Serhiy Storchaka wrote: > >> Currently the re module ignores only 6 ASCII whitespaces in the >> re.VERBOSE mode: >> >> U+0009 CHARACTER TABULATION >> U+000A LINE FEED >> U+000B LINE TABULATION >> U+000C FORM FEED >> U+000D CARRIAGE RETURN >> U+0020 SPACE >> >> Perl ignores characters that Unicode calls "Pattern White Space" in the >> /x mode. It ignores additional 5 non-ASCII characters. >> >> U+0085 NEXT LINE >> U+200E LEFT-TO-RIGHT MARK >> U+200F RIGHT-TO-LEFT MARK >> U+2028 LINE SEPARATOR >> U+2029 PARAGRAPH SEPARATOR >> >> The regex module just ignores characters for which str.isspace() returns >> True. It ignores additional 20 non-ASCII whitespace characters, >> including characters U+001C..001F whose classification as whitespaces is >> questionable, but doesn't ignore LEFT-TO-RIGHT MARK and RIGHT-TO-LEFT >> MARK. >> >> U+001C [FILE SEPARATOR] >> U+001D [GROUP SEPARATOR] >> U+001E [RECORD SEPARATOR] >> U+001F [UNIT SEPARATOR] >> U+00A0 NO-BREAK SPACE >> U+1680 OGHAM SPACE MARK >> U+2000 EN QUAD >> U+2001 EM QUAD >> U+2002 EN SPACE >> U+2003 EM SPACE >> U+2004 THREE-PER-EM SPACE >> U+2005 FOUR-PER-EM SPACE >> U+2006 SIX-PER-EM SPACE >> U+2007 FIGURE SPACE >> U+2008 PUNCTUATION SPACE >> U+2009 THIN SPACE >> U+200A HAIR SPACE >> U+202F NARROW NO-BREAK SPACE >> U+205F MEDIUM MATHEMATICAL SPACE >> U+3000 IDEOGRAPHIC SPACE >> >> str.isspace appears to be Unicode "Whitespace" plus those 4 > "questionable" codepoints. > > > Is it worth to extend the set of ignored whitespaces to "Pattern >> Whitespaces"? Would it add any benefit? Or add confusion? Should this >> depend on the re.ASCII mode? Should the byte b'\x85' be ignorable in >> verbose bytes patterns? >> >> And there is a similar question about the Python parser. If Python uses >> Unicode definition for identifier, shouldn't it accept non-ASCII >> "Pattern Whitespaces" as whitespaces? There will be technical problems >> with supporting this, but are there any benefits? >> >> >> https://perldoc.perl.org/perlre.html >> https://www.unicode.org/reports/tr31/tr31-4.html#Pattern_Syntax >> https://unicode.org/L2/L2005/05012r-pattern.html >> >> _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Thu Nov 16 14:03:42 2017 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 16 Nov 2017 11:03:42 -0800 Subject: [Python-ideas] A proliferation of (un-)Pythonically programmatic pragmas In-Reply-To: <20171115021457.GM19802@ando.pearwood.info> References: <20171115021457.GM19802@ando.pearwood.info> Message-ID: On Tue, Nov 14, 2017 at 6:14 PM, Steven D'Aprano wrote: > > and then, even if you wanted to put it on the previous line, you would't > > have the line-length issue: > > > > # flake8: disable=unused-import mypy: alias=pathlib2.Path coverage: > > ignore=when>py2.7 > > And the irony is that this shows exactly why it is still a problem. Code > gets emailed, and lines get wrapped, breaking the too-long line. > that wrapped a 85 char line (probably to 72) -- lots of folks have set their line length limit to 90+ these days, so I"m not sure we'll evr be safe... but whatever -- multiple line pragmas are fine, too -- I'm hoping putting this much crap in your code will be rare :-) -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Thu Nov 16 14:15:27 2017 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 16 Nov 2017 11:15:27 -0800 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: <8d5574dc-b614-ccfe-3ce8-f1ba86508179@python.org> References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> <5A087DE7.4050306@brenbarn.net> <8d5574dc-b614-ccfe-3ce8-f1ba86508179@python.org> Message-ID: On Wed, Nov 15, 2017 at 11:07 AM, Steve Dower wrote: > If you write such a PEP, please also research and write up the issues with > modifying PATH on Windows (they're largely scattered throughout bugs.p.o > and earlier discussions on python-dev). > Is anyone proposing doing anything new with that? (other than changing the default) My preferred solution for this is to rename "py.exe" to "python.exe" I was going to propose that in this thread, but then thought: "there has GOT to be a reason why that reall obvious solution wan't done in the first place", and didn't have time to go back and research it. But if it's feasible from a technical and backward-compatible perspective, great! -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From brent.bejot at gmail.com Thu Nov 16 14:42:41 2017 From: brent.bejot at gmail.com (brent bejot) Date: Thu, 16 Nov 2017 14:42:41 -0500 Subject: [Python-ideas] Rewriting the "roundrobin" recipe in the itertools documentation In-Reply-To: References: Message-ID: I think the idea behind the original recipe is that when one of the inner lists has been iterated through, it is removed and never looked at again. Imagine the following scenario: L is a list which contains one million empty lists and also a list containing one million numbers Then the original recipe will iterate over two million(ish) items: each inner list must get visited and each item from the long inner list must get visited. However, your use of zip_longest must visit one trillion items, which will likely not finish in a reasonable amount of time. I'm not saying that this is likely to be the case, but this is probably why the original recipe is what it is. It would be great to see a recipe that is more pythonic but that maintains the efficiencies of the first recipy, but I could not come up with one. -Brent On Thu, Nov 16, 2017 at 10:08 AM, David Mertz wrote: > I agree this is a much better recipe presented. > > Have you benchmarked the two on more realistically long iterators. E.g. a > hundred iterators of millions of items where many terminate much earlier > than others. I doubt the repeated 'is not' comparison makes much > difference, but it would be good to see. > > On Nov 16, 2017 5:57 AM, "bunslow" wrote: > >> For taking values alternately from a series of iterables, there's two >> primary functions: >> >> builtin.zip >> itertools.zip_longest >> >> zip of course stops when the shortest iterable ends. zip_longest is >> generally a useful substitute for when you don't want the zip behavior, but >> it fills extra values in the blanks rather than just ignoring a finished >> iterator and moving on with the rest. >> >> This latter most use case is at least somewhat common, according to >> this[1] StackOverflow question (and other duplicates), in addition to the >> existence of the `roundrobin` recipe[2] in the itertools docs. The recipe >> satisfies this use case, and its code is repeated in the StackOverflow >> answer. >> >> However, it is remarkably unpythonic, in my opinion, which is one thing >> when such is necessary to achieve a goal, but for this functionality, such >> is most definitely *not* necessary. I'll paste the code here for quick >> reference: >> >> >> def roundrobin(*iterables): >> "roundrobin('ABC', 'D', 'EF') --> A D E B F C" >> pending = len(iterables) >> nexts = cycle(iter(it).__next__ for it in iterables) >> while pending: >> try: >> for next in nexts: >> yield next() >> except StopIteration: >> pending -= 1 >> nexts = cycle(islice(nexts, pending)) >> >> >> Things that strike me as unpythonic: 1) requiring the total number of >> input iterables 2) making gratuitous use of `next`, 3) using a while loop >> in code dealing with iterables, 4) combining loops, exceptions, and >> composed itertools functions in non-obvious ways that make control flow >> difficult to determine >> >> Now, I get it, looking at the "roughly equivalent to" code for >> zip_longest in the docs, there doesn't seem to be much way around it for >> generally similar goals, and as I said above, unpythonic is fine when >> necessary (practicality beats purity), but in this case, for being a >> "recipe" in the itertools docs, it should *make use* of the zip_longest >> which already does all the unpythonic stuff for you (though honestly I'm >> not convinced either that the zip_longest code in the docs is the most >> possible pythonic-ness). Instead, the following recipe (which I also >> submitted to the StackOverflow question, and which is generally similar to >> several other later answers, all remarking that they believe it's more >> pythonic) is much cleaner and more suited to demonstrating the power of >> itertools to new developers than the mess of a "recipe" pasted above. >> >> >> def roundrobin(*iters): >> "roundrobin('ABC', 'D', 'EF') --> A D E B F C" >> # Perhaps "flat_zip_nofill" is a better name, or something similar >> sentinel = object() >> for tup in it.zip_longest(*iters, fillvalue=sentinel): >> yield from (x for x in tup if x is not sentinel) >> >> >> In particular, this is just an extremely thin wrapper around zip_longest, >> whose primary purpose is to eliminate the otherwise-mandatory "fillvalues" >> that zip_longest requires to produce uniform-length tuples. It's also an >> excellent example of how to make best pythonic use of iterables in general, >> and itertools in particular, and as such a much better implementation to be >> demonstrated in documentation. >> >> I would thus advocate that the former recipe is replaced with the latter >> recipe, being much more pythonic, understandable, and useful for helping >> new developers acquire the style of python. (Using the common linguistics >> analogy: a dictionary and grammar for a spoken language may be enough to >> communicate, but we rely on a large body of literature -- fiction, >> research, poetry, etc -- as children to get that special flavor and most >> expressive taste to the language. The stdlib is no Shakespeare, but it and >> its docs still form an important part of the formative literature of the >> Python language.) >> >> I realize at the end of the day this is a pretty trivial and ultimately >> meaningless nit to pick, but I've never contributed before and have a >> variety of similar minor pain points in the docs/stdlib, and I'm trying to >> gauge 1) how well this sort of minor QoL improvement is wanted, and 2) even >> if it is wanted, am I going about it the right way. If the answers to both >> of these questions are positive regarding this particular case, then I'll >> look into making a BPO issue and pull request on GitHub, which IIUC is the >> standard path for contributions. >> >> Thank you for your consideration. >> >> ~~~~ >> >> [1]: https://stackoverflow.com/questions/3678869/pythonic-wa >> y-to-combine-two-lists-in-an-alternating-fashion/ >> >> [2]: https://docs.python.org/3/library/itertools.html#itertools-recipes >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Thu Nov 16 14:56:29 2017 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 16 Nov 2017 14:56:29 -0500 Subject: [Python-ideas] Rewriting the "roundrobin" recipe in the itertools documentation In-Reply-To: References: Message-ID: On 11/16/2017 8:56 AM, bunslow wrote: > For taking values alternately from a series of iterables, there's two > primary functions: > > builtin.zip > itertools.zip_longest These bunch together the nth items of each iterable, while itertools.cycle does not. ... > def roundrobin(*iterables): > ? ? "roundrobin('ABC', 'D', 'EF') --> A D E B F C" > ? ? pending = len(iterables) > ? ? nexts = cycle(iter(it).__next__ for it in iterables) > ? ? while pending: > ? ? ? ? try: > ? ? ? ? ? ? for next in nexts: > ? ? ? ? ? ? ? ? yield next() > ? ? ? ? except StopIteration: > ? ? ? ? ? ? pending -= 1 > ? ? ? ? ? ? nexts = cycle(islice(nexts, pending)) > Things that strike me as unpythonic: > 1) requiring the total number of input iterable, > 2) making gratuitous use of `next`, I disagree that 1 and 2 are problems. > 3) using a while loop in code dealing with iterables, I agree that this is not necessary, and give a replacement below. > 4) combining loops, exceptions, and composed itertools functions in > non-obvious ways that make control flow difficult to determine I agree that the correctness of the last statement is slightly opaque. But this nicely demonstrates a non-trivial use of cycle. > Now, I get it, looking at the "roughly equivalent to" code for > zip_longest in the docs, there doesn't seem to be much way around it for > generally similar goals, and as I said above, unpythonic is fine when > necessary (practicality beats purity), but in this case, for being a > "recipe" in the itertools docs, it should *make use* of the zip_longest > which already does all the unpythonic stuff for you (though honestly I'm > not convinced either that the zip_longest code in the docs is the most > possible pythonic-ness). Instead, the following recipe (which I also > submitted to the StackOverflow question, and which is generally similar > to several other later answers, all remarking that they believe it's > more pythonic) is much cleaner and more suited to demonstrating the > power of itertools to new developers than the mess of a "recipe" pasted > above. > > def roundrobin(*iters): > ? ? "roundrobin('ABC', 'D', 'EF') --> A D E B F C" > ? ? # Perhaps "flat_zip_nofill" is a better name, or something similar > ? ? sentinel = object() > ? ? for tup in it.zip_longest(*iters, fillvalue=sentinel): > ? ? ? ? yield from (x for x in tup if x is not sentinel) This adds and then deletes grouping and fill values that are not wanted. To me, this is an 'algorithm smell'. One of the principles of algorithm design is to avoid unnecessary calculations. For an edge case such as roundrobin(1000000*'a', ''), the above mostly does unnecessary work. The following combines 3 statements into one for statement. def roundrobin(*iterables): "roundrobin('ABC', 'D', 'EF') --> A D E B F C" nexts = cycle(iter(it).__next__ for it in iterables) for reduced_len in reversed(range(1, len(iterables))): try: for next in nexts: yield next() except StopIteration: nexts = cycle(islice(nexts, reduced_len)) > In particular, this is just an extremely thin wrapper around > zip_longest, whose primary purpose is to eliminate the > otherwise-mandatory "fillvalues" that zip_longest requires to produce > uniform-length tuples. But we do not want tuples or fill values. > It's also an excellent example of how to make > best pythonic use of iterables in general, and itertools in particular, > and as such a much better implementation to be demonstrated in > documentation. I disagree. [I have mostly stopped using 'pythonic' because there is too much disagreement on particulars, and its use seems to inhibit as much as facilitate insight.] [snip more] > I realize at the end of the day this is a pretty trivial and ultimately > meaningless nit to pick, but I've never contributed before and have a > variety of similar minor pain points in the docs/stdlib, and I'm trying > to gauge 1) how well this sort of minor QoL improvement is wanted, We constantly improve the docs. > 2) even if it is wanted, am I going about it the right way. Typos and trivial grammar issues can be filed as a PR with no issue required. Clarifications usually require an issue and perhaps discussion. Since this is more about philosophy of algorithm design, python-ideas was a good place to start. > If the > answers to both of these questions are positive regarding this > particular case, then I'll look into making a BPO issue and pull request > on GitHub, which IIUC is the standard path for contributions. Since I have a competing 'improvement', I would hold off on a PR until Raymond Hettinger, the itertools author, comments. -- Terry Jan Reedy From ethan at stoneleaf.us Thu Nov 16 14:57:51 2017 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 16 Nov 2017 11:57:51 -0800 Subject: [Python-ideas] Rewriting the "roundrobin" recipe in the itertools documentation In-Reply-To: References: Message-ID: <5A0DEDBF.7050507@stoneleaf.us> On 11/16/2017 05:56 AM, bunslow wrote: > I realize at the end of the day this is a pretty trivial and ultimately meaningless nit to pick, Not at all -- documentation is extremely important. > but I've never > contributed before and have a variety of similar minor pain points in the docs/stdlib, and I'm trying to gauge 1) how > well this sort of minor QoL improvement is wanted, and 2) even if it is wanted, am I going about it the right way. If > the answers to both of these questions are positive regarding this particular case, then I'll look into making a BPO > issue and pull request on GitHub, which IIUC is the standard path for contributions. I found your example much easier to understand. Just include a note the the "roughly similar" python code is not performant in extreme situations; after all, it's there to aid understanding, not to be used. -- ~Ethan~ From mistersheik at gmail.com Thu Nov 16 15:16:29 2017 From: mistersheik at gmail.com (Neil Girdhar) Date: Thu, 16 Nov 2017 12:16:29 -0800 (PST) Subject: [Python-ideas] Rewriting the "roundrobin" recipe in the itertools documentation In-Reply-To: References: Message-ID: I like yours better. Plenty of recipes sit on top of other recipes, e.g., pairwise sits on top of tee. On Thursday, November 16, 2017 at 8:57:29 AM UTC-5, bunslow wrote: > > For taking values alternately from a series of iterables, there's two > primary functions: > > builtin.zip > itertools.zip_longest > > zip of course stops when the shortest iterable ends. zip_longest is > generally a useful substitute for when you don't want the zip behavior, but > it fills extra values in the blanks rather than just ignoring a finished > iterator and moving on with the rest. > > This latter most use case is at least somewhat common, according to > this[1] StackOverflow question (and other duplicates), in addition to the > existence of the `roundrobin` recipe[2] in the itertools docs. The recipe > satisfies this use case, and its code is repeated in the StackOverflow > answer. > > However, it is remarkably unpythonic, in my opinion, which is one thing > when such is necessary to achieve a goal, but for this functionality, such > is most definitely *not* necessary. I'll paste the code here for quick > reference: > > > def roundrobin(*iterables): > "roundrobin('ABC', 'D', 'EF') --> A D E B F C" > pending = len(iterables) > nexts = cycle(iter(it).__next__ for it in iterables) > while pending: > try: > for next in nexts: > yield next() > except StopIteration: > pending -= 1 > nexts = cycle(islice(nexts, pending)) > > > Things that strike me as unpythonic: 1) requiring the total number of > input iterables 2) making gratuitous use of `next`, 3) using a while loop > in code dealing with iterables, 4) combining loops, exceptions, and > composed itertools functions in non-obvious ways that make control flow > difficult to determine > > Now, I get it, looking at the "roughly equivalent to" code for zip_longest > in the docs, there doesn't seem to be much way around it for generally > similar goals, and as I said above, unpythonic is fine when necessary > (practicality beats purity), but in this case, for being a "recipe" in the > itertools docs, it should *make use* of the zip_longest which already does > all the unpythonic stuff for you (though honestly I'm not convinced either > that the zip_longest code in the docs is the most possible pythonic-ness). > Instead, the following recipe (which I also submitted to the StackOverflow > question, and which is generally similar to several other later answers, > all remarking that they believe it's more pythonic) is much cleaner and > more suited to demonstrating the power of itertools to new developers than > the mess of a "recipe" pasted above. > > > def roundrobin(*iters): > "roundrobin('ABC', 'D', 'EF') --> A D E B F C" > # Perhaps "flat_zip_nofill" is a better name, or something similar > sentinel = object() > for tup in it.zip_longest(*iters, fillvalue=sentinel): > yield from (x for x in tup if x is not sentinel) > > > In particular, this is just an extremely thin wrapper around zip_longest, > whose primary purpose is to eliminate the otherwise-mandatory "fillvalues" > that zip_longest requires to produce uniform-length tuples. It's also an > excellent example of how to make best pythonic use of iterables in general, > and itertools in particular, and as such a much better implementation to be > demonstrated in documentation. > > I would thus advocate that the former recipe is replaced with the latter > recipe, being much more pythonic, understandable, and useful for helping > new developers acquire the style of python. (Using the common linguistics > analogy: a dictionary and grammar for a spoken language may be enough to > communicate, but we rely on a large body of literature -- fiction, > research, poetry, etc -- as children to get that special flavor and most > expressive taste to the language. The stdlib is no Shakespeare, but it and > its docs still form an important part of the formative literature of the > Python language.) > > I realize at the end of the day this is a pretty trivial and ultimately > meaningless nit to pick, but I've never contributed before and have a > variety of similar minor pain points in the docs/stdlib, and I'm trying to > gauge 1) how well this sort of minor QoL improvement is wanted, and 2) even > if it is wanted, am I going about it the right way. If the answers to both > of these questions are positive regarding this particular case, then I'll > look into making a BPO issue and pull request on GitHub, which IIUC is the > standard path for contributions. > > Thank you for your consideration. > > ~~~~ > > [1]: > https://stackoverflow.com/questions/3678869/pythonic-way-to-combine-two-lists-in-an-alternating-fashion/ > > [2]: https://docs.python.org/3/library/itertools.html#itertools-recipes > -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Thu Nov 16 16:44:13 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 16 Nov 2017 23:44:13 +0200 Subject: [Python-ideas] Ignorable whitespaces in the re.VERBOSE mode In-Reply-To: References: <2b1a62e2-21bf-9521-f1d7-079107bf220c@mrabarnett.plus.com> Message-ID: 16.11.17 19:38, Guido van Rossum ????: > Who would benefit from changing this? Let's not change things just > because we can, or because Perl 6 does it. I don't know. I know the disadvantages of making this change, and I ask what is the benefit. If there is a benefit, and it is important for Python, I could implement this feature in re and regex. From python at mrabarnett.plus.com Thu Nov 16 17:09:44 2017 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 16 Nov 2017 22:09:44 +0000 Subject: [Python-ideas] Ignorable whitespaces in the re.VERBOSE mode In-Reply-To: References: <2b1a62e2-21bf-9521-f1d7-079107bf220c@mrabarnett.plus.com> Message-ID: <0cd94b8e-941d-3201-f65e-1706f6b52a86@mrabarnett.plus.com> On 2017-11-16 21:44, Serhiy Storchaka wrote: > 16.11.17 19:38, Guido van Rossum ????: >> Who would benefit from changing this? Let's not change things just >> because we can, or because Perl 6 does it. > > I don't know. I know the disadvantages of making this change, and I ask > what is the benefit. If there is a benefit, and it is important for > Python, I could implement this feature in re and regex. > You could see what some more languages, e.g. C#, do. If there isn't a consensus of some kind, it's best to leave it. From steve at pearwood.info Thu Nov 16 17:50:28 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 17 Nov 2017 09:50:28 +1100 Subject: [Python-ideas] Rewriting the "roundrobin" recipe in the itertools documentation In-Reply-To: References: Message-ID: <20171116225028.GO19802@ando.pearwood.info> On Thu, Nov 16, 2017 at 07:56:21AM -0600, bunslow wrote about the roundrobin recipe in the itertools docs: > However, it is remarkably unpythonic, in my opinion [...] > Things that strike me as unpythonic: 1) requiring the total number of input > iterables 2) making gratuitous use of `next`, 3) using a while loop in code > dealing with iterables, 4) combining loops, exceptions, and composed > itertools functions in non-obvious ways that make control flow difficult to > determine I don't find *any* of those things unPythonic in the least, nor the recipe difficult to follow. > Now, I get it, looking at the "roughly equivalent to" code for zip_longest > in the docs, there doesn't seem to be much way around it for generally > similar goals, and as I said above, unpythonic is fine when necessary > (practicality beats purity), but in this case, for being a "recipe" in the > itertools docs, it should *make use* of the zip_longest The roundrobin recipe was introduced to the docs some time during Python 2.5. zip_longest (or izip_longest as it was originally called) wasn't introduced until 2.6. [...] > Instead, the following recipe (which I also submitted to the StackOverflow > question, and which is generally similar to several other later answers, > all remarking that they believe it's more pythonic) is much cleaner and > more suited to demonstrating the power of itertools to new developers than > the mess of a "recipe" pasted above. A "mess" of a quote-unquote "recipe". That's an insulting way of describing it. I'd like to see how you would have solved this problem back in Python 2.5. Your post is a very long-winded way of saying: "I have an improved, more modern implementation for the itertools roundrobin recipe, should I raise a documentation issue on the tracker to replace it?" to which I would say, "Sure, why not?". > I realize at the end of the day this is a pretty trivial and ultimately > meaningless nit to pick, but I've never contributed before and have a > variety of similar minor pain points in the docs/stdlib, and I'm trying to > gauge 1) how well this sort of minor QoL improvement is wanted, and 2) even > if it is wanted, am I going about it the right way. 1) is fine. 2) is not. You don't need to put down existing code to make your point, nor do you have to sing your code's praises. Let the code speak for itself. And while its one thing to write detailed and pedantic posts (I have a tendency to do likewise) know when its necessary and when it isn't. Sometimes we need to discuss a lot of background material to explain something, but this wasn't one of those times. Thank you, -- Steve From tjreedy at udel.edu Thu Nov 16 17:57:59 2017 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 16 Nov 2017 17:57:59 -0500 Subject: [Python-ideas] Rewriting the "roundrobin" recipe in the itertools documentation In-Reply-To: References: Message-ID: On 11/16/2017 2:56 PM, Terry Reedy wrote: Correct off-by-one error. I should have tested with an edge case such as print(list(roundrobin('ABC', ''))) > The following combines 3 statements into one for statement. > > def roundrobin(*iterables): > ??? "roundrobin('ABC', 'D', 'EF') --> A D E B F C" > ??? nexts = cycle(iter(it).__next__ for it in iterables) > ??? for reduced_len in reversed(range(1, len(iterables))): Make that 0 rather than 1 for start value. > ??????? try: > ??????????? for next in nexts: > ??????????????? yield next() > ??????? except StopIteration: > ??????????? nexts = cycle(islice(nexts, reduced_len)) A slightly clearer, slightly less efficient alternative would be def roundrobin(*iterables): "roundrobin('ABC', 'D', 'EF') --> A D E B F C" nexts = cycle(iter(it).__next__ for it in iterables) for current_len in reversed(range(1, len(iterables)+1)): try: for next in nexts: yield next() except StopIteration: nexts = cycle(islice(nexts, current_len - 1)) -- Terry Jan Reedy From steve at pearwood.info Thu Nov 16 18:06:30 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 17 Nov 2017 10:06:30 +1100 Subject: [Python-ideas] Rewriting the "roundrobin" recipe in the itertools documentation In-Reply-To: References: Message-ID: <20171116230630.GP19802@ando.pearwood.info> On Thu, Nov 16, 2017 at 02:56:29PM -0500, Terry Reedy wrote: > >3) using a while loop in code dealing with iterables, > > I agree that this is not necessary, and give a replacement below. The OP isn't just saying that its unnecessary in this case, but that its unPythonic to ever use a while loop in code dealing with iterables. I disagree with that stronger statement. > >def roundrobin(*iters): > > ? ? "roundrobin('ABC', 'D', 'EF') --> A D E B F C" > > ? ? # Perhaps "flat_zip_nofill" is a better name, or something similar > > ? ? sentinel = object() > > ? ? for tup in it.zip_longest(*iters, fillvalue=sentinel): > > ? ? ? ? yield from (x for x in tup if x is not sentinel) > > This adds and then deletes grouping and fill values that are not wanted. > To me, this is an 'algorithm smell'. One of the principles of > algorithm design is to avoid unnecessary calculations. For an edge case > such as roundrobin(1000000*'a', ''), the above mostly does unnecessary work. Its a recipe, not a tuned and optimized piece of production code. And if you're going to criticise code on the basis of efficiency, then I would hope you've actually profiled the code first. Because it isn't clear to me at all that what you call "unnecessary work" is more expensive than re-writing the recipe using a more complex algorithm with calls to cycle and islice. But I'm not here to nit-pick your recipe over the earlier ones. [...] > Since I have a competing 'improvement', I would hold off on a PR until > Raymond Hettinger, the itertools author, comments. Raise a doc issue on the tracker, and take the discussion there. I think that this is too minor an issue to need long argument on the list. Besides, it's not really on-topic as such -- it isn't about a change to the language. Its about an implementation change to a recipe in the docs. -- Steve From tjreedy at udel.edu Thu Nov 16 18:15:53 2017 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 16 Nov 2017 18:15:53 -0500 Subject: [Python-ideas] Rewriting the "roundrobin" recipe in the itertools documentation In-Reply-To: References: Message-ID: On 11/16/2017 2:42 PM, brent bejot wrote: > I think the idea behind the original recipe is that when one of the > inner lists has been iterated through, it is removed and never looked at > again.? Imagine the following scenario: > > L is a list which contains one million empty lists and also a list > containing one million numbers > Then the original recipe will iterate over two million(ish) items: each > inner list must get visited and each item from the long inner list must > get visited.? However, your use of zip_longest must visit one trillion > items, which will likely not finish in a reasonable amount of time. > > I'm not saying that this is likely to be the case, but this is probably > why the original recipe is what it is.? It would be great to see a > recipe that is more pythonic but that maintains the efficiencies of the > first recipy, but I could not come up with one. The two assignments and while loop for 'pending' can be turned into a for loop, as in my response (with correction). -- Terry Jan Reedy From greg.ewing at canterbury.ac.nz Thu Nov 16 18:44:33 2017 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 17 Nov 2017 12:44:33 +1300 Subject: [Python-ideas] Consider (one day) adding an inheritance order class precedence mechanism In-Reply-To: <5A0D9972.8020905@stoneleaf.us> References: <20171116001841.GN19802@ando.pearwood.info> <5A0D2BC7.3040209@canterbury.ac.nz> <5A0D9972.8020905@stoneleaf.us> Message-ID: <5A0E22E1.6030301@canterbury.ac.nz> Ethan Furman wrote: > If they don't have any clashing attributes, why does order matter? It doesn't -- that's the point. Currently it's assumed that the order base classes appear in a class statement is the order that they must appear in the MRO. But that's not always true. I'm suggesting that the MRO algorithm should not assume that and should make its own assessment of the required ordering. -- Greg From gadgetsteve at live.co.uk Thu Nov 16 15:49:10 2017 From: gadgetsteve at live.co.uk (Steve Barnes) Date: Thu, 16 Nov 2017 20:49:10 +0000 Subject: [Python-ideas] A proliferation of (un-)Pythonically programmatic pragmas In-Reply-To: References: <20171115021457.GM19802@ando.pearwood.info> Message-ID: On 16/11/2017 19:03, Chris Barker wrote: > On Tue, Nov 14, 2017 at 6:14 PM, Steven D'Aprano > wrote: > > > and then, even if you wanted to put it on the previous line, you would't > > have the line-length issue: > > > > # flake8: disable=unused-import mypy: alias=pathlib2.Path coverage: > > ignore=when>py2.7 > > And the irony is that this shows exactly why it is still a problem. Code > gets emailed, and lines get wrapped, breaking the too-long line. > > > that wrapped a 85 char line (probably to 72) -- lots of folks have set > their line length limit to 90+ these days, so I"m not sure we'll evr be > safe... > > but whatever -- multiple line pragmas are fine, too -- I'm hoping > putting this much crap in your code will be rare :-) > > -CHB > > -- > > Christopher Barker, Ph.D. > Oceanographer > Obviously the best way to avoid multiple warning suppressions to to fix what is being warned about, unfortunately it is not always possible. -- Steve (Gadget) Barnes Any opinions in this message are my personal opinions and do not reflect those of my employer. --- This email has been checked for viruses by AVG. http://www.avg.com From chris.barker at noaa.gov Thu Nov 16 20:26:36 2017 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Thu, 16 Nov 2017 17:26:36 -0800 Subject: [Python-ideas] Consider (one day) adding an inheritance order class precedence mechanism In-Reply-To: <5A0E22E1.6030301@canterbury.ac.nz> References: <20171116001841.GN19802@ando.pearwood.info> <5A0D2BC7.3040209@canterbury.ac.nz> <5A0D9972.8020905@stoneleaf.us> <5A0E22E1.6030301@canterbury.ac.nz> Message-ID: <-5259629405952752645@unknownmsgid> It doesn't -- that's the point. Currently it's assumed that the order base classes appear in a class statement is the order that they must appear in the MRO. It?s not assumed ? it?s how order is specified by the coder... But that's not always true. Isn?t it true by definition? I'm suggesting that the MRO algorithm should not assume that and should make its own assessment of the required ordering. Isn?t that impossible? It could determine that order doesn?t matter (no name clashes), but when that?s the case, there?s nothing to do. What am I missing? -CHB -- Greg _______________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Nov 16 20:57:29 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 17 Nov 2017 11:57:29 +1000 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> <5A087DE7.4050306@brenbarn.net> <8d5574dc-b614-ccfe-3ce8-f1ba86508179@python.org> Message-ID: On 17 November 2017 at 05:15, Chris Barker wrote: > On Wed, Nov 15, 2017 at 11:07 AM, Steve Dower > wrote: > >> If you write such a PEP, please also research and write up the issues >> with modifying PATH on Windows (they're largely scattered throughout >> bugs.p.o and earlier discussions on python-dev). >> > > Is anyone proposing doing anything new with that? (other than changing the > default) > > My preferred solution for this is to rename "py.exe" to "python.exe" > > > I was going to propose that in this thread, but then thought: "there has > GOT to be a reason why that reall obvious solution wan't done in the first > place", and didn't have time to go back and research it. > As far as I recall, the arguments against it are: - wanting the regular executable and the launcher to be able to coexist in the same build target directory - not wanting the regular python executable to prevent access to the launcher at a shell prompt - not wanting the launcher at a shell prompt to prevent access to the regular executable at a shell prompt However, https://www.python.org/dev/peps/pep-0397/ doesn't spell those out, it just states what the launcher's name will be, and emphasises that the main purpose is to provide a sensible target for file associations after the "always use the most recently installed version" assumption broke down: https://www.python.org/dev/peps/pep-0397/#rationale Addressing them now: * as long as the extra hard links are only generated at install time, there also won't be any problems with build directory name conflicts. * the launcher will always be available a `py`, regardless of the current PATH * PATH in a venv will still put the regular python executable ahead of the launcher Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Thu Nov 16 23:37:23 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 17 Nov 2017 15:37:23 +1100 Subject: [Python-ideas] A proliferation of (un-)Pythonically programmatic pragmas In-Reply-To: References: Message-ID: <20171117043723.GR19802@ando.pearwood.info> On Mon, Nov 13, 2017 at 07:17:26PM -0500, Terry Reedy wrote: > Docstring directives might be considered for points to copy or not copy. > An example is # doctest: +ELLIPSIS, +NORMALIZE_WHITESPACE [...] > For binary options, '+' and '-' prefixes are much more compact than > '=True' and '=False' suffixes. This will be more important with > multiple directives in one comment. I'd just like to strongly +1 Terry's comments here. -- Steve From steve at pearwood.info Fri Nov 17 00:52:52 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 17 Nov 2017 16:52:52 +1100 Subject: [Python-ideas] Consider (one day) adding an inheritance order class precedence mechanism In-Reply-To: <5A0D2BC7.3040209@canterbury.ac.nz> References: <20171116001841.GN19802@ando.pearwood.info> <5A0D2BC7.3040209@canterbury.ac.nz> Message-ID: <20171117055252.GS19802@ando.pearwood.info> On Thu, Nov 16, 2017 at 07:10:15PM +1300, Greg Ewing wrote: > Steven D'Aprano wrote: > >These are not equivalent: > > > >B < S, E > >B < E, S > > Not in general, but in many cases they will be, e.g. if > E and S have no method names in common. I think the OP is > implying that his case is one of those. Explicit is better than implicit :-) If Neil meant his suggestion to only apply in the case where S and E have no methods in common, he should have said so. Given the possibility of __getattr__ or more exotic things like metaclass trickery, we might not even be able to tell what methods are possible. We'd need either some philosophy like "if you use this feature, you're responsible for ensuring it works" (an excellent way to guarantee hard-to-diagnose bugs in people's code *wink* ) or a more complex set of requirements: - none of the classes involved have a non-standard metaclass; - none of them override __getattribute__ or __getattr__ - none of them have any methods in common then, and only then, can Python attempt to reorder the MRO to suit. Did I miss any necessary conditions? But that seems like its adding a lot of complexity for little benefit. Its not even clear whether this is practical -- why would the author of class E specify __precedes__ to suit a class that they don't even know exists? What if two classes both want E to specify the order in opposite directions? If I am the author of both E and B, then why don't I just reorder E's superclasses directly, instead of using __precedes__? There's a lot of benefit to having a relatively simple, understandable algorithm for determining the MRO, as opposed to some sort of adaptive rule that will try to reorder classes according to potentially clashing constraints. If that means that some classes cannot go together in multiple inheritence because their MRO would be inconsistent, I think that's a price worth paying for not having to debug inheritence bugs caused by weirdly reordered MROs. -- Steve From ncoghlan at gmail.com Fri Nov 17 03:14:39 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 17 Nov 2017 18:14:39 +1000 Subject: [Python-ideas] Consider (one day) adding an inheritance order class precedence mechanism In-Reply-To: <20171117055252.GS19802@ando.pearwood.info> References: <20171116001841.GN19802@ando.pearwood.info> <5A0D2BC7.3040209@canterbury.ac.nz> <20171117055252.GS19802@ando.pearwood.info> Message-ID: On 17 November 2017 at 15:52, Steven D'Aprano wrote: > There's a lot of benefit to having a relatively simple, understandable > algorithm for determining the MRO, as opposed to some sort of adaptive > rule that will try to reorder classes according to potentially clashing > constraints. If that means that some classes cannot go together in > multiple inheritence because their MRO would be inconsistent, I think > that's a price worth paying for not having to debug inheritence bugs > caused by weirdly reordered MROs. I'll note that an interesting side effect of https://www.python.org/dev/peps/pep-0560/#mro-entries will be to allow folks to write: class MyCustomMRO: def __init__(self, *bases): self._resolved_bases = my_mro_resolver(bases) def __mro_entries(self, orig_bases): if len(orig_bases) > 1 or orig_bases[0] is not self: raise TypeError("CustomMRO instance must be sole base class") return self._resolved_bases class Z(MyCustomMRO(B, R)): ... The custom MRO algorithm may then allow for use case specific hints to handle situations that the default C3 resolver will reject as inconsistent or ambiguous. (I'll also note that such a custom resolver would be able to manufacture and inject synthetic metaclasses if that's what someone decided they really wanted to do, by also synthesising a custom base class to stick at the head of the list of bases). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From storchaka at gmail.com Fri Nov 17 04:06:39 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Fri, 17 Nov 2017 11:06:39 +0200 Subject: [Python-ideas] Ignorable whitespaces in the re.VERBOSE mode In-Reply-To: <0cd94b8e-941d-3201-f65e-1706f6b52a86@mrabarnett.plus.com> References: <2b1a62e2-21bf-9521-f1d7-079107bf220c@mrabarnett.plus.com> <0cd94b8e-941d-3201-f65e-1706f6b52a86@mrabarnett.plus.com> Message-ID: 17.11.17 00:09, MRAB ????: > On 2017-11-16 21:44, Serhiy Storchaka wrote: >> 16.11.17 19:38, Guido van Rossum ????: >>> Who would benefit from changing this? Let's not change things just >>> because we can, or because Perl 6 does it. >> >> I don't know. I know the disadvantages of making this change, and I ask >> what is the benefit. If there is a benefit, and it is important for >> Python, I could implement this feature in re and regex. >> > You could see what some more languages, e.g. C#, do. If there isn't a > consensus of some kind, it's best to leave it. I haven't found this in the documentation, but according to the sources it uses only 5 ASCII whitespaces (exluding \v). Java uses 6 ASCII whitespaces. From k7hoven at gmail.com Fri Nov 17 04:12:50 2017 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Fri, 17 Nov 2017 11:12:50 +0200 Subject: [Python-ideas] Consider (one day) adding an inheritance order class precedence mechanism In-Reply-To: References: <20171116001841.GN19802@ando.pearwood.info> <5A0D2BC7.3040209@canterbury.ac.nz> <20171117055252.GS19802@ando.pearwood.info> Message-ID: On Fri, Nov 17, 2017 at 10:14 AM, Nick Coghlan wrote: > On 17 November 2017 at 15:52, Steven D'Aprano wrote: > > There's a lot of benefit to having a relatively simple, understandable > > algorithm for determining the MRO, as opposed to some sort of adaptive > > rule that will try to reorder classes according to potentially clashing > > constraints. If that means that some classes cannot go together in > > multiple inheritence because their MRO would be inconsistent, I think > > that's a price worth paying for not having to debug inheritence bugs > > caused by weirdly reordered MROs. > > I'll note that an interesting side effect of > https://www.python.org/dev/peps/pep-0560/#mro-entries will be to allow > folks to write: > > class MyCustomMRO: > def __init__(self, *bases): > self._resolved_bases = my_mro_resolver(bases) > def __mro_entries(self, orig_bases): > if len(orig_bases) > 1 or orig_bases[0] is not self: > raise TypeError("CustomMRO instance must be sole base > class") > return self._resolved_bases > > > class Z(MyCustomMRO(B, R)): > ... > > The custom MRO algorithm may then allow for use case specific hints to > handle situations that the default C3 resolver will reject as > inconsistent or ambiguous. (I'll also note that such a custom resolver > would be able to manufacture and inject synthetic metaclasses if > that's what someone decided they really wanted to do, by also > synthesising a custom base class to stick at the head of the list of > bases). > > ?I can imagine someone wanting that, but it still doesn't allow customizing the *whole* MRO, AFAICT. Regarding the inheritance trees of B and R that is. Or at least trying to somehow achieve that without introducing a metaclass would probably become a terrible hack (if not just impossible). ??Koos ? -- + Koos Zevenhoven + http://twitter.com/k7hoven + -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Fri Nov 17 04:13:15 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 17 Nov 2017 09:13:15 +0000 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> <5A087DE7.4050306@brenbarn.net> <8d5574dc-b614-ccfe-3ce8-f1ba86508179@python.org> Message-ID: On 17 November 2017 at 01:57, Nick Coghlan wrote: > On 17 November 2017 at 05:15, Chris Barker wrote: >> >> On Wed, Nov 15, 2017 at 11:07 AM, Steve Dower >> wrote: >>> >>> If you write such a PEP, please also research and write up the issues >>> with modifying PATH on Windows (they're largely scattered throughout >>> bugs.p.o and earlier discussions on python-dev). >> >> >> Is anyone proposing doing anything new with that? (other than changing the >> default) >> >>> My preferred solution for this is to rename "py.exe" to "python.exe" >> >> >> I was going to propose that in this thread, but then thought: "there has >> GOT to be a reason why that reall obvious solution wan't done in the first >> place", and didn't have time to go back and research it. > > > As far as I recall, the arguments against it are: > > - wanting the regular executable and the launcher to be able to coexist in > the same build target directory > - not wanting the regular python executable to prevent access to the > launcher at a shell prompt > - not wanting the launcher at a shell prompt to prevent access to the > regular executable at a shell prompt > > However, https://www.python.org/dev/peps/pep-0397/ doesn't spell those out, > it just states what the launcher's name will be, and emphasises that the > main purpose is to provide a sensible target for file associations after the > "always use the most recently installed version" assumption broke down: > https://www.python.org/dev/peps/pep-0397/#rationale > > Addressing them now: > > * as long as the extra hard links are only generated at install time, there > also won't be any problems with build directory name conflicts. > * the launcher will always be available a `py`, regardless of the current > PATH > * PATH in a venv will still put the regular python executable ahead of the > launcher Note that if I *do* get round to working on this, my primary intention will be to propose altering the launcher to allow it to be used under different names, as described by Steve. However, I won't initially be proposing that we add any additional links to the launcher by default, leaving that for users to do manually, and/or for later revisions of the PEP (or further PEPs) to propose this. Reasons: 1. The launcher is typically in C:\Windows, which has a very high priority on PATH, so getting the environment right will be tricky. 2. I don't have any real experience with the installer. 3. Backporting this change, or dealing with older versions of Python that don't include the new launcher, is a complicated question. I'd rather keep the initial PEP restricted to the simple behaviour change. It's possible that the behaviour change may not even need a PEP, but I think it'd be better to have one. The "how do we use the new behaviour in the installers" question is likely to be much more controversial... Paul PS This (particularly the replacement of python.exe) is almost certainly too late for 3.7, so it's not as if there's a rush to answer all the questions at once :-) From gadgetsteve at live.co.uk Fri Nov 17 05:14:34 2017 From: gadgetsteve at live.co.uk (Steve Barnes) Date: Fri, 17 Nov 2017 10:14:34 +0000 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <5A087DE7.4050306@brenbarn.net> <8d5574dc-b614-ccfe-3ce8-f1ba86508179@python.org> Message-ID: On 17/11/2017 09:13, Paul Moore wrote: > On 17 November 2017 at 01:57, Nick Coghlan wrote: >> On 17 November 2017 at 05:15, Chris Barker wrote: >>> >>> On Wed, Nov 15, 2017 at 11:07 AM, Steve Dower >>> wrote: >>>> >>>> If you write such a PEP, please also research and write up the issues >>>> with modifying PATH on Windows (they're largely scattered throughout >>>> bugs.p.o and earlier discussions on python-dev). >>> >>> >>> Is anyone proposing doing anything new with that? (other than changing the >>> default) >>> >>>> My preferred solution for this is to rename "py.exe" to "python.exe" >>> >>> >>> I was going to propose that in this thread, but then thought: "there has >>> GOT to be a reason why that reall obvious solution wan't done in the first >>> place", and didn't have time to go back and research it. >> >> >> As far as I recall, the arguments against it are: >> >> - wanting the regular executable and the launcher to be able to coexist in >> the same build target directory >> - not wanting the regular python executable to prevent access to the >> launcher at a shell prompt >> - not wanting the launcher at a shell prompt to prevent access to the >> regular executable at a shell prompt >> >> However, https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.python.org%2Fdev%2Fpeps%2Fpep-0397%2F&data=02%7C01%7Cgadgetsteve%40live.co.uk%7C18fce28e3baa4e4e70c708d52d9b80c5%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636465068254194278&sdata=4%2BAusacb7TwM5NY9xEUXx%2B%2Ff7s%2BTvV05xIhUWtn9xjg%3D&reserved=0 doesn't spell those out, >> it just states what the launcher's name will be, and emphasises that the >> main purpose is to provide a sensible target for file associations after the >> "always use the most recently installed version" assumption broke down: >> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.python.org%2Fdev%2Fpeps%2Fpep-0397%2F%23rationale&data=02%7C01%7Cgadgetsteve%40live.co.uk%7C18fce28e3baa4e4e70c708d52d9b80c5%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636465068254194278&sdata=OVs0HEVnlob8Rj5LQewOE0YoyIvIA9ydar7i0b1%2BZfY%3D&reserved=0 >> >> Addressing them now: >> >> * as long as the extra hard links are only generated at install time, there >> also won't be any problems with build directory name conflicts. >> * the launcher will always be available a `py`, regardless of the current >> PATH >> * PATH in a venv will still put the regular python executable ahead of the >> launcher > > Note that if I *do* get round to working on this, my primary intention > will be to propose altering the launcher to allow it to be used under > different names, as described by Steve. However, I won't initially be > proposing that we add any additional links to the launcher by default, > leaving that for users to do manually, and/or for later revisions of > the PEP (or further PEPs) to propose this. > > Reasons: > > 1. The launcher is typically in C:\Windows, which has a very high > priority on PATH, so getting the environment right will be tricky. > 2. I don't have any real experience with the installer. > 3. Backporting this change, or dealing with older versions of Python > that don't include the new launcher, is a complicated question. > > I'd rather keep the initial PEP restricted to the simple behaviour > change. It's possible that the behaviour change may not even need a > PEP, but I think it'd be better to have one. The "how do we use the > new behaviour in the installers" question is likely to be much more > controversial... > > Paul > > PS This (particularly the replacement of python.exe) is almost > certainly too late for 3.7, so it's not as if there's a rush to answer > all the questions at once :-) > _______________________________________________ Note that renaming py.exe to python.exe would have several problems: 1. It would only address things on Windows or if py were created for other platforms as python there would be issues with trying to ensure that it was the one that was called first in every case. 1. We would have to also rename python.exe to avoid py.exe executing itself. 1. The current behaviour of py.exe doesn't quite match what I was suggesting in that it always executes a python, (the default or specified), or in 3.7 with the -0 lists the ones that it can find, so would not really address the users installing to the wrong context. I was suggesting that pip, and possibly other tools, would simply execute the command if there was only one python in the current context or prompt the user if there were more than one. (It occurs to me that it might also be valuable to have an all option, e.g. for admins installing security updates). I do like the idea to making this sort of behaviour available, possibly as soft-links or alternate entry points, to tools other than pip as that would be very handy in some cases, so having this behaviour in a python library that pip and other tools could invoke would be ideal. (Having it in an external library also takes care of the old installations issue as with back ports & requires.txt it should have a rapid uptake). A soft-link, entry point or batch file option might run something like, (using pydisamb as a possible name), pip.exe = pydisamb -m pip or possibly examining sys.argv[0] and invoking the found or selected python with -m the same name as the invocation if we can find a way to access that, (it will have to give the invocation name for exe, bat, ps1, sh, lnk, etc. to work). -- Steve (Gadget) Barnes Any opinions in this message are my personal opinions and do not reflect those of my employer. --- This email has been checked for viruses by AVG. http://www.avg.com From victor.stinner at gmail.com Fri Nov 17 10:38:46 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 17 Nov 2017 16:38:46 +0100 Subject: [Python-ideas] Ignorable whitespaces in the re.VERBOSE mode In-Reply-To: References: Message-ID: I don't think that we need more than space (U+0020) and Unix newline (U+000A) ;-) Victor 2017-11-16 11:23 GMT+01:00 Serhiy Storchaka : > Currently the re module ignores only 6 ASCII whitespaces in the re.VERBOSE > mode: > > U+0009 CHARACTER TABULATION > U+000A LINE FEED > U+000B LINE TABULATION > U+000C FORM FEED > U+000D CARRIAGE RETURN > U+0020 SPACE > > Perl ignores characters that Unicode calls "Pattern White Space" in the /x > mode. It ignores additional 5 non-ASCII characters. > > U+0085 NEXT LINE > U+200E LEFT-TO-RIGHT MARK > U+200F RIGHT-TO-LEFT MARK > U+2028 LINE SEPARATOR > U+2029 PARAGRAPH SEPARATOR > > The regex module just ignores characters for which str.isspace() returns > True. It ignores additional 20 non-ASCII whitespace characters, including > characters U+001C..001F whose classification as whitespaces is questionable, > but doesn't ignore LEFT-TO-RIGHT MARK and RIGHT-TO-LEFT MARK. > > U+001C [FILE SEPARATOR] > U+001D [GROUP SEPARATOR] > U+001E [RECORD SEPARATOR] > U+001F [UNIT SEPARATOR] > U+00A0 NO-BREAK SPACE > U+1680 OGHAM SPACE MARK > U+2000 EN QUAD > U+2001 EM QUAD > U+2002 EN SPACE > U+2003 EM SPACE > U+2004 THREE-PER-EM SPACE > U+2005 FOUR-PER-EM SPACE > U+2006 SIX-PER-EM SPACE > U+2007 FIGURE SPACE > U+2008 PUNCTUATION SPACE > U+2009 THIN SPACE > U+200A HAIR SPACE > U+202F NARROW NO-BREAK SPACE > U+205F MEDIUM MATHEMATICAL SPACE > U+3000 IDEOGRAPHIC SPACE > > Is it worth to extend the set of ignored whitespaces to "Pattern > Whitespaces"? Would it add any benefit? Or add confusion? Should this depend > on the re.ASCII mode? Should the byte b'\x85' be ignorable in verbose bytes > patterns? > > And there is a similar question about the Python parser. If Python uses > Unicode definition for identifier, shouldn't it accept non-ASCII "Pattern > Whitespaces" as whitespaces? There will be technical problems with > supporting this, but are there any benefits? > > > https://perldoc.perl.org/perlre.html > https://www.unicode.org/reports/tr31/tr31-4.html#Pattern_Syntax > https://unicode.org/L2/L2005/05012r-pattern.html > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From stephanh42 at gmail.com Fri Nov 17 11:12:49 2017 From: stephanh42 at gmail.com (Stephan Houben) Date: Fri, 17 Nov 2017 17:12:49 +0100 Subject: [Python-ideas] Ignorable whitespaces in the re.VERBOSE mode In-Reply-To: References: Message-ID: I put the actual space characters here so you can see them in a non-proportional font (which I assume most Python programmer use). https://gist.github.com/stephanh42/7c1c122154fd3f2cccc6d864233a40d8 The control characters aren't rendered at all (Vim renders them as ^\ ^] ^^ ^_, respectively). Most of the other spaces are rendered exactly like the normal space. The only ones which render differently are U+1680 | | OGHAM SPACE MARK U+3000 | | IDEOGRAPHIC SPACE I understand Ogham has recently (since 6th century CE) seen a decline in popularity. However, I think Python should totally adopt U+3000 as a new whitespace character and start promoting it as the One True Way to indent code, so as to finally end the age-old spaces vs tabs conflict. [That was supposed to be a joke.] Stephan 2017-11-17 16:38 GMT+01:00 Victor Stinner : > I don't think that we need more than space (U+0020) and Unix newline > (U+000A) ;-) > > Victor > > 2017-11-16 11:23 GMT+01:00 Serhiy Storchaka : > > Currently the re module ignores only 6 ASCII whitespaces in the > re.VERBOSE > > mode: > > > > U+0009 CHARACTER TABULATION > > U+000A LINE FEED > > U+000B LINE TABULATION > > U+000C FORM FEED > > U+000D CARRIAGE RETURN > > U+0020 SPACE > > > > Perl ignores characters that Unicode calls "Pattern White Space" in the > /x > > mode. It ignores additional 5 non-ASCII characters. > > > > U+0085 NEXT LINE > > U+200E LEFT-TO-RIGHT MARK > > U+200F RIGHT-TO-LEFT MARK > > U+2028 LINE SEPARATOR > > U+2029 PARAGRAPH SEPARATOR > > > > The regex module just ignores characters for which str.isspace() returns > > True. It ignores additional 20 non-ASCII whitespace characters, including > > characters U+001C..001F whose classification as whitespaces is > questionable, > > but doesn't ignore LEFT-TO-RIGHT MARK and RIGHT-TO-LEFT MARK. > > > > U+001C [FILE SEPARATOR] > > U+001D [GROUP SEPARATOR] > > U+001E [RECORD SEPARATOR] > > U+001F [UNIT SEPARATOR] > > U+00A0 NO-BREAK SPACE > > U+1680 OGHAM SPACE MARK > > U+2000 EN QUAD > > U+2001 EM QUAD > > U+2002 EN SPACE > > U+2003 EM SPACE > > U+2004 THREE-PER-EM SPACE > > U+2005 FOUR-PER-EM SPACE > > U+2006 SIX-PER-EM SPACE > > U+2007 FIGURE SPACE > > U+2008 PUNCTUATION SPACE > > U+2009 THIN SPACE > > U+200A HAIR SPACE > > U+202F NARROW NO-BREAK SPACE > > U+205F MEDIUM MATHEMATICAL SPACE > > U+3000 IDEOGRAPHIC SPACE > > > > Is it worth to extend the set of ignored whitespaces to "Pattern > > Whitespaces"? Would it add any benefit? Or add confusion? Should this > depend > > on the re.ASCII mode? Should the byte b'\x85' be ignorable in verbose > bytes > > patterns? > > > > And there is a similar question about the Python parser. If Python uses > > Unicode definition for identifier, shouldn't it accept non-ASCII "Pattern > > Whitespaces" as whitespaces? There will be technical problems with > > supporting this, but are there any benefits? > > > > > > https://perldoc.perl.org/perlre.html > > https://www.unicode.org/reports/tr31/tr31-4.html#Pattern_Syntax > > https://unicode.org/L2/L2005/05012r-pattern.html > > > > _______________________________________________ > > Python-ideas mailing list > > Python-ideas at python.org > > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pylang3 at gmail.com Fri Nov 17 13:29:39 2017 From: pylang3 at gmail.com (pylang) Date: Fri, 17 Nov 2017 13:29:39 -0500 Subject: [Python-ideas] Rewriting the "roundrobin" recipe in the itertools documentation In-Reply-To: References: Message-ID: I find this refactored code intriguing. While I would not suggest changes to the itertools recipes, it is a pleasant exercise to think of alternative recipes with an itertools way. There are a few third-party libraries dedicated to itertools recipes. For example, more-itertools has an iter-like recipe called `more_itertools.interleave_longest`. It turns out it behaves like `round_robin` and the implementation happens to be similar to your suggestion: _marker = object() def interleave_longest(*iterables): """Return a new iterable yielding from each iterable in turn, skipping any that are exhausted. >>> list(interleave_longest([1, 2, 3], [4, 5], [6, 7, 8])) [1, 4, 6, 2, 5, 7, 3, 8] """ i = chain.from_iterable(zip_longest(*iterables, fillvalue=_marker)) return filter(lambda x: x is not _marker, i) For comparison, your suggestion: def roundrobin(*iters): > "roundrobin('ABC', 'D', 'EF') --> A D E B F C" > # Perhaps "flat_zip_nofill" is a better name, or something similar > sentinel = object() > for tup in it.zip_longest(*iters, fillvalue=sentinel): > yield from (x for x in tup if x is not sentinel) > In summary, both versions zip iterables and filter sentinels. I believe `yield from` in your suggestion restricts your recipe to Python 3 (which can be resolved with a for loop). Aside, given some performance gain, such third-party libraries may be better suited for your particular contribution. On Thu, Nov 16, 2017 at 8:56 AM, bunslow wrote: > For taking values alternately from a series of iterables, there's two > primary functions: > > builtin.zip > itertools.zip_longest > > zip of course stops when the shortest iterable ends. zip_longest is > generally a useful substitute for when you don't want the zip behavior, but > it fills extra values in the blanks rather than just ignoring a finished > iterator and moving on with the rest. > > This latter most use case is at least somewhat common, according to > this[1] StackOverflow question (and other duplicates), in addition to the > existence of the `roundrobin` recipe[2] in the itertools docs. The recipe > satisfies this use case, and its code is repeated in the StackOverflow > answer. > > However, it is remarkably unpythonic, in my opinion, which is one thing > when such is necessary to achieve a goal, but for this functionality, such > is most definitely *not* necessary. I'll paste the code here for quick > reference: > > > def roundrobin(*iterables): > "roundrobin('ABC', 'D', 'EF') --> A D E B F C" > pending = len(iterables) > nexts = cycle(iter(it).__next__ for it in iterables) > while pending: > try: > for next in nexts: > yield next() > except StopIteration: > pending -= 1 > nexts = cycle(islice(nexts, pending)) > > > Things that strike me as unpythonic: 1) requiring the total number of > input iterables 2) making gratuitous use of `next`, 3) using a while loop > in code dealing with iterables, 4) combining loops, exceptions, and > composed itertools functions in non-obvious ways that make control flow > difficult to determine > > Now, I get it, looking at the "roughly equivalent to" code for zip_longest > in the docs, there doesn't seem to be much way around it for generally > similar goals, and as I said above, unpythonic is fine when necessary > (practicality beats purity), but in this case, for being a "recipe" in the > itertools docs, it should *make use* of the zip_longest which already does > all the unpythonic stuff for you (though honestly I'm not convinced either > that the zip_longest code in the docs is the most possible pythonic-ness). > Instead, the following recipe (which I also submitted to the StackOverflow > question, and which is generally similar to several other later answers, > all remarking that they believe it's more pythonic) is much cleaner and > more suited to demonstrating the power of itertools to new developers than > the mess of a "recipe" pasted above. > > > def roundrobin(*iters): > "roundrobin('ABC', 'D', 'EF') --> A D E B F C" > # Perhaps "flat_zip_nofill" is a better name, or something similar > sentinel = object() > for tup in it.zip_longest(*iters, fillvalue=sentinel): > yield from (x for x in tup if x is not sentinel) > > > In particular, this is just an extremely thin wrapper around zip_longest, > whose primary purpose is to eliminate the otherwise-mandatory "fillvalues" > that zip_longest requires to produce uniform-length tuples. It's also an > excellent example of how to make best pythonic use of iterables in general, > and itertools in particular, and as such a much better implementation to be > demonstrated in documentation. > > I would thus advocate that the former recipe is replaced with the latter > recipe, being much more pythonic, understandable, and useful for helping > new developers acquire the style of python. (Using the common linguistics > analogy: a dictionary and grammar for a spoken language may be enough to > communicate, but we rely on a large body of literature -- fiction, > research, poetry, etc -- as children to get that special flavor and most > expressive taste to the language. The stdlib is no Shakespeare, but it and > its docs still form an important part of the formative literature of the > Python language.) > > I realize at the end of the day this is a pretty trivial and ultimately > meaningless nit to pick, but I've never contributed before and have a > variety of similar minor pain points in the docs/stdlib, and I'm trying to > gauge 1) how well this sort of minor QoL improvement is wanted, and 2) even > if it is wanted, am I going about it the right way. If the answers to both > of these questions are positive regarding this particular case, then I'll > look into making a BPO issue and pull request on GitHub, which IIUC is the > standard path for contributions. > > Thank you for your consideration. > > ~~~~ > > [1]: https://stackoverflow.com/questions/3678869/pythonic- > way-to-combine-two-lists-in-an-alternating-fashion/ > > [2]: https://docs.python.org/3/library/itertools.html#itertools-recipes > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mistersheik at gmail.com Fri Nov 17 17:39:04 2017 From: mistersheik at gmail.com (Neil Girdhar) Date: Fri, 17 Nov 2017 22:39:04 +0000 Subject: [Python-ideas] Consider (one day) adding an inheritance order class precedence mechanism In-Reply-To: <5A0D2BC7.3040209@canterbury.ac.nz> References: <20171116001841.GN19802@ando.pearwood.info> <5A0D2BC7.3040209@canterbury.ac.nz> Message-ID: On Thu, Nov 16, 2017 at 1:11 AM Greg Ewing wrote: > Steven D'Aprano wrote: > > These are not equivalent: > > > > B < S, E > > B < E, S > > Not in general, but in many cases they will be, e.g. if > E and S have no method names in common. I think the OP is > implying that his case is one of those. > > Maybe what's really wanted is a way to say "B inherits from > S and E, but it doesn't care what order they go in". Then > the MRO generating algorithm could in principle swap them > if it would result in a consistent MRO. > Interesting idea, but unfortunately, reordering base classes doesn't solve all of the problems that a precedence specification does. For example, the original example was: R < E, C B < S, E S < C Z < B, R If we make it slightly more complicated: class Y: pass class X(Y): pass class E(X): pass class C: pass class R(E, C): pass class S(C, Y): pass class B(S, E): pass class Z(B, R): pass Then, S and E can't be swapped. > > Or maybe the MRO generator could decide for itself if the > order of two base classes can be swapped by inspecting > their attributes to see if any of them clash? > In general, I think that this would be a cool project, but is much hard than the user declaring a consistent order. > > -- > Greg > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/T7YNKZmwW1c/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mistersheik at gmail.com Fri Nov 17 18:03:13 2017 From: mistersheik at gmail.com (Neil Girdhar) Date: Fri, 17 Nov 2017 23:03:13 +0000 Subject: [Python-ideas] Consider (one day) adding an inheritance order class precedence mechanism In-Reply-To: References: <20171116001841.GN19802@ando.pearwood.info> <5A0D2BC7.3040209@canterbury.ac.nz> <20171117055252.GS19802@ando.pearwood.info> Message-ID: On Fri, Nov 17, 2017 at 3:15 AM Nick Coghlan wrote: > On 17 November 2017 at 15:52, Steven D'Aprano wrote: > > There's a lot of benefit to having a relatively simple, understandable > > algorithm for determining the MRO, as opposed to some sort of adaptive > > rule that will try to reorder classes according to potentially clashing > > constraints. If that means that some classes cannot go together in > > multiple inheritence because their MRO would be inconsistent, I think > > that's a price worth paying for not having to debug inheritence bugs > > caused by weirdly reordered MROs. > > I'll note that an interesting side effect of > https://www.python.org/dev/peps/pep-0560/#mro-entries will be to allow > folks to write: > > class MyCustomMRO: > def __init__(self, *bases): > self._resolved_bases = my_mro_resolver(bases) > def __mro_entries(self, orig_bases): > if len(orig_bases) > 1 or orig_bases[0] is not self: > raise TypeError("CustomMRO instance must be sole base > class") > return self._resolved_bases > > > class Z(MyCustomMRO(B, R)): > ... > > The custom MRO algorithm may then allow for use case specific hints to > handle situations that the default C3 resolver will reject as > inconsistent or ambiguous. (I'll also note that such a custom resolver > would be able to manufacture and inject synthetic metaclasses if > that's what someone decided they really wanted to do, by also > synthesising a custom base class to stick at the head of the list of > bases). > This is really cool! Like you said, this goes above and beyond what I was proposing. I guess my problem could be added as a motivation for PEP-560's __mro_entries__. (cc Ivan) > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- > > --- > You received this message because you are subscribed to a topic in the > Google Groups "python-ideas" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/python-ideas/T7YNKZmwW1c/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > python-ideas+unsubscribe at googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Fri Nov 17 18:09:20 2017 From: brett at python.org (Brett Cannon) Date: Fri, 17 Nov 2017 23:09:20 +0000 Subject: [Python-ideas] venv *is* provided in the standard Python install on Debian/Ubuntu In-Reply-To: References: <914a59a5-57eb-d2d5-cd66-3845af9a899c@gmail.com> <2091af22-1b71-b123-9355-560e205972db@gmail.com> <20171112193851.3c24085f@fsol> Message-ID: On Tue, Nov 14, 2017, 04:06 Stephan Houben, wrote: > Hi Brett, > > The current documentation *cannot* be fixed, since > fixing it would entail adding an initial two-page essay > on "how to start Python on various platforms/systems" > (it is really NOT as simple as Windows=python, Linux=python3) > and such a PR will certainly by rejected. > It doesn't have to be in the venv' docs. Such a page could be a HOWTO or in the tutorial (if it isn't already there). > In my opinion, the only alternatives are > > 1. either harmonize the invocation of python across platforms > (and *then* adapt the docs to follow). > Which was pretty much the whole topic of this thread so far. > The idea has been discussed of developing a 'py' command fo all OSs like on Windows, but has never gone beyond just a discussion. > 2. or just use "python" consistently across all docs > (since that is the *only* command which is at least consistent among > python.org installers), and add weasel-wording to "consult > documentation > of third-party installers" > While 'python' is 'python2' on operating systems that won't really work out well. Best you could do is 'python3'. > 3. or leave the docs broken for at least some people some of the time. > I'm eyeing the removal of pyvenv come February when we fork for 3.8 development so I'm personally not planning to try and clarify anything (obviously others can make an attempt). -brett > Stephan > > > 2017-11-14 2:31 GMT+01:00 Brett Cannon : > >> >> >> On Mon, Nov 13, 2017, 00:01 Stephan Houben, wrote: >> >>> Hi all, >>> >>> Related to this text on https://docs.python.org/3/library/venv.html : >>> >>> ============ >>> >>> Note >>> The pyvenv script has been deprecated as of Python 3.6 in favor of >>> using python3 -m venv to help prevent any potential confusion as to >>> which Python interpreter a virtual environment will be based on. >>> ============ >>> >>> It's clearer than the text below to which I originally referred. >>> >>> However, this text has also problems in that it is too unix-specific. >>> In particular: >>> * Most seriously, it refers to "python3" which doesn't work with the >>> python.org Windows installer. >>> >> >> It can, but it's opt-in. It's just one of those things that's easy to >> forget. >> >> * Less seriously, it refers to "pyenv" as a "script" which is unix jargon >>> and moreover technically >>> incorrect on Windows. (Also, needlessly specific, it should just be >>> "the pyenv command", >>> how it is implemented is irrelevant for this section). >>> >> >> I disagree with this as Python refers to .Py files that you execute >> directly as "scripts", so I don't think this requires clarification. >> >> >> Anyway, a pull request with suggested wording to address your concerns >> would be the best way to try and rectify the issue. >> >> -brett >> >> >> >>> Stephan >>> >>> 2017-11-13 0:32 GMT+01:00 Chris Angelico : >>> >>>> On Mon, Nov 13, 2017 at 10:29 AM, Nick Coghlan >>>> wrote: >>>> > On 13 November 2017 at 07:11, Chris Angelico >>>> wrote: >>>> >> On Mon, Nov 13, 2017 at 6:24 AM, Stephan Houben < >>>> stephanh42 at gmail.com> wrote: >>>> >>> Hi Antoine, >>>> >>> >>>> >>> The venv module is included, >>>> >>> however the pyvenv script is in a separate package >>>> >>> python3.5-venv . >>>> >>> >>>> >>> By the way, I was totally confused by the following text form the >>>> doc. >>>> >>> >>>> >>> https://docs.python.org/3/library/venv.html >>>> >>> >>>> >>> ======== >>>> >>> Deprecated since version 3.6: pyvenv was the recommended tool for >>>> creating >>>> >>> virtual environments for Python 3.3 and 3.4, and is deprecated in >>>> Python >>>> >>> 3.6. >>>> >>> >>>> >>> Changed in version 3.5: The use of venv is now recommended for >>>> creating >>>> >>> virtual environments. >>>> >>> >>>> >>> ======== >>>> >> >>>> >> Not sure where you're reading that. I'm seeing: >>>> >> >>>> >> """ >>>> >> Note >>>> >> The pyvenv script has been deprecated as of Python 3.6 in favor of >>>> >> using python3 -m venv to help prevent any potential confusion as to >>>> >> which Python interpreter a virtual environment will be based on. >>>> >> """ >>>> >> >>>> >> I think that's pretty clear. "python3 -m venv env" is the standard >>>> and >>>> >> recommended way to spin up a virtual environment. >>>> > >>>> > It's further down in the page, under >>>> > >>>> https://docs.python.org/3/library/venv.html#creating-virtual-environments >>>> > >>>> > I think the deprecation notice for pyvenv should just be deleted, >>>> > since it renders like the *module* is deprecated. >>>> >>>> Ah, I see it now, thanks. >>>> >>>> Agreed; or maybe downgrade it to a parenthetical comment. Focus on >>>> "this is how to do the obvious thing", and only as an afterthought >>>> mention "it used to be done differently" in case someone greps for >>>> pyvenv. >>>> >>>> ChrisA >>>> _______________________________________________ >>>> Python-ideas mailing list >>>> Python-ideas at python.org >>>> https://mail.python.org/mailman/listinfo/python-ideas >>>> Code of Conduct: http://python.org/psf/codeofconduct/ >>>> >>> >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Nov 18 09:34:10 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 19 Nov 2017 00:34:10 +1000 Subject: [Python-ideas] Consider (one day) adding an inheritance order class precedence mechanism In-Reply-To: References: <20171116001841.GN19802@ando.pearwood.info> <5A0D2BC7.3040209@canterbury.ac.nz> <20171117055252.GS19802@ando.pearwood.info> Message-ID: On 18 November 2017 at 09:03, Neil Girdhar wrote: > On Fri, Nov 17, 2017 at 3:15 AM Nick Coghlan wrote: >> I'll note that an interesting side effect of >> https://www.python.org/dev/peps/pep-0560/#mro-entries will be to allow >> folks to write: >> >> class MyCustomMRO: >> def __init__(self, *bases): >> self._resolved_bases = my_mro_resolver(bases) >> def __mro_entries(self, orig_bases): >> if len(orig_bases) > 1 or orig_bases[0] is not self: >> raise TypeError("CustomMRO instance must be sole base >> class") >> return self._resolved_bases >> >> >> class Z(MyCustomMRO(B, R)): >> ... >> >> The custom MRO algorithm may then allow for use case specific hints to >> handle situations that the default C3 resolver will reject as >> inconsistent or ambiguous. (I'll also note that such a custom resolver >> would be able to manufacture and inject synthetic metaclasses if >> that's what someone decided they really wanted to do, by also >> synthesising a custom base class to stick at the head of the list of >> bases). > > This is really cool! Unfortunately, as Koos noted, it doesn't actually work as simply as I presented it: even if you spell out a full MRO in the most-derived class, the C3 resolution algorithm will complain that it's inconsistent with the order in one of the two conflicting base classes. So to truly get a custom method resolution order, you're likely going to end up needing a custom metaclass involved. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From mistersheik at gmail.com Sat Nov 18 15:56:43 2017 From: mistersheik at gmail.com (Neil Girdhar) Date: Sat, 18 Nov 2017 20:56:43 +0000 Subject: [Python-ideas] Consider (one day) adding an inheritance order class precedence mechanism In-Reply-To: References: <20171116001841.GN19802@ando.pearwood.info> <5A0D2BC7.3040209@canterbury.ac.nz> <20171117055252.GS19802@ando.pearwood.info> Message-ID: On Sat, Nov 18, 2017 at 9:34 AM Nick Coghlan wrote: > On 18 November 2017 at 09:03, Neil Girdhar wrote: > > On Fri, Nov 17, 2017 at 3:15 AM Nick Coghlan wrote: > >> I'll note that an interesting side effect of > >> https://www.python.org/dev/peps/pep-0560/#mro-entries will be to allow > >> folks to write: > >> > >> class MyCustomMRO: > >> def __init__(self, *bases): > >> self._resolved_bases = my_mro_resolver(bases) > >> def __mro_entries(self, orig_bases): > >> if len(orig_bases) > 1 or orig_bases[0] is not self: > >> raise TypeError("CustomMRO instance must be sole base > >> class") > >> return self._resolved_bases > >> > >> > >> class Z(MyCustomMRO(B, R)): > >> ... > >> > >> The custom MRO algorithm may then allow for use case specific hints to > >> handle situations that the default C3 resolver will reject as > >> inconsistent or ambiguous. (I'll also note that such a custom resolver > >> would be able to manufacture and inject synthetic metaclasses if > >> that's what someone decided they really wanted to do, by also > >> synthesising a custom base class to stick at the head of the list of > >> bases). > > > > This is really cool! > > Unfortunately, as Koos noted, it doesn't actually work as simply as I > presented it: even if you spell out a full MRO in the most-derived > class, the C3 resolution algorithm will complain that it's > inconsistent with the order in one of the two conflicting base > classes. > Would you mind explaining why it's necessary for C3 to complain? In: S < C B < S, E R < E, C Z < B, R If Z is told to have MRO: (Z, B, S, R, E, C) then there are no conflicts with any base classes. > > So to truly get a custom method resolution order, you're likely going > to end up needing a custom metaclass involved. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikhailwas at gmail.com Sat Nov 18 21:01:16 2017 From: mikhailwas at gmail.com (Mikhail V) Date: Sun, 19 Nov 2017 03:01:16 +0100 Subject: [Python-ideas] Allow additional separator character in variables Message-ID: Python allows underscore character as a separator in variables. This is better than nothing, still it does not make the look much better. **Proposal**: allow additional separator, namely hyphen character. **Benefits**: this should provide significant readability improvement. Compared to most syntax change proposals that I remember till now, this seems to have really tangible benefits. So all in all I see it as a significant language improvement. Besides its direct benefit as a good looking separator, it also gives an opprtunity to reduce "camelCase" or similar ugly inclusions in code. So one can easily compose human-readable variable names e.g.: my-variable figure-shape---width etc. **Problem**: currently hyphen character is used as a minus operator! The problem is as old as the history of most programming languages, and inherited from initial poorness of character sets. Therefore I don't see a single optimal solution to this. Possible solutions: Solution 1: Use another similar looking character from unicode, for example: U+02D7 (modifier letter minus sign). At the same time IMO it is needed to allow the minus character for the minus operator, namely U+2212 Minus sign. This will allow proper typography of source code. Main benefit of such approach: no breakage of current code base, since new chars are additional to existing ones. Solution 2: (radical) Disallow hyphen as minus operator, and use only U+2212 Minus sign. I.e. "a-b" would be a variable and "a ? b" a minus operation. Advantage: opportunity to correct the improper character usage once and for all. Disadvantage: breakage of current code base + force UTF-8 storage use (consider e.g. editors without unicode support). Thus most probably such solution will cause howl reaching to the sky among users, despite many modern editors allow unicode and custom operator styling, for example to distinguish dash from hyphen in a monospaced editor. So is my proposal, and as usual urging for constructive conversation. (i.e. proposing to write own language/translator is not constructive conversation) Cheers, Mikhail From antoine.rozo at gmail.com Sat Nov 18 21:11:41 2017 From: antoine.rozo at gmail.com (Antoine Rozo) Date: Sun, 19 Nov 2017 03:11:41 +0100 Subject: [Python-ideas] Allow additional separator character in variables In-Reply-To: References: Message-ID: What will the hyphen bring to what is already allowed with underscore? It is not possible to break compatibility with '-' character. 2017-11-19 3:01 GMT+01:00 Mikhail V : > Python allows underscore character as a separator in variables. > This is better than nothing, still it does not make the look much better. > > **Proposal**: allow additional separator, namely hyphen character. > > **Benefits**: this should provide significant readability improvement. > Compared to most syntax change proposals that I remember till now, > this seems to have really tangible benefits. So all in all I see it as > a significant > language improvement. > Besides its direct benefit as a good looking separator, it also gives > an opprtunity > to reduce "camelCase" or similar ugly inclusions in code. > > So one can easily compose human-readable variable names e.g.: > my-variable > figure-shape---width > etc. > > **Problem**: currently hyphen character is used as a minus operator! > The problem is as old as the history of most programming languages, > and inherited from initial poorness of character sets. > Therefore I don't see a single optimal solution to this. > > Possible solutions: > > Solution 1: > Use another similar looking character from unicode, > for example: U+02D7 (modifier letter minus sign). > At the same time IMO it is needed to allow the minus character > for the minus operator, namely U+2212 Minus sign. This will > allow proper typography of source code. > Main benefit of such approach: no breakage of current code base, > since new chars are additional to existing ones. > > Solution 2: (radical) > Disallow hyphen as minus operator, and use only U+2212 Minus sign. > I.e. "a-b" would be a variable and "a ? b" a minus operation. > Advantage: opportunity to correct the improper character usage once and > for all. > Disadvantage: breakage of current code base + force UTF-8 storage use > (consider > e.g. editors without unicode support). > Thus most probably such solution will cause howl reaching to the sky among > users, despite many modern editors allow unicode and custom operator > styling, > for example to distinguish dash from hyphen in a monospaced editor. > > So is my proposal, and as usual urging for constructive conversation. > (i.e. proposing to write own language/translator is not constructive > conversation) > > > Cheers, > Mikhail > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Antoine Rozo -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sat Nov 18 21:14:44 2017 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 19 Nov 2017 13:14:44 +1100 Subject: [Python-ideas] Allow additional separator character in variables In-Reply-To: References: Message-ID: On Sun, Nov 19, 2017 at 1:01 PM, Mikhail V wrote: > Python allows underscore character as a separator in variables. > This is better than nothing, still it does not make the look much better. > > **Proposal**: allow additional separator, namely hyphen character. > > **Benefits**: this should provide significant readability improvement. > Compared to most syntax change proposals that I remember till now, > this seems to have really tangible benefits. So all in all I see it as > a significant > language improvement. > Besides its direct benefit as a good looking separator, it also gives > an opprtunity > to reduce "camelCase" or similar ugly inclusions in code. Since you can already avoid camelCase by using snake_case, I'm not sure how much you really gain by adding the hyphen. > So one can easily compose human-readable variable names e.g.: > my-variable > figure-shape---width > etc. While I agree with "my-variable", I don't like the triple hyphen. What's the benefit? > **Problem**: currently hyphen character is used as a minus operator! > The problem is as old as the history of most programming languages, > and inherited from initial poorness of character sets. > Therefore I don't see a single optimal solution to this. > > Possible solutions: > > Solution 1: > Use another similar looking character from unicode, > for example: U+02D7 (modifier letter minus sign). > At the same time IMO it is needed to allow the minus character > for the minus operator, namely U+2212 Minus sign. This will > allow proper typography of source code. > Main benefit of such approach: no breakage of current code base, > since new chars are additional to existing ones. > > Solution 2: (radical) > Disallow hyphen as minus operator, and use only U+2212 Minus sign. > I.e. "a-b" would be a variable and "a ? b" a minus operation. > Advantage: opportunity to correct the improper character usage once and for all. > Disadvantage: breakage of current code base + force UTF-8 storage use (consider > e.g. editors without unicode support). > Thus most probably such solution will cause howl reaching to the sky among > users, despite many modern editors allow unicode and custom operator styling, > for example to distinguish dash from hyphen in a monospaced editor. Both of these create extremely confusing situations, where two nearly-identical symbols have completely different meanings. Solution 2 is a massive backward-compatibility break. You're not just disallowing something that's been legal since the language was introduced - you're giving it a completely different meaning. That's basically a non-starter right there. Solution 1 is at least reasonably plausible, in that you're taking something that's currently a SyntaxError and giving it a valid meaning. There is no code that could be broken by that (AFAIK). However, there's still the problem that you're introducing a marginal benefit and a significant confusion potential; plus, you'd be adding a special case to the Unicode identifier rules, which is not something to be done lightly. How much benefit do you REALLY get from using hyphens rather than underscores? ChrisA From ncoghlan at gmail.com Sat Nov 18 21:29:46 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 19 Nov 2017 12:29:46 +1000 Subject: [Python-ideas] Consider (one day) adding an inheritance order class precedence mechanism In-Reply-To: References: <20171116001841.GN19802@ando.pearwood.info> <5A0D2BC7.3040209@canterbury.ac.nz> <20171117055252.GS19802@ando.pearwood.info> Message-ID: On 19 November 2017 at 06:56, Neil Girdhar wrote: > Would you mind explaining why it's necessary for C3 to complain? > > In: > > S < C > B < S, E > R < E, C > Z < B, R > > If Z is told to have MRO: > > (Z, B, S, R, E, C) > > then there are no conflicts with any base classes. I don't actually know what C3 allows in principle, I only know that CPython's resolver still complains in practice: >>> class C: pass ... >>> class S(C): pass ... >>> class E: pass ... >>> class B(S, E): pass ... >>> class R(E, C): pass ... >>> class Z(B, S, R, E, C): pass ... Traceback (most recent call last): File "", line 1, in TypeError: Cannot create a consistent method resolution order (MRO) for bases C, E I think the problem is that the resolver isn't looking at the declared bases of "B", and "R", it's looking at their full MRO: >>> B.__mro__ (, , , , ) >>> R.__mro__ (, , , ) Changing the heuristics used to generate B's MRO such that "C" and "E" appeared in the opposite order wouldn't really help, since that would just flip the problematic case to be the "R(C, E)" declaration. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sat Nov 18 21:32:21 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 19 Nov 2017 12:32:21 +1000 Subject: [Python-ideas] Allow additional separator character in variables In-Reply-To: References: Message-ID: On 19 November 2017 at 12:01, Mikhail V wrote: > Python allows underscore character as a separator in variables. > This is better than nothing, still it does not make the look much better. > > **Proposal**: allow additional separator, namely hyphen character. Regardless of any potential readability merits, backwards compatibility requirements combined with the use of the hyphen character as a binary operator prohibit such a change: >>> my = variable = 1 >>> my-variable 0 Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sat Nov 18 21:42:59 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 19 Nov 2017 12:42:59 +1000 Subject: [Python-ideas] Allow additional separator character in variables In-Reply-To: References: Message-ID: On 19 November 2017 at 12:32, Nick Coghlan wrote: > On 19 November 2017 at 12:01, Mikhail V wrote: >> Python allows underscore character as a separator in variables. >> This is better than nothing, still it does not make the look much better. >> >> **Proposal**: allow additional separator, namely hyphen character. > > Regardless of any potential readability merits, backwards > compatibility requirements combined with the use of the hyphen > character as a binary operator prohibit such a change: > > >>> my = variable = 1 > >>> my-variable > 0 Ah, sorry - I now see you addressed the basic version of that. The alternative of "Use a character that computers can distinguish, but humans can't" isn't an improvement, since it means introducing the exact kind of ambiguity that Python seeks to avoid by using indentation for block delimeters (rather than having the computer read braces, and humans read indentation). The difficulty of reliably distinguishing backticks from regular single quotes is also the main reason they're generally discounted from reintroduction for any other use case after their usage as an alternative to the repr builtin was dropped in Python 3.0, and it's why Python 3 prohibits mixing tabs and spaces for indentation by default. For anyone tempted to suggest "What about multiple underscores indicating continuation of the variable name?", that's still a compatibility problem due to the unary minus operator: >>> my--variable 2 >>> my---variable 0 Would hyphens in variable names improve readability sometimes? Potentially, but not enough to live with make binary subtraction expressions ambiguous (hence the consistency amongst almost all current text based programming languages on this point). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From mikhailwas at gmail.com Sat Nov 18 22:22:51 2017 From: mikhailwas at gmail.com (Mikhail V) Date: Sun, 19 Nov 2017 04:22:51 +0100 Subject: [Python-ideas] Allow additional separator character in variables In-Reply-To: References: Message-ID: Chris A wrote: > Both of these create extremely confusing situations, where two > nearly-identical symbols have completely different meanings. In reality, hyphen and Minus sign are not even closely similar - Minus is ca. twice as wide, however the citizens of the Monospaced Kingdom may disagree ; ) Though I think its population will dramatically decrease in one or two decades. > Solution 2 is a massive backward-compatibility break. Yep, although elimination of improper usage is always good thing in longer perspective (and less new additional chars). But I do realise that it is a non-starter. > ... a marginal benefit ... How much benefit do you REALLY get > from using hyphens rather than underscores? IMO it's far higher than marginal, at least compared to most syntax proposals I remember. One of the hardest and most important tasks which a programmer is faced, is making readable variable names. Underscores are still one of the MOST ugly things I observe currently in Python syntax. This means, if fixings this, then there will be only "small warts" left (such as e.g. single quotes). For me, one "cheap" solution against underscores is to use syntax highlighting which grays them out, but if those become like spaces, then it becomes a bit confusing, e.g. in function with many arguments. Also, unfortunately, not many editors allow easy (if any) highlighting customisation on that level. One possible solution is to use a custom font that has hyphen instead of the underscore, but this is not a proper solution, because, well, the character standard is still there, regardless I like it or not. And one should still have an alternative, i.e. *not only one* separator, for example to denote something "special". Also it can enrich some semantical emphasis, e.g.: my-variable_global Mikhail From ncoghlan at gmail.com Sat Nov 18 23:16:54 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 19 Nov 2017 14:16:54 +1000 Subject: [Python-ideas] Allow additional separator character in variables In-Reply-To: References: Message-ID: On 19 November 2017 at 13:22, Mikhail V wrote: > For me, one "cheap" solution against underscores is to use > syntax highlighting which grays them out, but if those become like > spaces, then it becomes a bit confusing, e.g. in function with many arguments. > Also, unfortunately, not many editors allow easy (if any) highlighting > customisation on that level. Changing the way editors display underscore-using variable names still seems like a more productive direction to explore than changing the text encoding read by the compiler. The current source code structure is well-defined and unambiguous, so there's no clear benefit to change things at that level, and significant downsides in terms of complexity, forwards and backwards compatibility concerns, and high barriers to pervasive adoption. By contrast, if the argument for using a different Unicode character is "Editors will reliably display Unicode hyphen characters differently from the way they display minus signs (or vice-versa)", then we can just as easily say "If users are finding the way that text editors display snake_cased_names to be consistently hard to read, then text editors should change the way that they display snake_cased_names (or at least make it easy for users to opt-in to displaying them differently)". For example, they could decide to replace underscores in variable names for display purposes with hyphens plus the underscore combining diacritic, or the combining macron below: - https://en.wikipedia.org/wiki/Underline#Unicode - https://en.wikipedia.org/wiki/Macron_below Then when the cursor was placed inside the variable name, they could revert to displaying those characters as regular underscores. This kind of editor level modification would also extend itself well to underscores in numeric literals, as there the appropriate pseudo-separator shown when the literal wasn't being edited would be locale dependent. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From mikhailwas at gmail.com Sat Nov 18 23:44:18 2017 From: mikhailwas at gmail.com (Mikhail V) Date: Sun, 19 Nov 2017 05:44:18 +0100 Subject: [Python-ideas] Allow additional separator character in variables In-Reply-To: References: Message-ID: On Sun, Nov 19, 2017 at 3:42 AM, Nick Coghlan wrote: > For anyone tempted to suggest "What about multiple underscores > indicating continuation of the variable name?", that's still a > compatibility problem due to the unary minus operator: > > >>> my--variable > 2 > >>> my---variable > 0 That seems to be another showcase of misfotune that Python uses hyphen for minus operator. I know it is not language designer's fault, because basic ASCII simply did not not include minus character. But do you realise that the **current** problem you are adressing is that font designers forgot to make the minus character (in monospaced font) distinctive from the hyphen character? Well, what can I say, I just think it should be a reason to make a collective complain to font providers, but not that you should silently accept this and adopt the language design to someone's sloppy font design. As an aid for monospace die-hards, to minimise the confusion one could publish a style-guide that recommends to disclose the minus operator (currently hyphen char) in spaces, like a - b, and probably disallow the new proposed hyphen character in the beginning of the identifiers. That would still leave potential for confusion because you cant' force everyone to follow style-guides, but one should struggle to break from this cycle anyway. > > Would hyphens in variable names improve readability sometimes? For reading code, indeed, always and very much. Of course not in case I would be forced to use monospaced font with a similar minus and hyphen. But in that case I am already accepting the level of readability of 12th century, so this would not make things much worse, and I would simply put spaces around the minus operator and try to highlight it with some strong color. Mikhail From prometheus235 at gmail.com Sun Nov 19 00:01:55 2017 From: prometheus235 at gmail.com (Nick Timkovich) Date: Sat, 18 Nov 2017 23:01:55 -0600 Subject: [Python-ideas] Allow additional separator character in variables In-Reply-To: References: Message-ID: Python does not use U+2010 HYPHEN for the minus operator, it uses the U+002D (-) HYPHEN-MINUS. In some monospace fonts, there is a subtle difference between U+002D, U+2013 EN DASH, and U+2014 EM DASH, but it's usually hard to tell them *all* apart. If you want to make a proposal, I'd suggest that you limit it to allowing the U+2010 HYPHEN to be used for names. U+002D simply cannot be changed because it would break billions of lines of code. On Sat, Nov 18, 2017 at 10:44 PM, Mikhail V wrote: > On Sun, Nov 19, 2017 at 3:42 AM, Nick Coghlan wrote: > > > For anyone tempted to suggest "What about multiple underscores > > indicating continuation of the variable name?", that's still a > > compatibility problem due to the unary minus operator: > > > > >>> my--variable > > 2 > > >>> my---variable > > 0 > > That seems to be another showcase of misfotune that Python > uses hyphen for minus operator. I know it is not language designer's > fault, because basic ASCII simply did not not include minus character. > But do you realise that the **current** problem you are adressing is that > font designers forgot to make the minus character (in monospaced font) > distinctive from the hyphen character? > > Well, what can I say, I just think it should be a reason to make a > collective complain to font providers, but not that you should silently > accept this and adopt the language design to someone's sloppy font design. > > As an aid for monospace die-hards, to minimise the confusion one could > publish a style-guide that recommends to disclose the minus operator > (currently > hyphen char) in spaces, like a - b, and probably disallow the new proposed > hyphen character in the beginning of the identifiers. > That would still leave potential for confusion because you cant' force > everyone > to follow style-guides, but one should struggle to break from this cycle > anyway. > > > > > Would hyphens in variable names improve readability sometimes? > > For reading code, indeed, always and very much. Of course not in case > I would be forced > to use monospaced font with a similar minus and hyphen. But > in that case I am already accepting the level of readability of > 12th century, so this would not make things much worse, and I > would simply put spaces around the minus operator and try to highlight > it with some strong color. > > > > Mikhail > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bruce at leban.us Sun Nov 19 02:11:58 2017 From: bruce at leban.us (Bruce Leban) Date: Sat, 18 Nov 2017 23:11:58 -0800 Subject: [Python-ideas] Allow additional separator character in variables In-Reply-To: References: Message-ID: On Sat, Nov 18, 2017 at 8:44 PM, Mikhail V wrote: > > > That seems to be another showcase of misfotune that Python > uses hyphen for minus operator. I know it is not language designer's > fault, because basic ASCII simply did not not include minus character. > But do you realise that the **current** problem you are adressing is that > font designers forgot to make the minus character (in monospaced font) > distinctive from the hyphen character? It is not a misfortune or even true that Python uses hyphen for minus. The name of the character used in Python is HYPHEN-MINUS. http://unicode.org/cldr/utility/character.jsp?a=002D It is both a hyphen and a minus. And it served double-duty even in ASCII. A language that requires using characters not present on standard keyboards is unlikely to be successful. Or we would all be programming in APL. And it's not as if no one every thought of this before. Maybe you've heard of COBOL? > > > Would hyphens in variable names improve readability sometimes? > > For reading code, indeed, always and very much. No it wouldn't. You're personal preference is hardly authoritative. I am extremely skeptical that a legitimate usability study would find that record-count is better than record_count. There are studies that monospace fonts are harder to read than proportionally spaced, e.g., http://journals.sagepub.com/doi/pdf/10.1177/001872088302500303. Yet many programmers use monospace fonts because the advantages -- in our opinions -- outweigh the disadvantages. And the reality is that only my opinion matters when I'm choosing the fonts to display my code in, not yours. You-know-what-really-would-increase-readability? Allowing-the-use-of-spaces-in-variable-names. As-you-can-see-from-this-example-hyphens-between-words-decreases-readability. And because spaces between words is mostly not valid syntax currently, this change would be easier to introduce than breaking every single program out there by re-purposing hyphen-minus. But I'm not seriously proposing this because I think the modest benefits are outweighed by the many problems it would introduce. --- Bruce -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Sun Nov 19 02:38:17 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 19 Nov 2017 09:38:17 +0200 Subject: [Python-ideas] Allow additional separator character in variables In-Reply-To: References: Message-ID: 19.11.17 04:01, Mikhail V ????: > Python allows underscore character as a separator in variables. > This is better than nothing, still it does not make the look much better. > > **Proposal**: allow additional separator, namely hyphen character. You already can use "separators" different from the underscore. my?variable my?variable my?variable my?variable my?variable my?variable Be lucky to distinguish one "separator" from other. From stephanh42 at gmail.com Sun Nov 19 02:42:51 2017 From: stephanh42 at gmail.com (Stephan Houben) Date: Sun, 19 Nov 2017 08:42:51 +0100 Subject: [Python-ideas] Allow additional separator character in variables In-Reply-To: References: Message-ID: Note that Python already allows Unicode characters from class "connector punctuations (Pc)" in identifiers. No dashes in that, but if you are into that kind of thing, then this?is?a?valid?identifier . Stephan Op 19 nov. 2017 08:13 schreef "Bruce Leban" : > > On Sat, Nov 18, 2017 at 8:44 PM, Mikhail V wrote: >> >> >> That seems to be another showcase of misfotune that Python >> uses hyphen for minus operator. I know it is not language designer's >> fault, because basic ASCII simply did not not include minus character. >> But do you realise that the **current** problem you are adressing is that >> font designers forgot to make the minus character (in monospaced font) >> distinctive from the hyphen character? > > > It is not a misfortune or even true that Python uses hyphen for minus. > The name of the character used in Python is HYPHEN-MINUS. > http://unicode.org/cldr/utility/character.jsp?a=002D > It is both a hyphen and a minus. And it served double-duty even in ASCII. > > A language that requires using characters not present on standard > keyboards is unlikely to be successful. > Or we would all be programming in APL. > > And it's not as if no one every thought of this before. Maybe you've heard > of COBOL? > > >> >> > > Would hyphens in variable names improve readability sometimes? >> >> For reading code, indeed, always and very much. > > > No it wouldn't. You're personal preference is hardly authoritative. I am > extremely skeptical that a legitimate usability study would find that > record-count is better than record_count. > > There are studies that monospace fonts are harder to read than > proportionally spaced, e.g., http://journals.sagepub.com/doi/pdf/10.1177/ > 001872088302500303. Yet many programmers use monospace fonts because the > advantages -- in our opinions -- outweigh the disadvantages. And the > reality is that only my opinion matters when I'm choosing the fonts to > display my code in, not yours. > > You-know-what-really-would-increase-readability? > Allowing-the-use-of-spaces-in-variable-names. As-you-can-see-from-this- > example-hyphens-between-words-decreases-readability. > > And because spaces between words is mostly not valid syntax currently, > this change would be easier to introduce than breaking every single program > out there by re-purposing hyphen-minus. But I'm not seriously proposing > this because I think the modest benefits are outweighed by the many > problems it would introduce. > > --- Bruce > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From prometheus235 at gmail.com Sun Nov 19 03:09:14 2017 From: prometheus235 at gmail.com (Nick Timkovich) Date: Sun, 19 Nov 2017 02:09:14 -0600 Subject: [Python-ideas] Allow additional separator character in variables In-Reply-To: References: Message-ID: I guess for reference: exec('a\N{MIDDLE DOT} = 0') exec('\N{BUHID LETTER RA} = 1') exec('\N{HANGUL LETTER EU} = 2') exec('\N{TIFINAGH LETTER YO} = 3') exec('\N{BOPOMOFO LETTER I} = 4') exec('\N{HANGUL LETTER ARAEA} = 5') On Sun, Nov 19, 2017 at 1:38 AM, Serhiy Storchaka wrote: > 19.11.17 04:01, Mikhail V ????: > >> Python allows underscore character as a separator in variables. >> This is better than nothing, still it does not make the look much better. >> >> **Proposal**: allow additional separator, namely hyphen character. >> > > You already can use "separators" different from the underscore. > > my?variable > my?variable > my?variable > my?variable > my?variable > my?variable > > Be lucky to distinguish one "separator" from other. > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gadgetsteve at live.co.uk Sun Nov 19 01:33:09 2017 From: gadgetsteve at live.co.uk (Steve Barnes) Date: Sun, 19 Nov 2017 06:33:09 +0000 Subject: [Python-ideas] Allow additional separator character in variables In-Reply-To: References: Message-ID: On 19/11/2017 05:01, Nick Timkovich wrote: > Python does not use U+2010 HYPHEN for the minus operator, it uses the > U+002D (-) HYPHEN-MINUS. > > In some monospace fonts, there is a subtle difference between U+002D, > U+2013 EN DASH, and U+2014 EM DASH, but it's usually hard to tell them > *all* apart. > > If you want to make a proposal, I'd suggest that you limit it to > allowing the U+2010 HYPHEN to be used for names. U+002D simply cannot be > changed because it would break billions of lines of code. > > On Sat, Nov 18, 2017 at 10:44 PM, Mikhail V > wrote: > > On Sun, Nov 19, 2017 at 3:42 AM, Nick Coghlan > wrote: > > > For anyone tempted to suggest "What about multiple underscores > > indicating continuation of the variable name?", that's still a > > compatibility problem due to the unary minus operator: > > > >? ? ?>>> my--variable > >? ? ?2 > >? ? ?>>> my---variable > >? ? ?0 > > That seems to be another showcase of misfotune that Python > uses hyphen for minus operator. I know it is not language designer's > fault, because basic ASCII simply did not not include minus character. > But do you realise that the **current** problem you are adressing is > that > font designers forgot to make the minus character (in monospaced font) > distinctive from the hyphen character? > > Well, what can I say, I just think it should be a reason to make a > collective complain to font providers, but not that you should silently > accept this and adopt the language design to someone's sloppy font > design. > > As an aid for monospace die-hards, to minimise the confusion one could > publish a style-guide that recommends to disclose the minus operator > (currently > hyphen char)? in spaces, like a - b, and probably disallow the new > proposed > hyphen character in the beginning of the identifiers. > That would still leave potential for confusion because you cant' > force everyone > to follow style-guides, but one should struggle to break from this > cycle anyway. > > > > > Would hyphens in variable names improve readability sometimes? > > For reading code, indeed, always and very much. Of course not in case > I would be forced > to use monospaced font with a similar minus and hyphen. But > in that case I am already accepting the level of readability of > 12th century, so this would not make things much worse, and I > would simply put spaces around the minus operator and try to highlight > it with some strong color. > How about allowing ?, (ASCII 172, U+00ac, NOT sign), in variable names as in my?variable - it has the advantages that: - it is visually distinguishable even in mono-spaced fonts, (personally I use mono-spaced all of the time when programming but I know that I am a dinosaur), - is actually on many keyboards as a single character, (I don't know of any which actually produce different characters for minus on the numeric keypad and hyphen elsewhere), so can be typed as a single key press, - Is generally unused AFAIK other than in papers about logic, - It is currently unused in the Python language. This might upset some who would like use it to replace the unary not operator but I suspect that it would be far fewer people than the potential breakages discussed so far. -- Steve (Gadget) Barnes Any opinions in this message are my personal opinions and do not reflect those of my employer. --- This email has been checked for viruses by AVG. http://www.avg.com From stephanh42 at gmail.com Sun Nov 19 05:20:05 2017 From: stephanh42 at gmail.com (Stephan Houben) Date: Sun, 19 Nov 2017 11:20:05 +0100 Subject: [Python-ideas] Allow additional separator character in variables In-Reply-To: References: Message-ID: There is an unfortunate ambiguity in using a character that means "not" as a word separator: nuke.do?launch() "But... I called the method which explicitly did *not* launch the nuke!" Stephan Op 19 nov. 2017 11:05 schreef "Steve Barnes" : On 19/11/2017 05:01, Nick Timkovich wrote: > Python does not use U+2010 HYPHEN for the minus operator, it uses the > U+002D (-) HYPHEN-MINUS. > > In some monospace fonts, there is a subtle difference between U+002D, > U+2013 EN DASH, and U+2014 EM DASH, but it's usually hard to tell them > *all* apart. > > If you want to make a proposal, I'd suggest that you limit it to > allowing the U+2010 HYPHEN to be used for names. U+002D simply cannot be > changed because it would break billions of lines of code. > > On Sat, Nov 18, 2017 at 10:44 PM, Mikhail V > wrote: > > On Sun, Nov 19, 2017 at 3:42 AM, Nick Coghlan > wrote: > > > For anyone tempted to suggest "What about multiple underscores > > indicating continuation of the variable name?", that's still a > > compatibility problem due to the unary minus operator: > > > > >>> my--variable > > 2 > > >>> my---variable > > 0 > > That seems to be another showcase of misfotune that Python > uses hyphen for minus operator. I know it is not language designer's > fault, because basic ASCII simply did not not include minus character. > But do you realise that the **current** problem you are adressing is > that > font designers forgot to make the minus character (in monospaced font) > distinctive from the hyphen character? > > Well, what can I say, I just think it should be a reason to make a > collective complain to font providers, but not that you should silently > accept this and adopt the language design to someone's sloppy font > design. > > As an aid for monospace die-hards, to minimise the confusion one could > publish a style-guide that recommends to disclose the minus operator > (currently > hyphen char) in spaces, like a - b, and probably disallow the new > proposed > hyphen character in the beginning of the identifiers. > That would still leave potential for confusion because you cant' > force everyone > to follow style-guides, but one should struggle to break from this > cycle anyway. > > > > > Would hyphens in variable names improve readability sometimes? > > For reading code, indeed, always and very much. Of course not in case > I would be forced > to use monospaced font with a similar minus and hyphen. But > in that case I am already accepting the level of readability of > 12th century, so this would not make things much worse, and I > would simply put spaces around the minus operator and try to highlight > it with some strong color. > How about allowing ?, (ASCII 172, U+00ac, NOT sign), in variable names as in my?variable - it has the advantages that: - it is visually distinguishable even in mono-spaced fonts, (personally I use mono-spaced all of the time when programming but I know that I am a dinosaur), - is actually on many keyboards as a single character, (I don't know of any which actually produce different characters for minus on the numeric keypad and hyphen elsewhere), so can be typed as a single key press, - Is generally unused AFAIK other than in papers about logic, - It is currently unused in the Python language. This might upset some who would like use it to replace the unary not operator but I suspect that it would be far fewer people than the potential breakages discussed so far. -- Steve (Gadget) Barnes Any opinions in this message are my personal opinions and do not reflect those of my employer. --- This email has been checked for viruses by AVG. http://www.avg.com _______________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sun Nov 19 06:05:33 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 19 Nov 2017 12:05:33 +0100 Subject: [Python-ideas] Allow additional separator character in variables References: Message-ID: <20171119120533.04efa9c4@fsol> On Sun, 19 Nov 2017 09:38:17 +0200 Serhiy Storchaka wrote: > 19.11.17 04:01, Mikhail V ????: > > Python allows underscore character as a separator in variables. > > This is better than nothing, still it does not make the look much better. > > > > **Proposal**: allow additional separator, namely hyphen character. > > You already can use "separators" different from the underscore. > > my?variable > my?variable > my?variable > my?variable > my?variable > my?variable This is going to make for some fun code reviews. Regards Antoine. From stephanh42 at gmail.com Sun Nov 19 06:20:34 2017 From: stephanh42 at gmail.com (Stephan Houben) Date: Sun, 19 Nov 2017 12:20:34 +0100 Subject: [Python-ideas] Allow additional separator character in variables In-Reply-To: <20171119120533.04efa9c4@fsol> References: <20171119120533.04efa9c4@fsol> Message-ID: You think that's bad? https://github.com/reinderien/mimic/blob/master/README.md Abandon all hope ye who use Unicode. Op 19 nov. 2017 12:06 schreef "Antoine Pitrou" : > On Sun, 19 Nov 2017 09:38:17 +0200 > Serhiy Storchaka > wrote: > > 19.11.17 04:01, Mikhail V ????: > > > Python allows underscore character as a separator in variables. > > > This is better than nothing, still it does not make the look much > better. > > > > > > **Proposal**: allow additional separator, namely hyphen character. > > > > You already can use "separators" different from the underscore. > > > > my?variable > > my?variable > > my?variable > > my?variable > > my?variable > > my?variable > > This is going to make for some fun code reviews. > > Regards > > Antoine. > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gadgetsteve at live.co.uk Sun Nov 19 08:18:52 2017 From: gadgetsteve at live.co.uk (Steve Barnes) Date: Sun, 19 Nov 2017 13:18:52 +0000 Subject: [Python-ideas] Allow additional separator character in variables In-Reply-To: References: <20171119120533.04efa9c4@fsol> Message-ID: On 19/11/2017 11:20, Stephan Houben wrote: > You think that's bad? > https://github.com/reinderien/mimic/blob/master/README.md > Abandon all hope ye who use Unicode. > > Op 19 nov. 2017 12:06 schreef "Antoine Pitrou" References: Message-ID: <82a79928-739d-e37a-a6d7-44fd53eb3b0f@Damon-Family.org> On 11/19/17 1:33 AM, Steve Barnes wrote: > How about allowing ?, (ASCII 172, U+00ac, NOT sign), in variable names > as in my?variable - it has the advantages that: There is NO such character in ASCII. ASCII is a 7 bit character set, and no ASCII code has a value bigger than 127. There are a number of Extended ASCII character sets (hundreds if not thousands). One common one is ISO 8859-1 also called ISO LATIN-1 which has this character at this location, but Extended ASCII is NOT ASCII (Note, it is even produced by a very different standards body). The character also occurs here in ANSI Extended ASCII, but again, this is NOT ASCII. -- Richard Damon From greg.ewing at canterbury.ac.nz Sun Nov 19 16:24:39 2017 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 20 Nov 2017 10:24:39 +1300 Subject: [Python-ideas] Allow additional separator character in variables In-Reply-To: References: Message-ID: <5A11F697.30600@canterbury.ac.nz> In summary, this proposal seems to be: Give two visually indistinguishible characters different meanings to improve readability. I'm not sure, but something about that sentence doesn't seem quite right. -- Greg From greg.ewing at canterbury.ac.nz Sun Nov 19 16:40:55 2017 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 20 Nov 2017 10:40:55 +1300 Subject: [Python-ideas] Allow additional separator character in variables In-Reply-To: References: Message-ID: <5A11FA67.30107@canterbury.ac.nz> Mikhail V wrote: > In reality, hyphen and Minus sign are not even closely similar - > Minus is ca. twice as wide, If you are using a font that distinguishes them that clearly, and if the human reader is sufficiently typographically aware to notice the distinction. Both of those are big ifs. And what about handwritten code? I don't know anyone who handwrites hyphens and minuses with precision measured in points. -- Greg From mikhailwas at gmail.com Sun Nov 19 19:01:59 2017 From: mikhailwas at gmail.com (Mikhail V) Date: Mon, 20 Nov 2017 01:01:59 +0100 Subject: [Python-ideas] Allow additional separator character in variables In-Reply-To: References: Message-ID: On Sun, Nov 19, 2017 at 5:16 AM, Nick Coghlan wrote: > On 19 November 2017 at 13:22, Mikhail V wrote: >> For me, one "cheap" solution against underscores is to use >> syntax highlighting which grays them out, but if those become like >> spaces, then it becomes a bit confusing, e.g. in function with many arguments. >> Also, unfortunately, not many editors allow easy (if any) highlighting >> customisation on that level. > > Changing the way editors display underscore-using variable names still > seems like a more productive direction to explore than changing the > text encoding read by the compiler. Indeed that would be a solution. *Would* be. But I don't know of any editor that does that afaik (and they should not in this case, see below). My view on pros&cons for this solution: Pros: other languages also have the same issue, so if editors maintainers would agree to compromise and introduce feature of dynamic substitution, that would give users possibility to face-lift other syntaxes as well. Cons: this feature would make sense if the substitution happens only in those part where it should, namely it should not touch anything in string literals, comment blocks. So the lexer should 'know' where to substitute or not and it is not the same as just passing the internal memory representation through a translation table. My opinion about this however is based on other principles. Imagine that you are the language designer and I am responsible for the typesetting component of some editor, and we have such a dialogue: you: "hey Mikhail, we use hyphen for minus operator, now can you please patch the renderer so that our users see the minus instead of hyphen, and please make sure users can also toggle it in real time to see what actual char is there and also make the substitution only in the places where hyphen is used as the operator." me: "well, I understand your complain, but my renderer already supports Unicode, and I do my best to support typography practices, namely render hyphen as *hyphen*, which is well established for centuries in typography, and defined as a dash of 50% width of the letter "o" and is aligned to lowercase. As well as the Minus glyph which is defined as ca. 110% of "o" width and is aligned to the digits&caps. So you as the language designer should be interested to deliver best practices to the users, and hyphen is way more important for the lexical structure of the written language, than the minus operator. Why would not you just try to solve the issue in a "fair" way?" By the fair way I understand the way which tends to bring the correct usage of characters back, instead of trying to hide the problem with some patch. Now I can't say what is the least problematic way for Python, but if I were responsible for that, I would base the solution on these principles: 1. The future versions of syntax, ideally, must allow ONLY minus U2212 for the minus operator, and allow hyphens 002D in identifiers. Since it is impossible to the current moment, I must think out the least painful transition. 2. I want users to be able to use underscore as well. Underscore is derived from the mechanical type-writers - to make an underlined text one pushed the carriage back and tipped the underscore to make the line under the text. Currently in digital print it does not make much sense and as a separator looks ugly, but still it not so hopeless. Currently the underscore lies below the font baseline but if one makes it closer to the baseline, then it can be used as a fairly adequate additional separator, so a user would become more ways to denote lexical identifiers. 3. I don't want to break the backward-compatibility but still I am oriented on compliance with typography practices and standards for charcodes. Also I want users who are interested in better UX become the benefits out-of-the-box, without forcing them to tweak the text-editors or writing own translators. What to do? One option IMO would be to introduce a header in the sources, e.g.: # opt-in: hyphen-minus Which would tell the parser to toggle the "new" rules, namely U+2212 would be parsed as minus operator and hyphens as part of identifiers. Then users who are aware of benefits and remember monospaced fonts only as unpleasant incident from their youth, can enjoy the beauty of source code without any tweaks, and the only thing they need to do is to bind a key to input the U+2212 sign. The users who do not want it, just leave this out. Further, I'd add a command-line util that can directly translate to the "old" syntax, in case one want to export a project in old syntax. So one could avoid backward compatibility issue. That is just one option that comes to my mind. Another thing which might be important in this regard: Say you want to publish a book about Python. With such syntax you could directly import the code into a DTP software, and you don't need to make any corrections, so it looks almost as a normal English text, and no worries about strange looking minus operators. Mikhail From mikhailwas at gmail.com Sun Nov 19 19:18:24 2017 From: mikhailwas at gmail.com (Mikhail V) Date: Mon, 20 Nov 2017 01:18:24 +0100 Subject: [Python-ideas] Allow additional separator character in variables In-Reply-To: References: Message-ID: Bruce Leban wrote: > It is not a misfortune or even true that Python uses hyphen for minus. > The name of the character used in Python is HYPHEN-MINUS. This is pure demagogy, name it HYPHEN-MINUS-TINYDASH if you like, but what aspect of reality does it change apart of its name? "Hyphen-minus" would make sense for mechanical type-writers. So it is a hyphen, a character used for centuries before typewriters even appeared, and used as such now in 99 percent of medium. Just take some Python sources and count the amount of underscores and minus operators. This will give you an image of how important separators are compared to minus operator. Don't forget also to include cases where variables are written without any separator, but should do so. > I am extremely skeptical that a legitimate usability study would > find that record-count is better than record_count. Oh come on, probably you also want study for emoticons as a separators? On Sun, Nov 19, 2017 at 5:16 AM, Nick Coghlan wrote: > On 19 November 2017 at 13:22, Mikhail V wrote: >> For me, one "cheap" solution against underscores is to use >> syntax highlighting which grays them out, but if those become like >> spaces, then it becomes a bit confusing, e.g. in function with many arguments. >> Also, unfortunately, not many editors allow easy (if any) highlighting >> customisation on that level. > > Changing the way editors display underscore-using variable names still > seems like a more productive direction to explore than changing the > text encoding read by the compiler. The current source code structure > is well-defined and unambiguous, so there's no clear benefit to change > things at that level, and significant downsides in terms of > complexity, forwards and backwards compatibility concerns, and high > barriers to pervasive adoption. > > By contrast, if the argument for using a different Unicode character > is "Editors will reliably display Unicode hyphen characters > differently from the way they display minus signs (or vice-versa)", > then we can just as easily say "If users are finding the way that text > editors display snake_cased_names to be consistently hard to read, > then text editors should change the way that they display > snake_cased_names (or at least make it easy for users to opt-in to > displaying them differently)". > > For example, they could decide to replace underscores in variable > names for display purposes with hyphens plus the underscore combining > diacritic, or the combining macron below: > > - https://en.wikipedia.org/wiki/Underline#Unicode > - https://en.wikipedia.org/wiki/Macron_below > > Then when the cursor was placed inside the variable name, they could > revert to displaying those characters as regular underscores. > > This kind of editor level modification would also extend itself well > to underscores in numeric literals, as there the appropriate > pseudo-separator shown when the literal wasn't being edited would be > locale dependent. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From rosuav at gmail.com Sun Nov 19 19:20:01 2017 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 20 Nov 2017 11:20:01 +1100 Subject: [Python-ideas] Allow additional separator character in variables In-Reply-To: References: Message-ID: On Mon, Nov 20, 2017 at 11:01 AM, Mikhail V wrote: > 1. The future versions of syntax, ideally, must allow ONLY minus U2212 for > the minus operator, and allow hyphens 002D in identifiers. Since it is > impossible > to the current moment, I must think out the least painful transition. The least painful transition is to devise an entirely new language, one that is built around whatever rules you like. That way, there's no backward compatibility problem - you pick a new file extension, a new executable name, etc, etc, and nobody gets confused. Of course, since actually building a cross-platform language interpreter is a ton of work, and getting an ecosystem of libraries is even more work, you'll want to make your language compile to Python, but *in your source code* you can use whatever symbols you want. Since you want U+2212 for subtraction, you probably want to use a few other non-ASCII operators too. U+2044 FRACTION SLASH presents itself as a viable way to create a fractions.Fraction literal. Instead of * and @ for multiplication, you could have U+00D7 and... uhh, I'm not a mathematician, but I'm sure there's an appropriate character. For the most part, you'd have code that is trivially transformable to and from Python. Start by writing the "my language to Python" translator (it can throw away comments and stuff, the Python code should be considered "object code" rather than "source code"), and then look into the reverse transformation for the benefit of people trying to learn your language. As long as you don't actually call your language "Python", you're free to do what you like without worrying about compatibility etc. ChrisA From rosuav at gmail.com Sun Nov 19 19:24:29 2017 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 20 Nov 2017 11:24:29 +1100 Subject: [Python-ideas] Allow additional separator character in variables In-Reply-To: References: Message-ID: On Mon, Nov 20, 2017 at 11:18 AM, Mikhail V wrote: > Bruce Leban wrote: > >> It is not a misfortune or even true that Python uses hyphen for minus. >> The name of the character used in Python is HYPHEN-MINUS. > > This is pure demagogy, name it HYPHEN-MINUS-TINYDASH if you like, > but what aspect of reality does it change apart of its name? > "Hyphen-minus" would make sense for mechanical type-writers. > So it is a hyphen, a character used for centuries before typewriters even > appeared, and used as such now in 99 percent of medium. > Just take some Python sources and count the amount of underscores > and minus operators. This will give you an image of how important > separators are compared to minus operator. > Don't forget also to include cases where variables are written without > any separator, but should do so. > >> I am extremely skeptical that a legitimate usability study would >> find that record-count is better than record_count. > > Oh come on, probably you also want study for emoticons as a separators? If you want to. But a simple a-b test (or is that an a_b test?) of hyphens and underscores would be sufficient. For anecdotal evidence, I prefer to write git branch names with hyphens, eg "git checkout rosuav/process-check-run". It's not about the typing (tab completion means I don't have to type either form), it's about the way it looks. So there definitely is _some_ advantage here. I just don't think it's significant, not worth the hassle of changing things around. And this is still ASCII-only. ChrisA From python at mrabarnett.plus.com Sun Nov 19 22:12:50 2017 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 20 Nov 2017 03:12:50 +0000 Subject: [Python-ideas] Allow additional separator character in variables In-Reply-To: References: Message-ID: <337b28e7-100f-5407-195d-9f6010a9a49a@mrabarnett.plus.com> On 2017-11-20 00:20, Chris Angelico wrote: > On Mon, Nov 20, 2017 at 11:01 AM, Mikhail V wrote: >> 1. The future versions of syntax, ideally, must allow ONLY minus U2212 for >> the minus operator, and allow hyphens 002D in identifiers. Since it is >> impossible >> to the current moment, I must think out the least painful transition. > > The least painful transition is to devise an entirely new language, > one that is built around whatever rules you like. That way, there's no > backward compatibility problem - you pick a new file extension, a new > executable name, etc, etc, and nobody gets confused. Of course, since > actually building a cross-platform language interpreter is a ton of > work, and getting an ecosystem of libraries is even more work, you'll > want to make your language compile to Python, but *in your source > code* you can use whatever symbols you want. > > Since you want U+2212 for subtraction, you probably want to use a few > other non-ASCII operators too. U+2044 FRACTION SLASH presents itself > as a viable way to create a fractions.Fraction literal. Instead of * > and @ for multiplication, you could have U+00D7 and... uhh, I'm not a > mathematician, but I'm sure there's an appropriate character. > If we must use U+2212 (MINUS SIGN) for the minus sign, then it's only right that we must also use U+2010 (HYPHEN) for the hyphen. U+002D (HYPHEN-MINUS) can be left alone, its meaning depending on the programming language, as at present. > For the most part, you'd have code that is trivially transformable to > and from Python. Start by writing the "my language to Python" > translator (it can throw away comments and stuff, the Python code > should be considered "object code" rather than "source code"), and > then look into the reverse transformation for the benefit of people > trying to learn your language. > > As long as you don't actually call your language "Python", you're free > to do what you like without worrying about compatibility etc. > From bruce at leban.us Sun Nov 19 22:57:02 2017 From: bruce at leban.us (Bruce Leban) Date: Sun, 19 Nov 2017 19:57:02 -0800 Subject: [Python-ideas] Allow additional separator character in variables In-Reply-To: References: Message-ID: On Sun, Nov 19, 2017 at 4:18 PM, Mikhail V wrote: > Bruce Leban wrote: > > > It is not a misfortune or even true that Python uses hyphen for minus. > > The name of the character used in Python is HYPHEN-MINUS. > > This is pure demagogy, name it HYPHEN-MINUS-TINYDASH if you like, > but what aspect of reality does it change apart of its name? > You've gone from making a bad suggestion to trolling. While you may *think* this character is a hyphen, you're simply wrong. When ASCII was created it was a 7-bit character set limited to 94 printable characters plus space and 33 control characters. The designers explicitly added a single double-duty character for both hyphen and minus, just as they added a single character for single and double quotes rather than left and right quotes. Were they mimicking the typewriter? Maybe they were following the example of Hollerith code which only had uppercase. It doesn't matter. It's not that they were unaware of the different uses or the existence of typographic quotes. Just as monospace fonts were not created because people didn't know about variable width fonts. It is what it is. And pretending other people are idiots is inappropriate. You can use accent grave as a left quote and apostrophe as a right quote if you want to, but if you insist that Python is living in the dark ages because it doesn't do things *your way* then you're just being rude. ... render > hyphen as *hyphen*, which is well established for centuries in typography, > and defined as a dash of 50% width of the letter "o" and is aligned to > lowercase. > As well as the Minus glyph which is defined as ca. 110% of "o" width > and is aligned False. There is no standard going back centuries defining the widths of the different kinds of dashes. For that matter, there is no standard *today* for what letters and symbols look like. See Doug Hofstadter's great paper on this https://web.stanford.edu/group/SHR/4-2/text/hofstadter.html or the Unicode consortium list of emoji https://unicode.org/emoji/charts/full-emoji-list.html for great examples of the non-standard nature of typography. Heck almost all vendors put cheese on the *hamburger* emoji when obviously it only belongs on the cheeseburger emoji. And Google puts the cheese below the meat which is clearly wrong as the international standard for cheeseburgers puts the cheese on the top. Just take some Python sources and count the amount of underscores > and minus operators. This will give you an image of how important > separators are compared to minus operator. > A non sequitur. Count the number of instances of the letter Z in English vs. the letter E which tells you that Z is unimportant. So let's get rid of it. Of course that may piss off the Polish people since it's the 9th most frequent letter (4.9%) in Polish. While this makes a great story -- see "Meihem In Ce Klasrum" http://www.tau.ac.il/~pauzner/funs/simpler.html -- but not a great reality. That said, no one has argued that a word separator in names is a bad idea and we have two choices: capitalizingEachWord and underscores_between_words. These work well enough that the idea of breaking every single Python program that uses subtraction just because one person believes we are being antediluvian -- without any evidence -- is just not going to happen. (Ooh. See what I did there. I typed two hyphen-minus characters to get an "em dash" and you probably didn't even notice that I was breaking centuries of tradition that the only proper way to write an em dash is with a single piece of metal type.) If you want to make serious contributions to Python or any other project you need to understand why this is a bad idea. > > > I am extremely skeptical that a legitimate usability study would > > find that record-count is better than record_count. > > Oh come on, probably you also want study for emoticons as a separators? > Yes, if someone insisted that emoticons were superior to underscores as separators and implied I was an idiot for not agreeing with that. --- Bruce -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sun Nov 19 23:46:39 2017 From: guido at python.org (Guido van Rossum) Date: Sun, 19 Nov 2017 20:46:39 -0800 Subject: [Python-ideas] Allow additional separator character in variables In-Reply-To: References: Message-ID: Please kill this thread. On Sun, Nov 19, 2017 at 7:57 PM, Bruce Leban wrote: > > On Sun, Nov 19, 2017 at 4:18 PM, Mikhail V wrote: > >> Bruce Leban wrote: >> >> > It is not a misfortune or even true that Python uses hyphen for minus. >> > The name of the character used in Python is HYPHEN-MINUS. >> >> This is pure demagogy, name it HYPHEN-MINUS-TINYDASH if you like, >> but what aspect of reality does it change apart of its name? >> > > You've gone from making a bad suggestion to trolling. > > While you may *think* this character is a hyphen, you're simply wrong. > When ASCII was created it was a 7-bit character set limited to 94 printable > characters plus space and 33 control characters. The designers explicitly > added a single double-duty character for both hyphen and minus, just as > they added a single character for single and double quotes rather than left > and right quotes. Were they mimicking the typewriter? Maybe they were > following the example of Hollerith code which only had uppercase. It > doesn't matter. It's not that they were unaware of the different uses or > the existence of typographic quotes. Just as monospace fonts were not > created because people didn't know about variable width fonts. It is what > it is. And pretending other people are idiots is inappropriate. > > You can use accent grave as a left quote and apostrophe as a right quote > if you want to, but if you insist that Python is living in the dark ages > because it doesn't do things *your way* then you're just being rude. > > ... render >> hyphen as *hyphen*, which is well established for centuries in typography, >> and defined as a dash of 50% width of the letter "o" and is aligned to >> lowercase. >> As well as the Minus glyph which is defined as ca. 110% of "o" width >> and is aligned > > > False. There is no standard going back centuries defining the widths of > the different kinds of dashes. For that matter, there is no standard > *today* for what letters and symbols look like. See Doug Hofstadter's great > paper on this https://web.stanford.edu/group/SHR/4-2/text/hofstadter.html > or the Unicode consortium list of emoji https://unicode.org/emoji/ > charts/full-emoji-list.html for great examples of the non-standard nature > of typography. Heck almost all vendors put cheese on the *hamburger* emoji > when obviously it only belongs on the cheeseburger emoji. And Google puts > the cheese below the meat which is clearly wrong as the international > standard for cheeseburgers puts the cheese on the top. > > Just take some Python sources and count the amount of underscores >> and minus operators. This will give you an image of how important >> separators are compared to minus operator. >> > > A non sequitur. Count the number of instances of the letter Z in English > vs. the letter E which tells you that Z is unimportant. So let's get rid of > it. Of course that may piss off the Polish people since it's the 9th most > frequent letter (4.9%) in Polish. While this makes a great story -- see > "Meihem In Ce Klasrum" http://www.tau.ac.il/~pauzner/funs/simpler.html -- > but not a great reality. > > That said, no one has argued that a word separator in names is a bad idea > and we have two choices: capitalizingEachWord and > underscores_between_words. These work well enough that the idea of breaking > every single Python program that uses subtraction just because one person > believes we are being antediluvian -- without any evidence -- is just not > going to happen. (Ooh. See what I did there. I typed two hyphen-minus > characters to get an "em dash" and you probably didn't even notice that I > was breaking centuries of tradition that the only proper way to write an em > dash is with a single piece of metal type.) > > If you want to make serious contributions to Python or any other project > you need to understand why this is a bad idea. > > >> >> > I am extremely skeptical that a legitimate usability study would >> > find that record-count is better than record_count. >> >> Oh come on, probably you also want study for emoticons as a separators? >> > > Yes, if someone insisted that emoticons were superior to underscores as > separators and implied I was an idiot for not agreeing with that. > > --- Bruce > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikhailwas at gmail.com Mon Nov 20 00:48:08 2017 From: mikhailwas at gmail.com (Mikhail V) Date: Mon, 20 Nov 2017 06:48:08 +0100 Subject: [Python-ideas] Allow additional separator character in variables In-Reply-To: References: Message-ID: On Mon, Nov 20, 2017 at 5:46 AM, Guido van Rossum wrote: > Please kill this thread. So the idea is too bad? From turnbull.stephen.fw at u.tsukuba.ac.jp Mon Nov 20 03:13:48 2017 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Mon, 20 Nov 2017 17:13:48 +0900 Subject: [Python-ideas] Ignorable whitespaces in the re.VERBOSE mode In-Reply-To: References: Message-ID: <23058.36540.802392.890565@turnbull.sk.tsukuba.ac.jp> All, If we *do* seriously consider adding those characters as ignorable in re.VERBOSE, that's because somebody is using them in text a lot, and it slips into their coding. Given frequent use, we should consider how a lot more whitespace characters can be conveniently searched individually and readably, because whitespace characters are the ultimate confusables. This may be a no-op, given the \N and \u notations, but \u is pretty opaque and \N leads to character-per-line regexes. ;-) Otherwise I'm with Paul, who writes: > My instinct is not to worry about it unless someone has actually hit > the issue in practice and raised a bug. After the tabs vs. spaces fiasco, I lean steeply to the right for code -- including embedded languages like regexes. *We* say what is allowed there, and *you* can find an editor that does it our way. The point of re.VERBOSE is to allow writing regexes the way we write Python code, formatting to emphasize structure and improve readability. I don't see why we would want to allow more than we already do, given that any fancy whitespace formatting for "literate programming" will be done by the code formatting engine of the document preparation system anyway. From storchaka at gmail.com Mon Nov 20 03:59:15 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 20 Nov 2017 10:59:15 +0200 Subject: [Python-ideas] Ignorable whitespaces in the re.VERBOSE mode In-Reply-To: <23058.36540.802392.890565@turnbull.sk.tsukuba.ac.jp> References: <23058.36540.802392.890565@turnbull.sk.tsukuba.ac.jp> Message-ID: 20.11.17 10:13, Stephen J. Turnbull ????: > Otherwise I'm with Paul, who writes: > > > My instinct is not to worry about it unless someone has actually hit > > the issue in practice and raised a bug. > > After the tabs vs. spaces fiasco, I lean steeply to the right for code > -- including embedded languages like regexes. *We* say what is > allowed there, and *you* can find an editor that does it our way. I agree. But if there is a special part of the Unicode standard for Pattern White Spaces which includes non-ASCII characters, perhaps there is a need in them. I asked for the case if Python developers with very different cultures have need in additional whitespaces in regular expressions, but I don't know. Seems nobody has claimed their need. In particularly I don't know how helpful would be supporting right-to-left and left-to-right marks in verbose regular expressions (or even in Python code), or this will just add confusion? Unicode identifiers already can be misused for confusion due to homoglyphs. The problem is not that correctly looking program can be rejected by the compiler, but that the program can work differently from expected because it uses different names that look the same. From mal at egenix.com Mon Nov 20 04:20:13 2017 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 20 Nov 2017 10:20:13 +0100 Subject: [Python-ideas] Ignorable whitespaces in the re.VERBOSE mode In-Reply-To: References: <23058.36540.802392.890565@turnbull.sk.tsukuba.ac.jp> Message-ID: <02808fbe-3bd3-f5f4-8b28-95f0abf271e0@egenix.com> For consistency, we should probably have "whitespace" for re equal to whatever "\s" matches, since this is what the engine itself considers as whitespace (and then also covers the special case where you use the re.ASCII flag). Still, the only practical case I could imagine, where extending the list would indeed make sense, is to have the   character qualify as whitespace for re.VERBOSE, since this can sometimes be introduced via copy&paste from other sources (e.g. web pages showing a regular expression). Due to whitespace being what it is, it's hard to tell whether you've just copied a \u0020 or a \u00a0. The latter can easily render the regular expression non-working with the current interpretation of re.VERBOSE. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Nov 20 2017) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From bunslow at gmail.com Mon Nov 20 03:03:13 2017 From: bunslow at gmail.com (bunslow) Date: Mon, 20 Nov 2017 02:03:13 -0600 Subject: [Python-ideas] Adding a new function "zip_flat" to itertools (Re: Rewriting the "roundrobin" recipe in the itertools documentation) Message-ID: Dear all: thank you for your replies and thoughts, most especially Steve and Terry. I am more-or-less new to contributing to Python, so I wasn't sure that the bug tracker was the best way to start -- I was looking for a sanity check and received exactly what I wanted :) Thanks to the feedback here, the bug tracker issue will be much cleaner from the start. Regarding the meta discussion about my OP: yes it was long winded, detailed, pedantic and (to a certain extent) bombastic, but I was imaging the many possible responses to the suggestion (and suggested replacement) the I felt I should explain where I was coming from. Even though there was a lot of disagreement about how I described the current recipe as not very Pythonic, I'm very glad for all the perspectives and lessons I got from reading the ensuing discussions. Now, having had some time to think, I've come up with further thoughts on the topic. In particular, I was going to create a new bug, and wrote several paragraphs on the topic summarizing this thread, and my thoughts. I'll paste those paragraphs here: """ To summarize, I found the current implementation of the "roundrobin" function difficult to understand, and proposed a much simpler solution that, while producing correct results, isn't quite "correct" in the sense that it glosses over a detail before "manually" correcting the detail at the end, and as such is prone to severe inefficiency in extreme cases. There were a smattering of comments from several people indicating that they liked the simpler recipe better, despite its performance drawbacks. Terry Reedy proposed a slightly rewritten version of the current recipe, which implements the correct algorithm (without glossing over and manually correcting the details). Although I have since changed my perceptions of the original, now understanding how it works, the rewritten version from Terry was clearer enough that I was able to understand *it* where I could not previously understand the original. (My newfound understanding of the original is largely derived from making sense of the rewritten version, which properly clued me in to what cycle and islice actually do. Either way, the current recipe can certainly be improved. Although I now find the original and its rewrite to be algorithmically clean and correct (even if the code and inline comments can be improved), the StackOverflow question ( https://stackoverflow.com/questions/3678869/pythonic-way-to-combine-two-lists-in-an-alternating-fashion/) that originally got me thinking about the problem demonstrates that the algorithmically clean way is *not* obvious at all to people who aren't very familiar with the itertools module -- which is the large majority of people who use Python (even if that's a very small fraction of the people reading this bug). The second from top answer is the answer which references the recipe in the docs, but as my own first post to python-ideas demonstrates, the (large?) majority of python users aren't familiar enough with the itertools module to be able to understand that recipe, and I also believe many people were looking for one or two liners to use in their own code without a function call. Further confusion on the overall topic is the lack of a clear name -- "roundrobin", "alternate", "interleave", "merge", and variations and others. """ Having completed those, I found a roughly duplicate StackOverflow question to the one from my OP: https://stackoverflow.com/questions/243865/how-do-i-merge-two-python-iterators Besides emphasizing my points about not having even a clear name for the topic, a desire for one liners, mass confusion around the issue (especially regarding flattening zip [which terminates on the shortest input, a hidden gotcha], zip_longest [someone else found the same solution as me and others in this op), and all around failure to generate anything even resembling a consensus on the topic, I also found this answer: https://stackoverflow.com/questions/243865/how-do-i-merge-two-python-iterators/40498526#40498526 which proposes a solution that is both more correct and efficient than the zip_longest-with-sentinels, and also noticeably more readable than either the original doc recipe or even Terry's cleaned up replacement of it. Given this variety of problems with the issue, I now think that -- while updating the itertools recipe is certainly better than nothing -- the better thing to do might be to just add a new function to itertools called "zip_flat" which solves this problem. In addition to answering the stack overflow questions with ongoing debate about efficiency, correctness, and pythonicity (pythonicness?), it would also help to greatly clarify the naming issue as well. (Sidenote: whoever came up with "zip" as the name for the builtin was quite creative. It's a remarkably short and descriptive.) What are the sentiments of readers here? If positive, I'll create an issue on BPO about zip_flat (rather than just improving the docs recipe). (Sorry Steve for bringing this back to -ideas! At least this time I'm proposing an addition to the language itself! :) Thanks for your consideration, Bill On Thu, Nov 16, 2017 at 5:06 PM, Steven D'Aprano wrote: > On Thu, Nov 16, 2017 at 02:56:29PM -0500, Terry Reedy wrote: > > > >3) using a while loop in code dealing with iterables, > > > > I agree that this is not necessary, and give a replacement below. > > The OP isn't just saying that its unnecessary in this case, but that its > unPythonic to ever use a while loop in code dealing with iterables. I > disagree with that stronger statement. > > > > >def roundrobin(*iters): > > > "roundrobin('ABC', 'D', 'EF') --> A D E B F C" > > > # Perhaps "flat_zip_nofill" is a better name, or something similar > > > sentinel = object() > > > for tup in it.zip_longest(*iters, fillvalue=sentinel): > > > yield from (x for x in tup if x is not sentinel) > > > > This adds and then deletes grouping and fill values that are not wanted. > > To me, this is an 'algorithm smell'. One of the principles of > > algorithm design is to avoid unnecessary calculations. For an edge case > > such as roundrobin(1000000*'a', ''), the above mostly does unnecessary > work. > > Its a recipe, not a tuned and optimized piece of production code. > > And if you're going to criticise code on the basis of efficiency, then I > would hope you've actually profiled the code first. Because it isn't > clear to me at all that what you call "unnecessary work" is more > expensive than re-writing the recipe using a more complex algorithm with > calls to cycle and islice. > > But I'm not here to nit-pick your recipe over the earlier ones. > > > [...] > > Since I have a competing 'improvement', I would hold off on a PR until > > Raymond Hettinger, the itertools author, comments. > > Raise a doc issue on the tracker, and take the discussion there. I think > that this is too minor an issue to need long argument on the list. > Besides, it's not really on-topic as such -- it isn't about a change to > the language. Its about an implementation change to a recipe in the > docs. > > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon Nov 20 11:08:02 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 21 Nov 2017 03:08:02 +1100 Subject: [Python-ideas] Adding a new function "zip_flat" to itertools (Re: Rewriting the "roundrobin" recipe in the itertools documentation) In-Reply-To: References: Message-ID: <20171120160802.GH19802@ando.pearwood.info> Hi Bill, I don't have time to go through your email in detail and respond to every point you raise, but I'd like to respond to one point you made. On Mon, Nov 20, 2017 at 02:03:13AM -0600, bunslow wrote: > I also found this answer: > > https://stackoverflow.com/questions/243865/how-do-i-merge-two-python-iterators/40498526#40498526 > > which proposes a solution that is both more correct and efficient than the > zip_longest-with-sentinels, and also noticeably more readable than either > the original doc recipe or even Terry's cleaned up replacement of it. Please don't make claims about correctness and efficiency without testing the code first. The second suggestion given there, using deque, is *not* correct as provided, as it fails to work with iterables. It requires the caller to pass only iterators, unlike the existing roundrobin recipe which accepts any iterable. Nor is it more efficient, at least on my machine -- in fact the opposite, it is the worst performing of the four recipes I've tried: - the current recipe from the itertools docs; - your re-write, using zip_longest; - Terry's version; - and the one from stackoverflow. I've attached my test code, in case you want to play around with it. Apologies in advance for any bugs in the test code (its 2 in the morning here and I've had a long day). According to my testing, on my computer using Python 3.5, Terry's code is by far the fastest in all three separate test cases, but that probably shouldn't count since it's buggy (it truncates the results and bails out early under some circumstances). Out of the implementations that don't truncate, the existing recipe is by far the fastest. Terry, if you're reading this, try: list(roundrobin('A', 'B', 'CDE')) Your version truncates the results to A B C instead of A B C D E as the itertools recipe gives. But I digress... the point is, if you're going to make claims about the correctness and efficiency of code, you really ought to actually check the correctness and efficiency first. I may or may not get around to responding to some of your other points over the next few days. -- Steve -------------- next part -------------- import itertools from itertools import * from timeit import Timer # From the itertools recipe def roundrobin(*iterables): "roundrobin('ABC', 'D', 'EF') --> A D E B F C" pending = len(iterables) nexts = cycle(iter(it).__next__ for it in iterables) while pending: try: for next in nexts: yield next() except StopIteration: pending -= 1 nexts = cycle(islice(nexts, pending)) def bunslow(*iters): "roundrobin('ABC', 'D', 'EF') --> A D E B F C" sentinel = object() for tup in itertools.zip_longest(*iters, fillvalue=sentinel): yield from (x for x in tup if x is not sentinel) def terry(*iterables): "roundrobin('ABC', 'D', 'EF') --> A D E B F C" nexts = cycle(iter(it).__next__ for it in iterables) for reduced_len in reversed(range(1, len(iterables))): try: for next in nexts: yield next() except StopIteration: nexts = cycle(islice(nexts, reduced_len)) from collections import deque def stackoverflow(*iterators): queue = deque(iter(it) for it in iterators) while len(queue) > 0: iterator = queue.popleft() try: yield next(iterator) queue.append(iterator) except StopIteration: pass FUNCTIONS = (roundrobin, bunslow, terry, stackoverflow) equal_data = [range(500)]*4 unequal_data = [range(500), range(600), range(700), range(400)] extreme_data = [range(1), range(1), range(1), range(500)] def trial(data_name, number, repeat): print("testing with %s" % data_name) template = 'from __main__ import %s as rr, %s as data' data = globals()[data_name] assert len(data) == 4 and all(type(x) is range for x in data) expected = list(roundrobin(*data)) failed = [] for func in FUNCTIONS: actual = list(func(*data)) if actual != expected: failed.append(func) setup = template % (func.__name__, data_name) T = Timer('list(rr(*data))', setup=setup) t = min(T.repeat(number=number, repeat=repeat)) print(' %-15s %f' % (func.__name__, t)) for f in failed: print(' *** %s returned incorrect results ***' % f.__name__) NUMBER = 30 REPEAT = 5 trial('equal_data', NUMBER, REPEAT) print("*"*50) trial('unequal_data', NUMBER, REPEAT) print("*"*50) trial('extreme_data', NUMBER, REPEAT) From steve at pearwood.info Mon Nov 20 11:18:46 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 21 Nov 2017 03:18:46 +1100 Subject: [Python-ideas] Adding a new function "zip_flat" to itertools (Re: Rewriting the "roundrobin" recipe in the itertools documentation) In-Reply-To: <20171120160802.GH19802@ando.pearwood.info> References: <20171120160802.GH19802@ando.pearwood.info> Message-ID: <20171120161846.GI19802@ando.pearwood.info> Oops, a slight buglet in my test code: [Attachment: rr_tester.py] Change the constant NUMBER = 30 to the largest value you can bear for more reliable timing tests. I lowered the value for a quick test and forgot to increase it again. -- Steve From tjreedy at udel.edu Mon Nov 20 12:15:50 2017 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 20 Nov 2017 12:15:50 -0500 Subject: [Python-ideas] Adding a new function "zip_flat" to itertools (Re: Rewriting the "roundrobin" recipe in the itertools documentation) In-Reply-To: <20171120160802.GH19802@ando.pearwood.info> References: <20171120160802.GH19802@ando.pearwood.info> Message-ID: On 11/20/2017 11:08 AM, Steven D'Aprano wrote: > Please don't make claims about correctness and efficiency without > testing the code first. The second suggestion given there, using deque, > is *not* correct as provided, as it fails to work with iterables. It > requires the caller to pass only iterators, unlike the existing > roundrobin recipe which accepts any iterable. > > Nor is it more efficient, at least on my machine -- in fact the > opposite, it is the worst performing of the four recipes I've tried: > > - the current recipe from the itertools docs; > - your re-write, using zip_longest; > - Terry's version; > - and the one from stackoverflow. > > I've attached my test code, in case you want to play around with it. > Apologies in advance for any bugs in the test code (its 2 in the > morning here and I've had a long day). > > According to my testing, on my computer using Python 3.5, Terry's code > is by far the fastest in all three separate test cases, but that > probably shouldn't count since it's buggy (it truncates the results and > bails out early under some circumstances). Out of the implementations > that don't truncate, the existing recipe is by far the fastest. > > Terry, if you're reading this, try: > > list(roundrobin('A', 'B', 'CDE')) > Your version truncates the results to A B C instead of A B C D E as the > itertools recipe gives. This is due to an off-by-1 error which I corrected 3 hours later in a follow-up post, repeated below. --- Correct off-by-one error. I should have tested with an edge case such as print(list(roundrobin('ABC', ''))) > The following combines 3 statements into one for statement. > > def roundrobin(*iterables): > "roundrobin('ABC', 'D', 'EF') --> A D E B F C" > nexts = cycle(iter(it).__next__ for it in iterables) > for reduced_len in reversed(range(1, len(iterables))): Make that 0 rather than 1 for start value. > try: > for next in nexts: > yield next() > except StopIteration: > nexts = cycle(islice(nexts, reduced_len)) A slightly clearer, slightly less efficient alternative would be def roundrobin(*iterables): "roundrobin('ABC', 'D', 'EF') --> A D E B F C" nexts = cycle(iter(it).__next__ for it in iterables) for current_len in reversed(range(1, len(iterables)+1)): try: for next in nexts: yield next() except StopIteration: nexts = cycle(islice(nexts, current_len - 1)) -- Terry Jan Reedy From mertz at gnosis.cx Mon Nov 20 13:04:31 2017 From: mertz at gnosis.cx (David Mertz) Date: Mon, 20 Nov 2017 10:04:31 -0800 Subject: [Python-ideas] Adding a new function "zip_flat" to itertools (Re: Rewriting the "roundrobin" recipe in the itertools documentation) In-Reply-To: References: <20171120160802.GH19802@ando.pearwood.info> Message-ID: After correcting Terry's off-by-one error, I get this on Python 3.6 (and bumping NUMBER to 1000). In speed, Terry's is either very slightly faster or very slightly slower than the recipe, depending on the data. I think Terry's is more readable that roundrobin(), but still requires a little thought compared to Bunslow's. However, using the builtin name 'next' as a variable name seems like a bad choice to me (it doesn't break in this code, but seems like bad practice, i'd choose some other variable name). 507-code % python rr_tester.py testing with equal_data roundrobin 0.187816 bunslow 0.393162 terry 0.183851 stackoverflow 0.675543 ************************************************** testing with unequal_data roundrobin 0.209959 bunslow 0.555731 terry 0.233015 stackoverflow 0.746880 ************************************************** testing with extreme_data roundrobin 0.053149 bunslow 0.273607 terry 0.051963 stackoverflow 0.158515 508-code % python --version Python 3.6.2 :: Anaconda custom (x86_64) On Mon, Nov 20, 2017 at 9:15 AM, Terry Reedy wrote: > On 11/20/2017 11:08 AM, Steven D'Aprano wrote: > > Please don't make claims about correctness and efficiency without >> testing the code first. The second suggestion given there, using deque, >> is *not* correct as provided, as it fails to work with iterables. It >> requires the caller to pass only iterators, unlike the existing >> roundrobin recipe which accepts any iterable. >> >> Nor is it more efficient, at least on my machine -- in fact the >> opposite, it is the worst performing of the four recipes I've tried: >> >> - the current recipe from the itertools docs; >> - your re-write, using zip_longest; >> - Terry's version; >> - and the one from stackoverflow. >> >> I've attached my test code, in case you want to play around with it. >> Apologies in advance for any bugs in the test code (its 2 in the >> morning here and I've had a long day). >> >> According to my testing, on my computer using Python 3.5, Terry's code >> is by far the fastest in all three separate test cases, but that >> probably shouldn't count since it's buggy (it truncates the results and >> bails out early under some circumstances). Out of the implementations >> that don't truncate, the existing recipe is by far the fastest. >> >> Terry, if you're reading this, try: >> >> list(roundrobin('A', 'B', 'CDE')) >> > > Your version truncates the results to A B C instead of A B C D E as the >> itertools recipe gives. >> > > This is due to an off-by-1 error which I corrected 3 hours later in a > follow-up post, repeated below. > --- > > Correct off-by-one error. I should have tested with an edge case such as > print(list(roundrobin('ABC', ''))) > > > The following combines 3 statements into one for statement. > > > > def roundrobin(*iterables): > > "roundrobin('ABC', 'D', 'EF') --> A D E B F C" > > nexts = cycle(iter(it).__next__ for it in iterables) > > for reduced_len in reversed(range(1, len(iterables))): > > Make that 0 rather than 1 for start value. > > > try: > > for next in nexts: > > yield next() > > except StopIteration: > > nexts = cycle(islice(nexts, reduced_len)) > > A slightly clearer, slightly less efficient alternative would be > > def roundrobin(*iterables): > "roundrobin('ABC', 'D', 'EF') --> A D E B F C" > nexts = cycle(iter(it).__next__ for it in iterables) > for current_len in reversed(range(1, len(iterables)+1)): > try: > for next in nexts: > yield next() > except StopIteration: > nexts = cycle(islice(nexts, current_len - 1)) > > -- > Terry Jan Reedy > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Mon Nov 20 14:50:30 2017 From: barry at python.org (Barry Warsaw) Date: Mon, 20 Nov 2017 14:50:30 -0500 Subject: [Python-ideas] A proliferation of (un-)Pythonically programmatic pragmas In-Reply-To: References: <20171115021457.GM19802@ando.pearwood.info> Message-ID: Chris Barker wrote: > but whatever -- multiple line pragmas are fine, too -- I'm hoping putting > this much crap in your code will be rare :-) For sure. It's not uncommon for you to need two pragmas, e.g. flake8 and coverage, but it's pretty rare to need those *plus* mypy. It does happen though. And while I agree it's better to fix the problem, there are lots of reasons why you can't. Multiple-version support in libraries is one common reason (e.g. this code path will never be taken in Python 2 and the alternative will never be taken in Python 3), bugs in the tools (false positives), etc. Cheers, -Barry From Nikolaus at rath.org Mon Nov 20 16:04:00 2017 From: Nikolaus at rath.org (Nikolaus Rath) Date: Mon, 20 Nov 2017 21:04:00 +0000 Subject: [Python-ideas] Allow additional separator character in variables In-Reply-To: (Bruce Leban's message of "Sat, 18 Nov 2017 23:11:58 -0800") References: Message-ID: <87o9nwek73.fsf@vostro.rath.org> On Nov 18 2017, Bruce Leban wrote: > And because spaces between words is mostly not valid syntax currently, this > change would be easier to introduce than breaking every single program out > there by re-purposing hyphen-minus. But I'm not seriously proposing this > because I think the modest benefits are outweighed by the many problems it > would introduce. Luckily, there is a compromise: use backticks to quote identifiers: `test mode` = True if `test mode`: `display message`("just a test") I'm not seriously suggesting that, but I still wonder what people think about it. I sort of like it, actually. The `(" part is pretty ugly (which is why I included it in the example), but there's no syntax that can completely avoid ugly corner cases. I think in most cases the context would also make it easy to distinguish single quotes and backticks even when they're typographically similar. Cheers, -Nikolaus -- GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F ?Time flies like an arrow, fruit flies like a Banana.? From steve at pearwood.info Mon Nov 20 16:57:57 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 21 Nov 2017 08:57:57 +1100 Subject: [Python-ideas] Adding a new function "zip_flat" to itertools (Re: Rewriting the "roundrobin" recipe in the itertools documentation) In-Reply-To: References: <20171120160802.GH19802@ando.pearwood.info> Message-ID: <20171120215757.GJ19802@ando.pearwood.info> On Mon, Nov 20, 2017 at 12:15:50PM -0500, Terry Reedy wrote: > >Your version truncates the results to A B C instead of A B C D E as the > >itertools recipe gives. > > This is due to an off-by-1 error which I corrected 3 hours later in a > follow-up post, repeated below. Ah, I missed that, thanks. > > def roundrobin(*iterables): > > "roundrobin('ABC', 'D', 'EF') --> A D E B F C" > > nexts = cycle(iter(it).__next__ for it in iterables) > > for reduced_len in reversed(range(1, len(iterables))): > > Make that 0 rather than 1 for start value. Is there a reason for calling reversed() instead of reversing the order of range in the first place? range(len(iterables)-1, -1, -1) -- Steve From chris.barker at noaa.gov Mon Nov 20 16:59:29 2017 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 20 Nov 2017 13:59:29 -0800 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <5A087DE7.4050306@brenbarn.net> <8d5574dc-b614-ccfe-3ce8-f1ba86508179@python.org> Message-ID: On Fri, Nov 17, 2017 at 2:14 AM, Steve Barnes wrote: > > >>> On Wed, Nov 15, 2017 at 11:07 AM, Steve Dower > >>>> My preferred solution for this is to rename "py.exe" to "python.exe" > ... > > Note that renaming py.exe to python.exe would have several problems: > > ... I don't understand any of this enough to have an opinion, so while I'd like to see py.exe be renamed python.exe, let's not let "the perfect be the enemy of good enough". So, if someone can make the case that they can restructure the whole py.exe / python.exe thing nicely in a way that will work, AND write the code to do it fairly quickly, then great! Otherwise, let's at least get a python3.exe into 3.7 -- and ideally into updates to 3.5, 3.6, and (python2.exe in this case, 2.7) And maybe make "add to PATH" be the default in the installer. Those are quick and simple to do, result in little disruption, and make the whole cross-platform thing more manageable. BTW -- does pip install a "pip3" on Windows? -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericfahlgren at gmail.com Mon Nov 20 17:30:51 2017 From: ericfahlgren at gmail.com (Eric Fahlgren) Date: Mon, 20 Nov 2017 14:30:51 -0800 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <5A087DE7.4050306@brenbarn.net> <8d5574dc-b614-ccfe-3ce8-f1ba86508179@python.org> Message-ID: ? On Mon, Nov 20, 2017 at 1:59 PM, Chris Barker wrote: > BTW -- does pip install a "pip3" on Windows? > ? In 3.6 it does: > find t:/Python36/ -iname 'pip*exe' t:/Python36/Scripts/pip.exe t:/Python36/Scripts/pip3.6.exe t:/Python36/Scripts/pip3.exe ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From breamoreboy at gmail.com Mon Nov 20 17:35:24 2017 From: breamoreboy at gmail.com (Mark Lawrence) Date: Mon, 20 Nov 2017 22:35:24 +0000 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <5A087DE7.4050306@brenbarn.net> <8d5574dc-b614-ccfe-3ce8-f1ba86508179@python.org> Message-ID: On 20/11/17 21:59, Chris Barker wrote: > > BTW -- does pip install a "pip3" on Windows? > > -CHB > > -- > > Christopher Barker, Ph.D. > Oceanographer > pip.exe, pip3.exe and pip3.6.exe are all in C:\Program Files\Python36\Scripts on my Windows 10 setup installed for all users. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From tjreedy at udel.edu Mon Nov 20 18:00:32 2017 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 20 Nov 2017 18:00:32 -0500 Subject: [Python-ideas] Adding a new function "zip_flat" to itertools (Re: Rewriting the "roundrobin" recipe in the itertools documentation) In-Reply-To: <20171120215757.GJ19802@ando.pearwood.info> References: <20171120160802.GH19802@ando.pearwood.info> <20171120215757.GJ19802@ando.pearwood.info> Message-ID: On 11/20/2017 4:57 PM, Steven D'Aprano wrote: > On Mon, Nov 20, 2017 at 12:15:50PM -0500, Terry Reedy wrote: > >>> Your version truncates the results to A B C instead of A B C D E as the >>> itertools recipe gives. >> >> This is due to an off-by-1 error which I corrected 3 hours later in a >> follow-up post, repeated below. > > Ah, I missed that, thanks. > >>> def roundrobin(*iterables): >>> "roundrobin('ABC', 'D', 'EF') --> A D E B F C" >>> nexts = cycle(iter(it).__next__ for it in iterables) >>> for reduced_len in reversed(range(1, len(iterables))): >> >> Make that 0 rather than 1 for start value. > > Is there a reason for calling reversed() instead of reversing the order > of range in the first place? > > range(len(iterables)-1, -1, -1) Readability. Accurately and automatically reversing ranges was one of the use cases motivating the addition of 'reversed'. Ranges have a __reversed__ method which returns a iterator for a reversed range. >>> reversed(range(0, n)) >>> list(reversed(range(0, n))) == list(iter(range(n-1, -1, -1))) True -- Terry Jan Reedy From p.f.moore at gmail.com Mon Nov 20 18:24:15 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 20 Nov 2017 23:24:15 +0000 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <5A087DE7.4050306@brenbarn.net> <8d5574dc-b614-ccfe-3ce8-f1ba86508179@python.org> Message-ID: On 20 November 2017 at 21:59, Chris Barker wrote: > I don't understand any of this enough to have an opinion, so while I'd like > to see py.exe be renamed python.exe, let's not let "the perfect be the enemy > of good enough". So, if someone can make the case that they can restructure > the whole py.exe / python.exe thing nicely in a way that will work, AND > write the code to do it fairly quickly, then great! I'm happy enough to modify the py.exe code to base its behaviour on the name it's called with, as Steve suggested. I'm not sure if that would require a PEP, but I'm willing to do one if it's felt that there is a need. It should be possible to get that change in for 3.7. I don't know what would be the best approach for adding copies/links of the launcher under other names, though. And I don't have any experience with the tools we use to build the installer, so that would be for someone else. That part probably *would* need a PEP, and it may well be controversial enough that getting agreement in time for 3.7 (i.e. in the next 2 months or so) will be challenging. I think the launcher change is worthwhile in its own right. Even if we don't install any aliases by default, users will be able to manually add them. (And someone could write a 3rd party tool to manage creation of such aliases, making that capability available to users who don't have the necessary skills themselves). > Otherwise, let's at least get a python3.exe into 3.7 -- and ideally into > updates to 3.5, 3.6, and (python2.exe in this case, 2.7) It's not going to be acceptable for 3.5, as that is now in security fix only mode. I'm not certain it's even acceptable for bugfix-only releases. It's a backward incompatible change (3.6.3 has no python3.exe but python 3.6.4 does) but maybe an acceptable one - we'd need feedback from the 3.6 and 2.7 release managers on that. Also, I'm assuming here you mean "create a copy of python.exe called python3.exe". If we do that, we risk making it harder to later switch to making "python3.exe" a link to the launcher - for the same reason that making "python.exe" be an alternative name for the launcher is problematic right now. > And maybe make "add to PATH" be the default in the installer. I'm not willing to contradict Steve on this one. There are too many not-uncommon cases that we could mess up badly here. It's the right choice for "make new users' experience as easy as possible", but if we do this at the cost of making the experience of people upgrading to 3.7 (possibly by installing 3.7 to gradually switch over, or starting their migration from 2.7 with 3.7) unpleasant, then we risk getting bad publicity for the 3.7 release. The history of how we added Python to PATH across various versions is messy and inconsistent. Add to that other distributions (Anaconda, ActiveState) making different choices and adding yet more inconsistency, means that anyone who *isn't* a brand new user probably already has a tweaked, and likely fragile, setup. We're not doing them any favours by adding another new behaviour. (And enabling "Add to PATH" in 3.7 *would* be a new behaviour - I don't think we've enabled add to path by default in any version since we switched to per-user installs as the default). We need *another* solution here. Not just variations on the existing mess. That's why I like Steve's suggestion of making the launcher into the canonical entry point. It's not easy, but at least it stands a chance of breaking out of the cycle we're currently in, of switching back and forth between "add to PATH" and "don't add to PATH". > Those are quick and simple to do, result in little disruption, and make the > whole cross-platform thing more manageable. They aren't that simple. I've already discussed "add to PATH". And if we add python3.exe, we have to consider questions like will venv be modified to include it in virtual environments? Will virtualenv? If not, will "python3" run the system Python rather than the current virtualenv? That's just as much a problem for the option of having python3 be an alias for the launcher, of course - but my point here is you have to think about questions like this even for your "simple" approach. I just don't think there *is* a quick and simple solution here. So better not to rush into a solution that isn't fully thought through, IMO. "Although never is often better than *right* now" is the relevant Zen here, not "Now is better than never" (nobody's suggesting we *never* fix this). > BTW -- does pip install a "pip3" on Windows? No. Just "pip.exe". Paul From p.f.moore at gmail.com Mon Nov 20 18:27:37 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 20 Nov 2017 23:27:37 +0000 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <5A087DE7.4050306@brenbarn.net> <8d5574dc-b614-ccfe-3ce8-f1ba86508179@python.org> Message-ID: >> BTW -- does pip install a "pip3" on Windows? > > No. Just "pip.exe". Looks like I should have checked. As others pointed out, it does (pip3.exe and pip3.6.exe). Apologies. Paul From chris.barker at noaa.gov Mon Nov 20 19:32:13 2017 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 20 Nov 2017 16:32:13 -0800 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <5A087DE7.4050306@brenbarn.net> <8d5574dc-b614-ccfe-3ce8-f1ba86508179@python.org> Message-ID: On Mon, Nov 20, 2017 at 3:24 PM, Paul Moore wrote: > On 20 November 2017 at 21:59, Chris Barker wrote: > > I don't understand any of this enough to have an opinion, so while I'd > like > > to see py.exe be renamed python.exe, let's not let "the perfect be the > enemy > > of good enough". So, if someone can make the case that they can > restructure > > the whole py.exe / python.exe thing nicely in a way that will work, AND > > write the code to do it fairly quickly, then great! > > I'm happy enough to modify the py.exe code to base its behaviour on > the name it's called with, as Steve suggested. I'm not sure if that > would require a PEP, but I'm willing to do one if it's felt that there > is a need. It should be possible to get that change in for 3.7. very cool -- thanks! > I > don't know what would be the best approach for adding copies/links of > the launcher under other names, though. this is the thing -- I wonder if py.exe has been a success at all :-) I honestly don't know - it's a cool idea, and I liked the idea of replacing the #! line and all that, but the truth is that people simple don't write scripts on Windows and expect them to "just work" the same way they do on *nix systems. And if you do distutils/setuptools "scripts" or "entry points" or whatever we are calling them, right, then you can write utility scripts and install them, and use them.... So ti comes down to: does anyone USE py.exe??? The fact that it's unique to Windows makes it simply less discoverable, and the fact that Windows users aren't used to the whole #! line thing, means that as useful as it might be, it's not getting actual use. In fact, I happened to be paying a lot of attention to this conversion because at the very same moment, a group of us are developing a python curriculum, and working on the "how to run a python file" part. We are producing platform specific instructions, but decided NOT to include the py.exe option for Windows users .. just more confusing things among a lot of confusing things! So who does teach people about it??? Anyway, this whole thread has been about making the experience more platform-independent -- ideally users should see docs without Windows-specifc advice, and have it "just work" -- having a python3.exe would make that more likely, so let's do that. If we could py.exe enhanced and then use it instead of python.ex,m that would be great. Though I at least still have no idea of that's possible. But I don't see why we couldn't make a pyton3.exe NOW (in py3.7) that is a copy (Or link, still don't get linking on Windows...) of python.exe, that would help the situation, and if at some point the in the future, python.exe, python3.exe and py.exe all become the same thing, great! > I think the launcher change is worthwhile in its own right. Even if we > don't install any aliases by default, users will be able to manually > add them. sure -- but no one would -- or not really no one, but it would become maybe even a worse hassle of the same thing not working everywhere. we need th=e basic advice given out to *nix users to work: if you type "python" at the command line, and get python2, then you can type "python3" to get python 3. simple, straightforward, and it CAN help. > > Otherwise, let's at least get a python3.exe into 3.7 -- and ideally into > > updates to 3.5, 3.6, and (python2.exe in this case, 2.7) > > It's not going to be acceptable for 3.5, as that is now in security > fix only mode. I'm not certain it's even acceptable for bugfix-only > releases. It's a backward incompatible change (3.6.3 has no > python3.exe but python 3.6.4 does) but maybe an acceptable one - we'd > need feedback from the 3.6 and 2.7 release managers on that. > Fine -- we can leave that for future discussion.... > Also, I'm assuming here you mean "create a copy of python.exe called > python3.exe". If we do that, we risk making it harder to later switch > to making "python3.exe" a link to the launcher - for the same reason > that making "python.exe" be an alternative name for the launcher is > problematic right now. > I'm lost -- how does it make it any harder, rather than exactly the same ??? (except two changes, rather than one...but they are the same change) > And maybe make "add to PATH" be the default in the installer. > > I'm not willing to contradict Steve on this one. I didn't know he'd made a clear statement about it. But if people that understand these things say it's a bad idea, then it's a bad idea. But we SHOULD make sure we are considering not just what could go wrong, or how often it could go wrong, but who it is going to go wrong for... i.e. making it "do the right thing" for newbies is much more important that making sure it doesn't "do the wrong thing" for more experienced folks that can uncheck a box, or clean up their PATH after the fact.... There are too many > not-uncommon cases that we could mess up badly here. It's the right > choice for "make new users' experience as easy as possible", but if we > do this at the cost of making the experience of people upgrading to > 3.7 (possibly by installing 3.7 to gradually switch over, then you uncheck the box -- and if someone is installing 3.7 to test it out, they SHOULD be a experience enough to uncheck a box... > or starting > their migration from 2.7 with 3.7) unpleasant, why would that be a problem???? Though we may need a "python2.exe" executable for that to work well :-( > The history of how we added Python to PATH across various versions is > messy and inconsistent. Add to that other distributions (Anaconda, > ActiveState) making different choices and adding yet more > inconsistency, means that anyone who *isn't* a brand new user probably > already has a tweaked, and likely fragile, setup. We're not doing them > any favours by adding another new behaviour. I'm not trying to do them any favors -- I'm trying to do the favors for the newbies.. and YES, at the expense of people with "tweaked, and likely fragile, setup." > We need *another* solution here. Not just variations on the existing > mess. That's why I like Steve's suggestion of making the launcher into > the canonical entry point. only if it's named "python" (and python3) -- what we DON'T need is a different way to do it on Windows -- that's what we tried, and I thin it's failed. It's not easy, but at least it stands a > chance of breaking out of the cycle we're currently in, of switching > back and forth between "add to PATH" and "don't add to PATH". > It also seems completely orthogonal to the "add to PATH" question -- having this nifty new launcher setup, but not on a newbies PATH where they can use it, isn't going to help at all. > > Those are quick and simple to do, result in little disruption, and make > the > > whole cross-platform thing more manageable. > > They aren't that simple. I've already discussed "add to PATH". And if > we add python3.exe, we have to consider questions like will venv be > modified to include it in virtual environments? Will virtualenv? what do those systems do on *nix? do exactly the same thing on Windows. Done. Also -- I don't much care -- I want newbies that aren't using virtual environments to have a better expeience -- virtualenv pretty much solves this issue IF they are being used. > "Although never is often better than *right* now" is the relevant > Zen here, not "Now is better than never" (nobody's suggesting we > *never* fix this). if we take long enough, everyone will be using python3 anyway :-) -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Mon Nov 20 19:56:09 2017 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 20 Nov 2017 19:56:09 -0500 Subject: [Python-ideas] Rewriting the "roundrobin" recipe in the itertools documentation In-Reply-To: References: Message-ID: On 11/16/2017 5:57 PM, Terry Reedy wrote: > On 11/16/2017 2:56 PM, Terry Reedy wrote: > > Correct off-by-one error.? I should have tested with an edge case such as > print(list(roundrobin('ABC', ''))) > >> The following combines 3 statements into one for statement. >> >> def roundrobin(*iterables): >> ???? "roundrobin('ABC', 'D', 'EF') --> A D E B F C" >> ???? nexts = cycle(iter(it).__next__ for it in iterables) >> ???? for reduced_len in reversed(range(1, len(iterables))): > > Make that 0 rather than 1 for start value. > >> ???????? try: >> ???????????? for next in nexts: >> ???????????????? yield next() >> ???????? except StopIteration: >> ???????????? nexts = cycle(islice(nexts, reduced_len)) > > A slightly clearer, slightly less efficient alternative would be > > def roundrobin(*iterables): > ??? "roundrobin('ABC', 'D', 'EF') --> A D E B F C" > ??? nexts = cycle(iter(it).__next__ for it in iterables) > ??? for current_len in reversed(range(1, len(iterables)+1)): > ??????? try: > ??????????? for next in nexts: > ??????????????? yield next() > ??????? except StopIteration: > ??????????? nexts = cycle(islice(nexts, current_len - 1)) I submitted the 'current_len' version as https://bugs.python.org/issue32099 "Use range in itertools roundrobin recipe" -- Terry Jan Reedy From mistersheik at gmail.com Mon Nov 20 20:09:44 2017 From: mistersheik at gmail.com (Neil Girdhar) Date: Tue, 21 Nov 2017 01:09:44 +0000 Subject: [Python-ideas] Consider (one day) adding an inheritance order class precedence mechanism In-Reply-To: References: <20171116001841.GN19802@ando.pearwood.info> <5A0D2BC7.3040209@canterbury.ac.nz> <20171117055252.GS19802@ando.pearwood.info> Message-ID: On Sat, Nov 18, 2017 at 9:29 PM Nick Coghlan wrote: > On 19 November 2017 at 06:56, Neil Girdhar wrote: > > Would you mind explaining why it's necessary for C3 to complain? > > > > In: > > > > S < C > > B < S, E > > R < E, C > > Z < B, R > > > > If Z is told to have MRO: > > > > (Z, B, S, R, E, C) > > > > then there are no conflicts with any base classes. > > I don't actually know what C3 allows in principle, I only know that > CPython's resolver still complains in practice: > > >>> class C: pass > ... > >>> class S(C): pass > ... > >>> class E: pass > ... > >>> class B(S, E): pass > ... > >>> class R(E, C): pass > ... > >>> class Z(B, S, R, E, C): pass > ... > Traceback (most recent call last): > File "", line 1, in > TypeError: Cannot create a consistent method resolution order > (MRO) for bases C, E > > I think the problem is that the resolver isn't looking at the declared > bases of "B", and "R", it's looking at their full MRO: > > >>> B.__mro__ > (, , , > , ) > >>> R.__mro__ > (, , , > ) > > Changing the heuristics used to generate B's MRO such that "C" and "E" > appeared in the opposite order wouldn't really help, since that would > just flip the problematic case to be the "R(C, E)" declaration. > Sorry, I wrote back too quickly. I meant also to change B's requested MRO to be: (B, S, E, C) It works with that change. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bunslow at gmail.com Mon Nov 20 20:19:38 2017 From: bunslow at gmail.com (bunslow) Date: Mon, 20 Nov 2017 19:19:38 -0600 Subject: [Python-ideas] Adding a new function "zip_flat" to itertools (Re: Rewriting the "roundrobin" recipe in the itertools documentation) In-Reply-To: <20171120160802.GH19802@ando.pearwood.info> References: <20171120160802.GH19802@ando.pearwood.info> Message-ID: (Regarding the stackoverflow link, he does have a version that includes the trivial fix to apply iter() to args, at the bottom of the post.) I think that the variety of solutions with a variety of merits is indicative that it would be useful to have a "zip_flat" in the itertools module, but I guess support for that isn't very strong here. I see that Terry opened a bug to clean up the current recipe in the docs, which is better than nothing, but I think it's the absolute bare minimum improvement. I'm prepared to drop the subject unless there's further discussion in favor of it. Thanks again to all for the replies. Bill On Mon, Nov 20, 2017 at 10:08 AM, Steven D'Aprano wrote: > Hi Bill, > > I don't have time to go through your email in detail and respond to > every point you raise, but I'd like to respond to one point you made. > > On Mon, Nov 20, 2017 at 02:03:13AM -0600, bunslow wrote: > > > I also found this answer: > > > > https://stackoverflow.com/questions/243865/how-do-i- > merge-two-python-iterators/40498526#40498526 > > > > which proposes a solution that is both more correct and efficient than > the > > zip_longest-with-sentinels, and also noticeably more readable than either > > the original doc recipe or even Terry's cleaned up replacement of it. > > Please don't make claims about correctness and efficiency without > testing the code first. The second suggestion given there, using deque, > is *not* correct as provided, as it fails to work with iterables. It > requires the caller to pass only iterators, unlike the existing > roundrobin recipe which accepts any iterable. > > > Nor is it more efficient, at least on my machine -- in fact the > opposite, it is the worst performing of the four recipes I've tried: > > - the current recipe from the itertools docs; > - your re-write, using zip_longest; > - Terry's version; > - and the one from stackoverflow. > > I've attached my test code, in case you want to play around with it. > Apologies in advance for any bugs in the test code (its 2 in the > morning here and I've had a long day). > > According to my testing, on my computer using Python 3.5, Terry's code > is by far the fastest in all three separate test cases, but that > probably shouldn't count since it's buggy (it truncates the results and > bails out early under some circumstances). Out of the implementations > that don't truncate, the existing recipe is by far the fastest. > > Terry, if you're reading this, try: > > list(roundrobin('A', 'B', 'CDE')) > > Your version truncates the results to A B C instead of A B C D E as the > itertools recipe gives. > > But I digress... the point is, if you're going to make claims about the > correctness and efficiency of code, you really ought to actually check > the correctness and efficiency first. > > I may or may not get around to responding to some of your other points > over the next few days. > > > -- > Steve > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From turnbull.stephen.fw at u.tsukuba.ac.jp Mon Nov 20 20:51:30 2017 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Tue, 21 Nov 2017 10:51:30 +0900 Subject: [Python-ideas] Allow additional separator character in variables In-Reply-To: References: Message-ID: <23059.34466.1253.737550@turnbull.sk.tsukuba.ac.jp> Mikhail V writes: > On Mon, Nov 20, 2017 at 5:46 AM, Guido van Rossum wrote: > > Please kill this thread. > > So the idea is too bad? No, the idea is *not* bad, it's just not for Python. As has been true for every one of your ideas for language tweaks that I can recall. There are *millions* of Python programmers by now. There are more lines of Python being written and read in a day than you could write or read in your lifetime. It's just not practical to *change* the meaning of valid lexical constructs this way, and the rules you want could easily have edge cases that confuse a lot of people. We have a lot of experience with such edge cases, both in Python ("else" clauses on loops, and Python 3 itself, come to mind) and out. We don't like them, as a rule, and introduce them only only when they allow a better expression of something that is quite awkward without them, and preferably only when they express new semantics (ie, something previously impossible). If it were just one idea, I'd say "suck it up, Mikhail, and get with the programming language". But your ideas are consistently superficially plausible, taking a few seconds of reflection to see that, yes, they could be done, but they are not going to be accepted in mainline Python. The problem with them is that you propose them for Python, not the specific ideas themselves. The solution is as proposed earlier: create your own language. It shouldn't be excessively hard to write a preprocessor for "mvlang" targeting Python. It has historical precedent: that's how Stroustrup originally implemented C++. It allows smooth interchange of programs with people who know Python, no matter how much you add or change. If, having elaborated all your ideas into this new language, you find yourself unwilling to write in Python, then it's time to publish your language, because other people may feel the same level of attraction to it. But ... it *will* be a different language, not Python. Regards, Steve (not speaking for any other Steves, Stevens, or Stephens) -- Associate Professor Division of Policy and Planning Science http://turnbull/sk.tsukuba.ac.jp/ Faculty of Systems and Information Email: turnbull at sk.tsukuba.ac.jp University of Tsukuba Tel: 029-853-5175 Tennodai 1-1-1, Tsukuba 305-8573 JAPAN From bunslow at gmail.com Mon Nov 20 21:13:04 2017 From: bunslow at gmail.com (bunslow) Date: Mon, 20 Nov 2017 20:13:04 -0600 Subject: [Python-ideas] Adding a thin wrapper class around the functions in stdlib.heapq Message-ID: Nothing so bombastic this time. The heapq functions are basically all named "heapsomething", and basically all take a "heap" for their first argument, with supplementary args coming after. It's a textbook example of the (hypothetical) Object Oriented Manifesto? where defining a class increases type safety and programmers' conceptual clarity. There're practically no drawbacks, and the code to be added would be very simple. Updating the tests and docs would probably be harder. In pure Python, such a class would look like this: class Heap(list): def __init__(self, iterable=None): if iterable: super().__init__(iterable) else: super().__init__() self.heapify() push = heapq.heappush pop = heapq.heappop pushpop = heapq.heappushpop replace = heapq.heapreplace heapify = heapq.heapify # This could be a simple wrapper as well, but I had the following thoughts anyways, so here they are def nsmallest(self, n, key=None): # heapq.nsmallest makes a *max* heap of the first n elements, # while we know that self is already a min heap, so we can # make the max heap construction faster self[:n] = reversed(self[:n]) return heapq.nsmallest(n, self, key) # do we define nlargest on a minheap?? Wrapping around the C builtin functions (which aren't descriptors) would be a bit harder, but not much so: from functools import partial class Heap(list): def __init__(self, iterable=None): if iterable: super().__init__(iterable) else: super().__init__() self.heapify = partial(heapq.heapify, self) self.push = partial(heapq.heappush, self) ... self.heapify() Thoughts? -------------- next part -------------- An HTML attachment was scrubbed... URL: From turnbull.stephen.fw at u.tsukuba.ac.jp Mon Nov 20 21:20:09 2017 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Tue, 21 Nov 2017 11:20:09 +0900 Subject: [Python-ideas] Ignorable whitespaces in the re.VERBOSE mode In-Reply-To: References: <23058.36540.802392.890565@turnbull.sk.tsukuba.ac.jp> Message-ID: <23059.36185.852770.297907@turnbull.sk.tsukuba.ac.jp> Serhiy Storchaka writes: > I agree. But if there is a special part of the Unicode standard for > Pattern White Spaces which includes non-ASCII characters, perhaps there > is a need in them. I asked for the case if Python developers with very > different cultures have need in additional whitespaces in regular > expressions, but I don't know. Seems nobody has claimed their need. I doubt that Japanese would want it. I do use \N{IDEOGRAPHIC SPACE} a bit as a *target* of regular expressions, but I would never want it as non-syntactic in re.VERBOSE. (Of course, I'm not a native Japanese, but I have never heard a Japanese developer wish for use of that character in any programming language, outside of literal strings.) > In particularly I don't know how helpful would be supporting > right-to-left and left-to-right marks in verbose regular expressions That's a good question. Interpretation and display of R2L in programming constructs came up briefly in the discussions about BIDI on the emacs-devel list. I'll ask Eli Zaretskii, who implemented it for Emacs. Steve -- Associate Professor Division of Policy and Planning Science http://turnbull/sk.tsukuba.ac.jp/ Faculty of Systems and Information Email: turnbull at sk.tsukuba.ac.jp University of Tsukuba Tel: 029-853-5175 Tennodai 1-1-1, Tsukuba 305-8573 JAPAN From ncoghlan at gmail.com Mon Nov 20 21:34:10 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 21 Nov 2017 12:34:10 +1000 Subject: [Python-ideas] Consider (one day) adding an inheritance order class precedence mechanism In-Reply-To: References: <20171116001841.GN19802@ando.pearwood.info> <5A0D2BC7.3040209@canterbury.ac.nz> <20171117055252.GS19802@ando.pearwood.info> Message-ID: On 21 November 2017 at 11:09, Neil Girdhar wrote: > > On Sat, Nov 18, 2017 at 9:29 PM Nick Coghlan wrote: >> >> >>> class C: pass >> ... >> >>> class S(C): pass >> ... >> >>> class E: pass >> ... >> >>> class B(S, E): pass >> ... >> >>> class R(E, C): pass >> ... >> >>> class Z(B, S, R, E, C): pass >> ... >> Traceback (most recent call last): >> File "", line 1, in >> TypeError: Cannot create a consistent method resolution order >> (MRO) for bases C, E > Sorry, I wrote back too quickly. I meant also to change B's requested MRO > to be: > > (B, S, E, C) > > It works with that change. Right, but once you do that, then the existing resolver is already able to handle things: >>> class C: pass ... >>> class S(C): pass ... >>> class E: pass ... >>> class B(S, E, C): pass ... >>> class R(E, C): pass ... >>> class Z(B, R): pass ... >>> If you wanted to pick up cases where two classes generate inconsistent MROs that will prevent mutual subclasses (like "class B(S, E)" vs "class R(E, C)"), that feels like a job for a type checker, since it isn't obvious whether it's the definition of B or the definition of R that should be changed to reorder their MRO. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From skreft at gmail.com Mon Nov 20 21:38:18 2017 From: skreft at gmail.com (Sebastian Kreft) Date: Tue, 21 Nov 2017 03:38:18 +0100 Subject: [Python-ideas] Adding a thin wrapper class around the functions in stdlib.heapq In-Reply-To: References: Message-ID: This has been brought up multiple times. Last time was on this thread https://mail.python.org/pipermail/python-ideas/2016-October/043024.html. On Tue, Nov 21, 2017 at 3:13 AM, bunslow wrote: > Nothing so bombastic this time. The heapq functions are basically all > named "heapsomething", and basically all take a "heap" for their first > argument, with supplementary args coming after. It's a textbook example of > the (hypothetical) Object Oriented Manifesto? where defining a class > increases type safety and programmers' conceptual clarity. There're > practically no drawbacks, and the code to be added would be very simple. > Updating the tests and docs would probably be harder. > > In pure Python, such a class would look like this: > > class Heap(list): > > def __init__(self, iterable=None): > if iterable: > super().__init__(iterable) > else: > super().__init__() > > self.heapify() > > push = heapq.heappush > pop = heapq.heappop > pushpop = heapq.heappushpop > replace = heapq.heapreplace > heapify = heapq.heapify > > # This could be a simple wrapper as well, but I had the following > thoughts anyways, so here they are > def nsmallest(self, n, key=None): > # heapq.nsmallest makes a *max* heap of the first n elements, > # while we know that self is already a min heap, so we can > # make the max heap construction faster > self[:n] = reversed(self[:n]) > return heapq.nsmallest(n, self, key) > > # do we define nlargest on a minheap?? > > > Wrapping around the C builtin functions (which aren't descriptors) would > be a bit harder, but not much so: > > from functools import partial > > class Heap(list): > def __init__(self, iterable=None): > if iterable: > super().__init__(iterable) > else: > super().__init__() > > self.heapify = partial(heapq.heapify, self) > self.push = partial(heapq.heappush, self) > ... > > self.heapify() > > > Thoughts? > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- Sebastian Kreft -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Nov 20 21:41:18 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 21 Nov 2017 12:41:18 +1000 Subject: [Python-ideas] Consider (one day) adding an inheritance order class precedence mechanism In-Reply-To: References: <20171116001841.GN19802@ando.pearwood.info> <5A0D2BC7.3040209@canterbury.ac.nz> <20171117055252.GS19802@ando.pearwood.info> Message-ID: On 21 November 2017 at 12:34, Nick Coghlan wrote: > Right, but once you do that, then the existing resolver is already > able to handle things: > > >>> class C: pass > ... > >>> class S(C): pass > ... > >>> class E: pass > ... > >>> class B(S, E, C): pass > ... > >>> class R(E, C): pass > ... > >>> class Z(B, R): pass > ... > >>> > > If you wanted to pick up cases where two classes generate inconsistent > MROs that will prevent mutual subclasses (like "class B(S, E)" vs > "class R(E, C)"), that feels like a job for a type checker, since it > isn't obvious whether it's the definition of B or the definition of R > that should be changed to reorder their MRO. I do wonder if we might be able to make the error message here less cryptic by mentioning which *listed* base classes brought in the conflicting MRO entries. Consider the current: >>> class Z(B, R): pass ... Traceback (most recent call last): File "", line 1, in TypeError: Cannot create a consistent method resolution order (MRO) for bases C, E vs something like: TypeError: Cannot create a consistent method resolution order (MRO) for bases C, E (inherited through B, R) (Note: I haven't actually checked how practical it would be to implement something like that) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From mistersheik at gmail.com Mon Nov 20 21:42:47 2017 From: mistersheik at gmail.com (Neil Girdhar) Date: Tue, 21 Nov 2017 02:42:47 +0000 Subject: [Python-ideas] Consider (one day) adding an inheritance order class precedence mechanism In-Reply-To: References: <20171116001841.GN19802@ando.pearwood.info> <5A0D2BC7.3040209@canterbury.ac.nz> <20171117055252.GS19802@ando.pearwood.info> Message-ID: On Mon, Nov 20, 2017 at 9:34 PM Nick Coghlan wrote: > On 21 November 2017 at 11:09, Neil Girdhar wrote: > > > > On Sat, Nov 18, 2017 at 9:29 PM Nick Coghlan wrote: > >> > >> >>> class C: pass > >> ... > >> >>> class S(C): pass > >> ... > >> >>> class E: pass > >> ... > >> >>> class B(S, E): pass > >> ... > >> >>> class R(E, C): pass > >> ... > >> >>> class Z(B, S, R, E, C): pass > >> ... > >> Traceback (most recent call last): > >> File "", line 1, in > >> TypeError: Cannot create a consistent method resolution order > >> (MRO) for bases C, E > > > Sorry, I wrote back too quickly. I meant also to change B's requested > MRO > > to be: > > > > (B, S, E, C) > > > > It works with that change. > > Right, but once you do that, then the existing resolver is already > able to handle things: > > >>> class C: pass > ... > >>> class S(C): pass > ... > >>> class E: pass > ... > >>> class B(S, E, C): pass > ... > >>> class R(E, C): pass > ... > >>> class Z(B, R): pass > ... > >>> > Sure, but like I mentioned, that's really ugly because B doesn't care about C and shouldn't have to mention it as a base class. This solution can quickly spiral out of control so that changes in the inheritance graph require you to hunt down extraneous base classes. > > If you wanted to pick up cases where two classes generate inconsistent > MROs that will prevent mutual subclasses (like "class B(S, E)" vs > "class R(E, C)"), yes, exactly. > that feels like a job for a type checker, since it > isn't obvious whether it's the definition of B or the definition of R > that should be changed to reorder their MRO. > I don't know what the "type checker" means here. I figured that the easiest user specification would be precedence relationships between classes that could be applied (in the way I described). And I figured that that could be processed for given classes when their MRO is generated. The proposal of having a custom MRO generator would also be able to solve this problem. > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bunslow at gmail.com Mon Nov 20 21:46:34 2017 From: bunslow at gmail.com (bunslow) Date: Mon, 20 Nov 2017 20:46:34 -0600 Subject: [Python-ideas] Adding a thin wrapper class around the functions in stdlib.heapq In-Reply-To: References: Message-ID: Perhaps such repetition is a sign that *something* needs to be done... Thanks for the link though. I'm new enough to the community that it didn't even occur to me to search for prior discussions. On Mon, Nov 20, 2017 at 8:38 PM, Sebastian Kreft wrote: > This has been brought up multiple times. Last time was on this thread > https://mail.python.org/pipermail/python-ideas/2016-October/043024.html. > > On Tue, Nov 21, 2017 at 3:13 AM, bunslow wrote: > >> Nothing so bombastic this time. The heapq functions are basically all >> named "heapsomething", and basically all take a "heap" for their first >> argument, with supplementary args coming after. It's a textbook example of >> the (hypothetical) Object Oriented Manifesto? where defining a class >> increases type safety and programmers' conceptual clarity. There're >> practically no drawbacks, and the code to be added would be very simple. >> Updating the tests and docs would probably be harder. >> >> In pure Python, such a class would look like this: >> >> class Heap(list): >> >> def __init__(self, iterable=None): >> if iterable: >> super().__init__(iterable) >> else: >> super().__init__() >> >> self.heapify() >> >> push = heapq.heappush >> pop = heapq.heappop >> pushpop = heapq.heappushpop >> replace = heapq.heapreplace >> heapify = heapq.heapify >> >> # This could be a simple wrapper as well, but I had the following >> thoughts anyways, so here they are >> def nsmallest(self, n, key=None): >> # heapq.nsmallest makes a *max* heap of the first n elements, >> # while we know that self is already a min heap, so we can >> # make the max heap construction faster >> self[:n] = reversed(self[:n]) >> return heapq.nsmallest(n, self, key) >> >> # do we define nlargest on a minheap?? >> >> >> Wrapping around the C builtin functions (which aren't descriptors) would >> be a bit harder, but not much so: >> >> from functools import partial >> >> class Heap(list): >> def __init__(self, iterable=None): >> if iterable: >> super().__init__(iterable) >> else: >> super().__init__() >> >> self.heapify = partial(heapq.heapify, self) >> self.push = partial(heapq.heappush, self) >> ... >> >> self.heapify() >> >> >> Thoughts? >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> > > > -- > Sebastian Kreft > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mistersheik at gmail.com Mon Nov 20 21:47:24 2017 From: mistersheik at gmail.com (Neil Girdhar) Date: Tue, 21 Nov 2017 02:47:24 +0000 Subject: [Python-ideas] Consider (one day) adding an inheritance order class precedence mechanism In-Reply-To: References: <20171116001841.GN19802@ando.pearwood.info> <5A0D2BC7.3040209@canterbury.ac.nz> <20171117055252.GS19802@ando.pearwood.info> Message-ID: On Mon, Nov 20, 2017 at 9:41 PM Nick Coghlan wrote: > On 21 November 2017 at 12:34, Nick Coghlan wrote: > > Right, but once you do that, then the existing resolver is already > > able to handle things: > > > > >>> class C: pass > > ... > > >>> class S(C): pass > > ... > > >>> class E: pass > > ... > > >>> class B(S, E, C): pass > > ... > > >>> class R(E, C): pass > > ... > > >>> class Z(B, R): pass > > ... > > >>> > > > > If you wanted to pick up cases where two classes generate inconsistent > > MROs that will prevent mutual subclasses (like "class B(S, E)" vs > > "class R(E, C)"), that feels like a job for a type checker, since it > > isn't obvious whether it's the definition of B or the definition of R > > that should be changed to reorder their MRO. > > I do wonder if we might be able to make the error message here less > cryptic by mentioning which *listed* base classes brought in the > conflicting MRO entries. > > Consider the current: > > >>> class Z(B, R): pass > ... > Traceback (most recent call last): > File "", line 1, in > TypeError: Cannot create a consistent method resolution order > (MRO) for bases C, E > > vs something like: > > TypeError: Cannot create a consistent method resolution order > (MRO) for bases C, E (inherited through B, R) > > (Note: I haven't actually checked how practical it would be to > implement something like that) > I totally agree. I actually proposed this on ideas a few months ago and then wrote something here: https://github.com/NeilGirdhar/inheritance_graph If you have any suggestions or improvements, please feel free to improve the code. Best, Neil > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikhailwas at gmail.com Mon Nov 20 22:16:54 2017 From: mikhailwas at gmail.com (Mikhail V) Date: Tue, 21 Nov 2017 04:16:54 +0100 Subject: [Python-ideas] Allow additional separator character in variables In-Reply-To: <23059.34466.1253.737550@turnbull.sk.tsukuba.ac.jp> References: <23059.34466.1253.737550@turnbull.sk.tsukuba.ac.jp> Message-ID: On Tue, Nov 21, 2017 at 2:51 AM, Stephen J. Turnbull wrote: > Mikhail V writes: > > > On Mon, Nov 20, 2017 at 5:46 AM, Guido van Rossum wrote: > > > Please kill this thread. > > > > So the idea is too bad? > > No, the idea is *not* bad, it's just not for Python. As has been true > for every one of your ideas for language tweaks that I can recall. Not every, but many, yes. And there is plethora of proposals less plausible and not for Python. Anyway, I'll stick to python-list better for such topics. BTW, as per Serhiy Storchaka's note: my?variable my?variable my?variable my?variable my?variable my?variable ^ Is this good idea *for Python*? I mean this is not Python that I knew. I don't know how it is possible. Looks like a result of some unlucky nuclear experment. Might be it will not cause any possible confusion, or less than a hyphen and a minus. > The solution is as proposed earlier: create your own language. [...] > language, because other people may feel the same level of attraction > to it. Not much interested in *my own language*. Simple translator for hyphens and minuses I have already made, and I enjoy it. If the new language thing would happen and gained popularity - it would be the worst scenario - competing syntaxes, CO2 emissions, community splittage, etc. I don't endorse such ideas. Mikhail From storchaka at gmail.com Tue Nov 21 01:16:50 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 21 Nov 2017 08:16:50 +0200 Subject: [Python-ideas] Adding a new function "zip_flat" to itertools (Re: Rewriting the "roundrobin" recipe in the itertools documentation) In-Reply-To: References: <20171120160802.GH19802@ando.pearwood.info> <20171120215757.GJ19802@ando.pearwood.info> Message-ID: 21.11.17 01:00, Terry Reedy ????: > On 11/20/2017 4:57 PM, Steven D'Aprano wrote: >> Is there a reason for calling reversed() instead of reversing the order >> of range in the first place? >> >> range(len(iterables)-1, -1, -1) > > Readability.? Accurately and automatically reversing ranges was one of > the use cases motivating the addition of 'reversed'.? Ranges have a > __reversed__ method which returns a iterator for a reversed range. Agree. You also can write this as range(1, len(iterables))[::-1]. This is shorter and (I suppose) slightly faster. But reverse() can be better choice for a recipe, if it looks less confusing for beginners. From storchaka at gmail.com Tue Nov 21 01:47:59 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 21 Nov 2017 08:47:59 +0200 Subject: [Python-ideas] Allow additional separator character in variables In-Reply-To: References: <23059.34466.1253.737550@turnbull.sk.tsukuba.ac.jp> Message-ID: 21.11.17 05:16, Mikhail V ????: > my?variable > my?variable > my?variable > my?variable > my?variable > my?variable > > ^ Is this good idea *for Python*? I mean this is not Python that > I knew. I don't know how it is possible. Looks like a result of > some unlucky nuclear experment. Might be it will not cause any possible > confusion, or less than a hyphen and a minus. Yes, it causes less confusion that changing meaning of a minus. And yes, it can cause confusion if misused. As well as using the following variables: ?yvariable m?variable myv?riable myvar?able myvari?ble myvaria?le myvariab1e myvariabl? But the name ????????? doesn't cause any confusion if used in an appropriate context (for example in a lesson for young Ukrainian children). I believe the above dot- and hyphen-like characters don't cause confusion if used as letters in an appropriate language context. From saeedbaig616 at icloud.com Tue Nov 21 01:33:55 2017 From: saeedbaig616 at icloud.com (Saeed Baig) Date: Tue, 21 Nov 2017 17:33:55 +1100 Subject: [Python-ideas] Should Python have user-defined constants? Message-ID: <29DDA3B1-8ED1-46C6-BF4E-DBD4DE9240C3@icloud.com> Hey guys I am thinking of perhaps writing a PEP to introduce user-defined constants to Python. Something along the lines of Swift?s ?let? syntax (e.g. ?let pi = 3.14?). Do you guys think it would be a good idea? Why or why not? Do you think there?s a better way to do it? I?d like to know what others think about this idea before making any formal submission (I?ve already posted this same question on python-list, but I just wanted to gauge opinion here too). -------------- next part -------------- An HTML attachment was scrubbed... URL: From songofacandy at gmail.com Tue Nov 21 02:38:14 2017 From: songofacandy at gmail.com (INADA Naoki) Date: Tue, 21 Nov 2017 16:38:14 +0900 Subject: [Python-ideas] Should Python have user-defined constants? In-Reply-To: <29DDA3B1-8ED1-46C6-BF4E-DBD4DE9240C3@icloud.com> References: <29DDA3B1-8ED1-46C6-BF4E-DBD4DE9240C3@icloud.com> Message-ID: I'm -1. I feel Python is complex language already. And I don't want make it more complicate. INADA Naoki On Tue, Nov 21, 2017 at 3:33 PM, Saeed Baig wrote: > Hey guys I am thinking of perhaps writing a PEP to introduce user-defined > constants to Python. Something along the lines of Swift?s ?let? syntax (e.g. > ?let pi = 3.14?). > > Do you guys think it would be a good idea? Why or why not? Do you think > there?s a better way to do it? I?d like to know what others think about this > idea before making any formal submission (I?ve already posted this same > question on python-list, but I just wanted to gauge opinion here too). > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From storchaka at gmail.com Tue Nov 21 02:38:40 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 21 Nov 2017 09:38:40 +0200 Subject: [Python-ideas] Should Python have user-defined constants? In-Reply-To: <29DDA3B1-8ED1-46C6-BF4E-DBD4DE9240C3@icloud.com> References: <29DDA3B1-8ED1-46C6-BF4E-DBD4DE9240C3@icloud.com> Message-ID: 21.11.17 08:33, Saeed Baig ????: > Hey guys I am thinking of perhaps writing a PEP to introduce > user-defined constants to Python. Something along the lines of Swift?s > ?let? syntax (e.g. ?let pi = 3.14?). > > Do you guys think it would be a good idea? Why or why not? Do you think > there?s a better way to do it? To do what? What problem do you need to solve? From joejev at gmail.com Tue Nov 21 02:38:45 2017 From: joejev at gmail.com (Joseph Jevnik) Date: Tue, 21 Nov 2017 02:38:45 -0500 Subject: [Python-ideas] Should Python have user-defined constants? In-Reply-To: <29DDA3B1-8ED1-46C6-BF4E-DBD4DE9240C3@icloud.com> References: <29DDA3B1-8ED1-46C6-BF4E-DBD4DE9240C3@icloud.com> Message-ID: How is that different from "pi = 3.14"? On Tue, Nov 21, 2017 at 1:33 AM, Saeed Baig wrote: > Hey guys I am thinking of perhaps writing a PEP to introduce user-defined > constants to Python. Something along the lines of Swift?s ?let? syntax (e.g. > ?let pi = 3.14?). > > Do you guys think it would be a good idea? Why or why not? Do you think > there?s a better way to do it? I?d like to know what others think about this > idea before making any formal submission (I?ve already posted this same > question on python-list, but I just wanted to gauge opinion here too). > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From gadgetsteve at live.co.uk Tue Nov 21 01:21:37 2017 From: gadgetsteve at live.co.uk (Steve Barnes) Date: Tue, 21 Nov 2017 06:21:37 +0000 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <5A087DE7.4050306@brenbarn.net> <8d5574dc-b614-ccfe-3ce8-f1ba86508179@python.org> Message-ID: On 21/11/2017 00:32, Chris Barker wrote: > On Mon, Nov 20, 2017 at 3:24 PM, Paul Moore > wrote: > > On 20 November 2017 at 21:59, Chris Barker > wrote: > > I don't understand any of this enough to have an opinion, so > while I'd like > > to see py.exe be renamed python.exe, let's not let "the perfect > be the enemy > > of good enough". So, if someone can make the case that they can > restructure > > the whole py.exe / python.exe thing nicely in a way that will > work, AND > > write the code to do it fairly quickly, then great! > > I'm happy enough to modify the py.exe code to base its behaviour on > the name it's called with, as Steve suggested. I'm not sure if that > would require a PEP, but I'm willing to do one if it's felt that there > is a need. It should be possible to get that change in for 3.7. > > > very cool -- thanks! I agree that it would be. > > I > don't know what would be the best approach for adding copies/links of > the launcher under other names, though. > > > this is the thing -- I wonder if py.exe has been a success at all :-) A lot of people, on windows, are using py.exe and pyw.exe without realising it - the default behaviour of the recent installer is to associate py.exe with *.py and pyw.exe with *.pyw so any user that is running python code by just typing the name of the file, or double clicking it, will be using it. This is the equivalent of the Unix chmod + shebang - I don't think that Windows users are ready for chown/chmod. > > I honestly don't know - it's a cool idea, and I liked the idea of > replacing the #! line and all that, but the truth is that people simple > don't write scripts on Windows and expect them to "just work" the same > way they do on *nix systems. And if you do distutils/setuptools > "scripts" or "entry points" or whatever we are calling them, right, then > you can write utility scripts and install them, and use them.... > > So ti comes down to: does anyone USE py.exe??? > As above. > The fact that it's unique to Windows? makes it simply less discoverable, > and the fact that Windows users aren't used to the whole #! line thing, > means that as useful as it might be, it's not getting actual use. > > In fact, I happened to be paying a lot of attention to this conversion > because at the very same moment, a group of us are developing a python > curriculum, and working on the "how to run a python file" part. We are > producing platform specific instructions, but decided NOT to include the > py.exe option for Windows users .. just more confusing things among a > lot of confusing things! > > So who does teach people about it??? I quite regularly give people python courses and I always, regardless of target platform, tell them about the shebang and encoding lines, explain why they need them, including many editors basing syntax highlighting and character handling on them. I also point out that some nasty person, (me), has embedded the requirement to use them in the company coding standard for python and the review criteria for all code. (So carrot and stick). > > Anyway, this whole thread has been about making the experience more > platform-independent -- ideally users should see docs without > Windows-specifc advice, and have it "just work" -- having a python3.exe > would make that more likely, so let's do that. > > If we could py.exe enhanced and then use it instead of python.ex,m that > would be great. Though I at least still have no idea of that's possible. > > But I don't see why we couldn't make a pyton3.exe NOW (in py3.7) that is > a copy (Or link, still don't get linking on Windows...) of python.exe, > that would help the situation, and if at some point the in the future, > python.exe, python3.exe and py.exe all become the same thing, great! > > I think the launcher change is worthwhile in its own right. Even if we > don't install any aliases by default, users will be able to manually > add them. > > > sure -- but no one would -- or not really no one, but it would become > maybe even a worse hassle of the same thing not working everywhere. > > we need th=e basic advice given out to *nix users to work: > > if you type "python" at the command line, and get python2, then you can > type "python3" to get python 3. > > simple, straightforward, and it CAN help. > > > Otherwise, let's at least get a python3.exe into 3.7 -- and > ideally into > > updates to 3.5, 3.6, and (python2.exe in this case, 2.7) > > It's not going to be acceptable for 3.5, as that is now in security > fix only mode. I'm not certain it's even acceptable for bugfix-only > releases. It's a backward incompatible change (3.6.3 has no > python3.exe but python 3.6.4 does) but maybe an acceptable one - we'd > need feedback from the 3.6 and 2.7 release managers on that. > One of the nice things about embedding the behaviour into the launcher is that if the user/admin has installed a python that includes it it provides the behaviour for python installations that were already present and discoverable even if they didn't have it so if you have python 2.5 then after you install something with py.exe you can, currently, use py -2.5 to run it, (or py -2 if it is the only python 2). > > Fine -- we can leave that for future discussion.... > > Also, I'm assuming here you mean "create a copy of python.exe called > python3.exe". If we do that, we risk making it harder to later switch > to making "python3.exe" a link to the launcher - for the same reason > that making "python.exe" be an alternative name for the launcher is > problematic right now. > > > I'm lost -- how does it make it any harder, rather than exactly the same ??? > (except two changes, rather than one...but they are the same change) > > > And maybe make "add to PATH" be the default in the installer. > > I'm not willing to contradict Steve on this one. > > > I didn't know he'd made a clear statement about it. I am not sure that I spelt it out well enough but my personal feeling is that the installer should, (and I am reasonably sure that it should capable of this), default to add to path IF there is not a currently available python, (discoverable), and either default to don't add or prompt the installer if there is. Of course the whole add to path or not becomes moot if the py.exe launcher is present as it is installed on the path, (C:\Windows), and then tries very hard just to "do the right thing". > > But if people that understand these things say it's a bad idea, then > it's a bad idea. But we SHOULD make sure we are considering not just > what could go wrong, or how often it could go wrong, but who it is going > to go wrong for... > > i.e. making it "do the right thing" for newbies is much more important > that making sure it doesn't "do the wrong thing" for more experienced > folks that can uncheck a box, or clean up their PATH after the fact.... > > There are too many > not-uncommon cases that we could mess up badly here. It's the right > choice for "make new users' experience as easy as possible", but if we > do this at the cost of making the experience of people upgrading to > 3.7 (possibly by installing 3.7 to gradually switch over, > > > then you uncheck the box -- and if someone is installing 3.7 to test it > out, they SHOULD be a experience enough to uncheck a box... > > or starting > their migration from 2.7 with 3.7) unpleasant, > > > why would that be a problem???? > > Though we may need a "python2.exe" executable for that to work well :-( > > The history of how we added Python to PATH across various versions is > messy and inconsistent. Add to that other distributions (Anaconda, > ActiveState) making different choices and adding yet more > inconsistency, means that anyone who *isn't* a brand new user probably > already has a tweaked, and likely fragile, setup. > > We're not doing them > any favours by adding another new behaviour. > > > I'm not trying to do them any favors -- I'm trying to do the favors for > the newbies.. and YES, at the expense of people with "tweaked, and > likely fragile, setup." > > We need *another* solution here. Not just variations on the existing > mess. That's why I like Steve's suggestion of making the launcher into > the canonical entry point. > > > only if it's named "python" (and python3) -- what we DON'T need is a > different way to do it on Windows -- that's what we tried, and I thin > it's failed. > > It's not easy, but at least it stands a > chance of breaking out of the cycle we're currently in, of switching > back and forth between "add to PATH" and "don't add to PATH". > > > It also seems completely orthogonal to the "add to PATH" question -- > having this nifty new launcher setup, but not on a newbies PATH where > they can use it, isn't going to help at all. > > > Those are quick and simple to do, result in little disruption, > and make the > > whole cross-platform thing more manageable. > > They aren't that simple. I've already discussed "add to PATH". And if > we add python3.exe, we have to consider questions like will venv be > modified to include it in virtual environments? Will virtualenv? > > > what do those systems do on *nix? do exactly the same thing on Windows. > Done. > > Also -- I don't much care -- I want newbies that aren't using virtual > environments to have a better expeience -- virtualenv pretty much solves > this issue IF they are being used. > > "Although never is often better than *right* now" is the relevant > Zen here, not "Now is better than never" (nobody's suggesting we > *never* fix this). > > > if we take long enough, everyone will be using python3 anyway :-) > > -CHB > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R ? ? ? ? ? ?(206) 526-6959?? voice > 7600 Sand Point Way NE ??(206) 526-6329?? fax > Seattle, WA ?98115 ? ? ??(206) 526-6317?? main reception > > Chris.Barker at noaa.gov -- Steve (Gadget) Barnes Any opinions in this message are my personal opinions and do not reflect those of my employer. --- This email has been checked for viruses by AVG. http://www.avg.com From steve at pearwood.info Tue Nov 21 03:13:42 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 21 Nov 2017 19:13:42 +1100 Subject: [Python-ideas] Should Python have user-defined constants? In-Reply-To: References: <29DDA3B1-8ED1-46C6-BF4E-DBD4DE9240C3@icloud.com> Message-ID: <20171121081342.GK19802@ando.pearwood.info> On Tue, Nov 21, 2017 at 02:38:45AM -0500, Joseph Jevnik wrote: > How is that different from "pi = 3.14"? pi = 3.14 pi = 5 print(pi) # prints 5 let pi = 3.14 pi = 5 # raises an exception -- Steve From sf at fermigier.com Tue Nov 21 03:26:17 2017 From: sf at fermigier.com (=?UTF-8?Q?St=C3=A9fane_Fermigier?=) Date: Tue, 21 Nov 2017 09:26:17 +0100 Subject: [Python-ideas] Should Python have user-defined constants? In-Reply-To: <20171121081342.GK19802@ando.pearwood.info> References: <29DDA3B1-8ED1-46C6-BF4E-DBD4DE9240C3@icloud.com> <20171121081342.GK19802@ando.pearwood.info> Message-ID: Javascript (ES6) has 'let' and 'const' for mutable and constant variables, respectively. When programming in JS, I find myself aggressively aiming for as little 'let' and as much 'const' as reasonably possible, since reasoning about constants is much easier than about variables. In this context, 'const' is used as a marker, a runtime checker, and, with proper tooling (IDE or linter), a coding-time reminder to differentiate between these two fundamental behaviours. I'm not +1 on introducing this idea to Python yet, but not -1 either - this deserves some discussion, if this has not been discussed already. Cheers, S. On Tue, Nov 21, 2017 at 9:13 AM, Steven D'Aprano wrote: > On Tue, Nov 21, 2017 at 02:38:45AM -0500, Joseph Jevnik wrote: > > > How is that different from "pi = 3.14"? > > pi = 3.14 > pi = 5 > print(pi) > # prints 5 > > let pi = 3.14 > pi = 5 > # raises an exception > > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier - http://linkedin.com/in/sfermigier Founder & CEO, Abilian - Enterprise Social Software - http://www.abilian.com/ Chairman, Free&OSS Group / Systematic Cluster - http://www.gt-logiciel-libre.org/ Co-Chairman, National Council for Free & Open Source Software (CNLL) - http://cnll.fr/ Founder & Organiser, PyData Paris - http://pydata.fr/ --- ?You never change things by ?ghting the existing reality. To change something, build a new model that makes the existing model obsolete.? ? R. Buckminster Fuller -------------- next part -------------- An HTML attachment was scrubbed... URL: From mafagafogigante at gmail.com Tue Nov 21 03:36:47 2017 From: mafagafogigante at gmail.com (Bernardo Sulzbach) Date: Tue, 21 Nov 2017 06:36:47 -0200 Subject: [Python-ideas] Should Python have user-defined constants? In-Reply-To: References: <29DDA3B1-8ED1-46C6-BF4E-DBD4DE9240C3@icloud.com> <20171121081342.GK19802@ando.pearwood.info> Message-ID: <0028cb32-78e6-783e-e5d1-fc266e24e9fd@gmail.com> This could also be useful if one eventually wanted to implement constant optimizations in the interpreter as it would be possible to detect constants when they are declared without any code analysis. This would be "constant" as in Java constants right? Referential constants, so that if an object were mutable you would still be able to change its internal state. -- Bernardo Sulzbach http://www.mafagafogigante.org/ mafagafogigante at gmail.com From kirillbalunov at gmail.com Tue Nov 21 03:54:37 2017 From: kirillbalunov at gmail.com (Kirill Balunov) Date: Tue, 21 Nov 2017 11:54:37 +0300 Subject: [Python-ideas] Star assignment in iterator way? Message-ID: Hello. Currently during star assignement the new list is created. What was the idea to produce new list instead of returning an iterator? It seems to me that returning an iterator more suited to the spirit of Python 3. There are three cases: 1. a,b,c,*d = something_iterable 2. *a,b,c,d = something_iterable 3. a,*b,c,d = something_iterable The first one is obvious. For the rest two we always need to iterate through entire iterable to achieve values for b,c,d (or c,d) binding. But this can be done more memory effiecient than currently (may be I'm wrong). And we can iterate in space of last three (or two) variables. Some rough (simplified) Python code: from itertools import islice, chain from collections import deque def good_star_exp(signature, seq): if signature.count('*') > 1: raise SyntaxError('two starred expressions in assignment') vrs = signature.split(',') idx_max = len(vrs) - 1 star_pos, = (i for i,v in enumerate(vrs) if '*' in v) #First case if star_pos == idx_max: head = islice(seq, idx_max) tail = islice(seq, idx_max, None) return chain(head, (tail,)) #Second case elif star_pos == 0: tail = deque(maxlen=idx_max) for seq_idx_max, v in enumerate(seq): tail.append(v) head = islice(seq, 0, seq_idx_max-(idx_max-1)) return chain([head], tail) #Third case else: head = islice(seq, star_pos) tail = deque(maxlen=(idx_max-star_pos)) for seq_idx_max, v in enumerate(seq): tail.append(v) mid = islice(seq, star_pos, seq_idx_max-(idx_max-2)) return chain(head, [mid], tail) ls = range(100000) a,b,c,d = good_star_exp('a,b,c,*d', ls) a,b,c,d = good_star_exp('*a,b,c,d', ls) a,b,c,d = good_star_exp('a,*b,c,d', ls) Of course this version has drawbacks (the first that come to mind): 1. Will *b see change if rhs is some muttable sequence? 2. Will *b one way iterator or somethong like range? But still it seems to me that the "iterator way" has more useful applications. With best regards, -gdg -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Tue Nov 21 04:07:58 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 21 Nov 2017 11:07:58 +0200 Subject: [Python-ideas] Star assignment in iterator way? In-Reply-To: References: Message-ID: 21.11.17 10:54, Kirill Balunov ????: > Of course this version has drawbacks (the first that come to mind): > 1. Will *b see change if rhs is some muttable sequence? > 2. Will *b one way iterator or somethong like range? > > But still it seems to me that the "iterator way" has more useful > applications. Your implementation iterates seq multiple times. But iterable unpacking syntax works with an arbitrary iterable, and iterates it only once. Changing the result of iterable unpacking will break existing code that depends on the result been a list. And you already have mentioned a question about mutable sequence. If these conditions and restrictions suit you, you can use your good_star_exp() in your code or share it with others. But the semantic of iterable unpacking can't be changed. From storchaka at gmail.com Tue Nov 21 04:18:19 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 21 Nov 2017 11:18:19 +0200 Subject: [Python-ideas] Ignorable whitespaces in the re.VERBOSE mode In-Reply-To: <23059.36185.852770.297907@turnbull.sk.tsukuba.ac.jp> References: <23058.36540.802392.890565@turnbull.sk.tsukuba.ac.jp> <23059.36185.852770.297907@turnbull.sk.tsukuba.ac.jp> Message-ID: 21.11.17 04:20, Stephen J. Turnbull ????: > Serhiy Storchaka writes: > > I agree. But if there is a special part of the Unicode standard for > > Pattern White Spaces which includes non-ASCII characters, perhaps there > > is a need in them. I asked for the case if Python developers with very > > different cultures have need in additional whitespaces in regular > > expressions, but I don't know. Seems nobody has claimed their need. > > I doubt that Japanese would want it. I do use \N{IDEOGRAPHIC SPACE} a > bit as a *target* of regular expressions, but I would never want it as > non-syntactic in re.VERBOSE. (Of course, I'm not a native Japanese, but > I have never heard a Japanese developer wish for use of that character > in any programming language, outside of literal strings.) > > > In particularly I don't know how helpful would be supporting > > right-to-left and left-to-right marks in verbose regular expressions > > That's a good question. Interpretation and display of R2L in > programming constructs came up briefly in the discussions about BIDI > on the emacs-devel list. I'll ask Eli Zaretskii, who implemented it > for Emacs. Thank you Stephen. I would prefer to not change anything (because supporting additional whitespaces will complicate and slow down the code, and can add subtle bugs, add likely will add a confusion for users). But I want to know whether there is a real need in supporting additional whitespaces and rtl and ltr marks in regular expressions and Python syntax. From kirillbalunov at gmail.com Tue Nov 21 04:27:32 2017 From: kirillbalunov at gmail.com (Kirill Balunov) Date: Tue, 21 Nov 2017 12:27:32 +0300 Subject: [Python-ideas] Star assignment in iterator way? In-Reply-To: References: Message-ID: > Your implementation iterates seq multiple times. But iterable unpacking > syntax works with an arbitrary iterable, and iterates it only once. > Oh sorry, I know that my implementation iterates seq multiple times, I only provide this to show the idea. It can be much optimized at C level. I just want to understand if it's worth the time and effort. > Changing the result of iterable unpacking will break existing code that > depends on the result been a list. > Backward compatibility is an important issue, but at the same time it is the main brake on progress. > And you already have mentioned a question about mutable sequence. > > If these conditions and restrictions suit you, you can use your > good_star_exp() in your code or share it with others. But the semantic of > iterable unpacking can't be changed. > And how do you look at something like this (deferred star evaluation)?: a, ?*b, c, d = something_iterable With kind regards, -gdg -------------- next part -------------- An HTML attachment was scrubbed... URL: From saeedbaig616 at icloud.com Tue Nov 21 04:34:31 2017 From: saeedbaig616 at icloud.com (Saeed Baig) Date: Tue, 21 Nov 2017 20:34:31 +1100 Subject: [Python-ideas] Should Python have user-defined constants? In-Reply-To: References: Message-ID: <742BABA7-C73A-490F-96EB-B65E44EAC236@icloud.com> Hi. Just to respond to some peoples? points? Bernardo asked 'This would be "constant" as in Java constants right? Referential constants, so that if an object were mutable you would still be able to change its internal state.? Um, I?m not entirely sure what you mean, since I?ve never used Java. But if it means what I think it means, I would be INCLINED to say no. If, for example, I saw code like this: x = [9, 5, 7, 1] let y = x y[0] = 42 print(y) ? I would expect that code to raise an error, not print ?[42, 5, 7, 1]?, since you?re trying to modify the ?constant? y in place, which? well, would kind of go against the idea of y being a constant, I thought (unless I?m misunderstanding what you meant by ?object", since everything in Python?s an object). However, I might change my opinion if I heard a good reason why Java-like constants would be better. Serhiy asked, in relation to constants, ?To do what? What problem do you need to solve??. No problem in particular, to be honest. I just thought they?d be nice since they?d increase confidence that my variables-intended-to-be-constants wouldn?t get reassigned, and just to increase readability/explicitness to other programmers. > On 21 Nov 2017, at 20:08, python-ideas-request at python.org wrote: > > Send Python-ideas mailing list submissions to > python-ideas at python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman/listinfo/python-ideas > or, via email, send a message with subject or body 'help' to > python-ideas-request at python.org > > You can reach the person managing the list at > python-ideas-owner at python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Python-ideas digest..." > > > Today's Topics: > > 1. Re: Should Python have user-defined constants? (Steven D'Aprano) > 2. Re: Should Python have user-defined constants? (St?fane Fermigier) > 3. Re: Should Python have user-defined constants? (Bernardo Sulzbach) > 4. Star assignment in iterator way? (Kirill Balunov) > 5. Re: Star assignment in iterator way? (Serhiy Storchaka) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 21 Nov 2017 19:13:42 +1100 > From: Steven D'Aprano > To: python-ideas at python.org > Subject: Re: [Python-ideas] Should Python have user-defined constants? > Message-ID: <20171121081342.GK19802 at ando.pearwood.info> > Content-Type: text/plain; charset=us-ascii > > On Tue, Nov 21, 2017 at 02:38:45AM -0500, Joseph Jevnik wrote: > >> How is that different from "pi = 3.14"? > > pi = 3.14 > pi = 5 > print(pi) > # prints 5 > > let pi = 3.14 > pi = 5 > # raises an exception > > > > -- > Steve > > > ------------------------------ > > Message: 2 > Date: Tue, 21 Nov 2017 09:26:17 +0100 > From: St?fane Fermigier > To: Python-Ideas > Subject: Re: [Python-ideas] Should Python have user-defined constants? > Message-ID: > > Content-Type: text/plain; charset="utf-8" > > Javascript (ES6) has 'let' and 'const' for mutable and constant variables, > respectively. > > When programming in JS, I find myself aggressively aiming for as little > 'let' and as much 'const' as reasonably possible, since reasoning about > constants is much easier than about variables. In this context, 'const' is > used as a marker, a runtime checker, and, with proper tooling (IDE or > linter), a coding-time reminder to differentiate between these two > fundamental behaviours. > > I'm not +1 on introducing this idea to Python yet, but not -1 either - this > deserves some discussion, if this has not been discussed already. > > Cheers, > > S. > > On Tue, Nov 21, 2017 at 9:13 AM, Steven D'Aprano > wrote: > >> On Tue, Nov 21, 2017 at 02:38:45AM -0500, Joseph Jevnik wrote: >> >>> How is that different from "pi = 3.14"? >> >> pi = 3.14 >> pi = 5 >> print(pi) >> # prints 5 >> >> let pi = 3.14 >> pi = 5 >> # raises an exception >> >> >> >> -- >> Steve >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > > -- > Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier - > http://linkedin.com/in/sfermigier > Founder & CEO, Abilian - Enterprise Social Software - > http://www.abilian.com/ > Chairman, Free&OSS Group / Systematic Cluster - > http://www.gt-logiciel-libre.org/ > Co-Chairman, National Council for Free & Open Source Software (CNLL) - > http://cnll.fr/ > Founder & Organiser, PyData Paris - http://pydata.fr/ > --- > ?You never change things by ?ghting the existing reality. To change > something, build a new model that makes the existing model obsolete.? ? R. > Buckminster Fuller > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > > ------------------------------ > > Message: 3 > Date: Tue, 21 Nov 2017 06:36:47 -0200 > From: Bernardo Sulzbach > To: python-ideas at python.org > Subject: Re: [Python-ideas] Should Python have user-defined constants? > Message-ID: <0028cb32-78e6-783e-e5d1-fc266e24e9fd at gmail.com> > Content-Type: text/plain; charset=utf-8; format=flowed > > This could also be useful if one eventually wanted to implement constant > optimizations in the interpreter as it would be possible to detect > constants when they are declared without any code analysis. > > This would be "constant" as in Java constants right? Referential > constants, so that if an object were mutable you would still be able to > change its internal state. > > -- > Bernardo Sulzbach > http://www.mafagafogigante.org/ > mafagafogigante at gmail.com > > > ------------------------------ > > Message: 4 > Date: Tue, 21 Nov 2017 11:54:37 +0300 > From: Kirill Balunov > To: python-ideas at python.org > Subject: [Python-ideas] Star assignment in iterator way? > Message-ID: > > Content-Type: text/plain; charset="utf-8" > > Hello. Currently during star assignement the new list is created. > What was the idea to produce new list instead of returning an iterator? > It seems to me that returning an iterator more suited to the spirit of > Python 3. > There are three cases: > > 1. a,b,c,*d = something_iterable > 2. *a,b,c,d = something_iterable > 3. a,*b,c,d = something_iterable > > The first one is obvious. For the rest two we always need to iterate > through > entire iterable to achieve values for b,c,d (or c,d) binding. But this can > be done > more memory effiecient than currently (may be I'm wrong). And we can > iterate in > space of last three (or two) variables. Some rough (simplified) Python code: > > from itertools import islice, chain > from collections import deque > > def good_star_exp(signature, seq): > > if signature.count('*') > 1: > raise SyntaxError('two starred expressions in assignment') > > vrs = signature.split(',') > idx_max = len(vrs) - 1 > star_pos, = (i for i,v in enumerate(vrs) if '*' in v) > > #First case > if star_pos == idx_max: > head = islice(seq, idx_max) > tail = islice(seq, idx_max, None) > return chain(head, (tail,)) > > #Second case > elif star_pos == 0: > tail = deque(maxlen=idx_max) > for seq_idx_max, v in enumerate(seq): > tail.append(v) > head = islice(seq, 0, seq_idx_max-(idx_max-1)) > return chain([head], tail) > > #Third case > else: > head = islice(seq, star_pos) > tail = deque(maxlen=(idx_max-star_pos)) > for seq_idx_max, v in enumerate(seq): > tail.append(v) > mid = islice(seq, star_pos, seq_idx_max-(idx_max-2)) > return chain(head, [mid], tail) > > ls = range(100000) > a,b,c,d = good_star_exp('a,b,c,*d', ls) > a,b,c,d = good_star_exp('*a,b,c,d', ls) > a,b,c,d = good_star_exp('a,*b,c,d', ls) > > Of course this version has drawbacks (the first that come to mind): > 1. Will *b see change if rhs is some muttable sequence? > 2. Will *b one way iterator or somethong like range? > > But still it seems to me that the "iterator way" has more useful > applications. > > With best regards, -gdg > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: > > ------------------------------ > > Message: 5 > Date: Tue, 21 Nov 2017 11:07:58 +0200 > From: Serhiy Storchaka > To: python-ideas at python.org > Subject: Re: [Python-ideas] Star assignment in iterator way? > Message-ID: > Content-Type: text/plain; charset=utf-8; format=flowed > > 21.11.17 10:54, Kirill Balunov ????: >> Of course this version has drawbacks (the first that come to mind): >> 1. Will *b see change if rhs is some muttable sequence? >> 2. Will *b one way iterator or somethong like range? >> >> But still it seems to me that the "iterator way" has more useful >> applications. > > Your implementation iterates seq multiple times. But iterable unpacking > syntax works with an arbitrary iterable, and iterates it only once. > > Changing the result of iterable unpacking will break existing code that > depends on the result been a list. > > And you already have mentioned a question about mutable sequence. > > If these conditions and restrictions suit you, you can use your > good_star_exp() in your code or share it with others. But the semantic > of iterable unpacking can't be changed. > > > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > > > ------------------------------ > > End of Python-ideas Digest, Vol 132, Issue 112 > ********************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Tue Nov 21 04:39:04 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 21 Nov 2017 09:39:04 +0000 Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <5A087DE7.4050306@brenbarn.net> <8d5574dc-b614-ccfe-3ce8-f1ba86508179@python.org> Message-ID: [Excuse any attribution errors, the quoting appears to have got badly messed up somehow] On 21 November 2017 at 06:21, Steve Barnes wrote: > > On 21/11/2017 00:32, Chris Barker wrote: >> I >> don't know what would be the best approach for adding copies/links of >> the launcher under other names, though. >> >> >> this is the thing -- I wonder if py.exe has been a success at all :-) > > A lot of people, on windows, are using py.exe and pyw.exe without > realising it - the default behaviour of the recent installer is to > associate py.exe with *.py and pyw.exe with *.pyw so any user that is > running python code by just typing the name of the file, or double > clicking it, will be using it. I use the py launcher a lot. Often just as "py" or "py -3.6". It certainly isn't just limited to use via shebang lines. I will say that I don't see beginners using it as much. That's probably because there's so much documentation that says "use the python command to start Python". That is of course an argument that we should make the canonical command "python" rather than "py", but it doesn't mean that if people find the "py" command they won't use it. There's not a lot of evidence that isn't anecdotal either way on that latter point. >> So ti comes down to: does anyone USE py.exe??? >> > As above. And me. >> we need th=e basic advice given out to *nix users to work: >> >> if you type "python" at the command line, and get python2, then you can >> type "python3" to get python 3. Um, we need the basic advice given to *everyone* to work. Insisting that advice has to be "what Unix people do right now" is not the only option. The advice "use python to run your default Python interpreter" is too entrenched to change. The python2 and python3 aliases are a platform-specific hack to get around the fact that some Linux distributions couldn't switch the default Python to Python 3. Most of the beginners I work with have no experience on Unix and no awareness of python2/python3 commands. "If you run python at the command line and get the wrong version of Python, then..." - In that case you already have multiple versions of Python on your system, and by the definitions we've been working to here, you're an advanced user so you can fix it yourself. Obviously, that's too harsh a stance for us to take. But let's not ignore the fact that Unix users are typically *already* having to deal with multiple versions of Python, and are typically more familiar with command lines than Windows users. So there's no guarantee that a solution that works for them is usable for Windows users who've never used Python or the command line before, who downloaded Python 2.7 and installed it, were then told "no, you should be using Python 3", and installed 3.7, and now don't understand why "upgrading" didn't uninstall the old version, what they need to do to work with Python 3, etc... > One of the nice things about embedding the behaviour into the launcher > is that if the user/admin has installed a python that includes it it > provides the behaviour for python installations that were already > present and discoverable even if they didn't have it so if you have > python 2.5 then after you install something with py.exe you can, > currently, use py -2.5 to run it, (or py -2 if it is the only python 2). +1. This is the biggest benefit for me. Also the fact that all the older Python versions *don't need to be on PATH*. Having multiple versions of Python on PATH is a bad thing (because of the Scripts directory - if you want, I can explain in detail, but it's a side issue for now). >> I'm not willing to contradict Steve on this one. >> > > I didn't know he'd made a clear statement about it. > > I am not sure that I spelt it out well enough but my personal feeling is > that the installer should, (and I am reasonably sure that it should > capable of this), default to add to path IF there is not a currently > available python, (discoverable), and either default to don't add or > prompt the installer if there is. My apologies - I was unclear. I meant Steve Dower, who maintains the installer, and who has strongly stated that the way the Windows PATH works (separate system and user parts) makes sensible "add to PATH" behaviour impossible. The reasons have all been described multiple times, and I won't go into them again. Search the archives if you want to see the details. Suffice it to say that wishing it was easy to "just add Python to PATH" doesn't make it so... > Of course the whole add to path or not becomes moot if the py.exe > launcher is present as it is installed on the path, (C:\Windows), and > then tries very hard just to "do the right thing". That's why I prefer this option. There are still issues to address (such as the Scripts directory), but it's much easier to keep the complexity manageable with this approach. >> But if people that understand these things say it's a bad idea, then >> it's a bad idea. But we SHOULD make sure we are considering not just >> what could go wrong, or how often it could go wrong, but who it is going >> to go wrong for... >> >> i.e. making it "do the right thing" for newbies is much more important >> that making sure it doesn't "do the wrong thing" for more experienced >> folks that can uncheck a box, or clean up their PATH after the fact.... >> >> There are too many >> not-uncommon cases that we could mess up badly here. It's the right >> choice for "make new users' experience as easy as possible", but if we >> do this at the cost of making the experience of people upgrading to >> 3.7 (possibly by installing 3.7 to gradually switch over, >> >> >> then you uncheck the box -- and if someone is installing 3.7 to test it >> out, they SHOULD be a experience enough to uncheck a box... But I (as an advanced user) don't *want* to uncheck the box. I *do* want 3.7 to be my default Python. But if I previously had 2.7 as my default, and that's in the *system* part of PATH, then checking the box *doesn't work* - Python 2.7 is still my default. OF COURSE I have enough knowledge to fix the issue. That's not my point. My point is, do we want experienced Python users' first experience with Python 3.7 to be "the installer didn't work right", or "installing 3.7 broke my system, and I had to fix it manually", or something similar? Experienced Python users on Windows have been remarkably tolerant over recent years of our continual changing of where we install Python, whether we add to PATH by default, etc. At some point we need to stop, think things through, and get it right once and for all. Or at least agree to stick with what we decide for longer than a couple of releases. >> We're not doing them >> any favours by adding another new behaviour. >> >> >> I'm not trying to do them any favors -- I'm trying to do the favors for >> the newbies.. and YES, at the expense of people with "tweaked, and >> likely fragile, setup." OK. Maybe "not doing them any favours" is not as well known a phrase as I thought, sorry. What I meant was we're repeatedly penalising such users and we need to stop. We don't need to cater for their specific environments (they know how to do that themselves, and have done so). But we do need to stop breaking their setup every release. >> We need *another* solution here. Not just variations on the existing >> mess. That's why I like Steve's suggestion of making the launcher into >> the canonical entry point. >> >> >> only if it's named "python" (and python3) -- what we DON'T need is a >> different way to do it on Windows -- that's what we tried, and I thin >> it's failed. "python" yes. Not so much "python3", which I don't believe is a solution anyone *likes* - it's just an expedient for a specific Unix issue. (If you don't believe me, there's plenty of issues on the pip tracker that boil down to Unix users getting confused over which Python version they are executing, so certainly "python3" isn't a perfect solution, or even a particularly good one IMO). Paul From storchaka at gmail.com Tue Nov 21 04:44:33 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 21 Nov 2017 11:44:33 +0200 Subject: [Python-ideas] Rewriting the "roundrobin" recipe in the itertools documentation In-Reply-To: References: Message-ID: The roundrobin() implementation in recipes has quadratic time for large number of iterables. As well as all other proposed implementations. This is a problem if you use it with hundreds or thousands of iterables. For example: list(roundrobin(*([[1]]*1000))) next(roundrobin(*([[]]*1000 + [[1]]]))) The following implementation has linear time. def roundrobin(*iterables): "roundrobin('ABC', 'D', 'EF') --> A D E B F C" nexts = [iter(it).__next__ for it in iterables] while nexts: next_nexts = [] append = next_nexts.append for next in nexts: try: yield next() except StopIteration: pass else: append(next) nexts = next_nexts Actually it expands cycle() in Python. And this makes it slower for smaller number of iterations. From p.f.moore at gmail.com Tue Nov 21 04:47:16 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 21 Nov 2017 09:47:16 +0000 Subject: [Python-ideas] Should Python have user-defined constants? In-Reply-To: <29DDA3B1-8ED1-46C6-BF4E-DBD4DE9240C3@icloud.com> References: <29DDA3B1-8ED1-46C6-BF4E-DBD4DE9240C3@icloud.com> Message-ID: -1. I don't see how this would improve any programs I've written or seen. Tools like mypy or linters might benefit from a feature to track constants and ensure they don't get changed, but I don't think it's needed as a language feature. Seriously, has anyone ever written "math.pi = 5" and been surprised that their code broke? Or even modified an application constant like BUFFER_LIMIT? Do you have any evidence that this would avoid bugs in real-world code? Paul PS Please fix your font - your posts are coming through with a huge font size for some reason, which makes them extremely difficult to read. On 21 November 2017 at 06:33, Saeed Baig wrote: > Hey guys I am thinking of perhaps writing a PEP to introduce user-defined > constants to Python. Something along the lines of Swift?s ?let? syntax (e.g. > ?let pi = 3.14?). > > Do you guys think it would be a good idea? Why or why not? Do you think > there?s a better way to do it? I?d like to know what others think about this > idea before making any formal submission (I?ve already posted this same > question on python-list, but I just wanted to gauge opinion here too). > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From storchaka at gmail.com Tue Nov 21 04:54:25 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 21 Nov 2017 11:54:25 +0200 Subject: [Python-ideas] Star assignment in iterator way? In-Reply-To: References: Message-ID: 21.11.17 11:27, Kirill Balunov ????: > Your implementation iterates seq multiple times. But iterable > unpacking syntax works with an arbitrary iterable, and iterates it > only once. > > > Oh sorry, I know that my implementation iterates seq multiple times, I > only provide this to show the idea. It can be much optimized at C level. > I just want to understand if it's worth the time and effort. You can implement the first case, but for other cases you will need a storage for saving intermediate items. And using a list is a good option. > And you already have mentioned a question about mutable sequence. > > If these conditions and restrictions suit you, you can use your > good_star_exp() in your code or share it with others. But the > semantic of iterable unpacking can't be changed. > > > And how do you look at something like this (deferred star evaluation)?: > > a, ?*b, c, d = something_iterable This will be not different from a, *b, c, d = something_iterable b = iter(b) There is nothing deferred here. The only possible benefit can be in the case a, b, ?*c = something_iterable But I have doubts that this special case deserves introducing a new syntax. From storchaka at gmail.com Tue Nov 21 05:03:17 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 21 Nov 2017 12:03:17 +0200 Subject: [Python-ideas] Should Python have user-defined constants? In-Reply-To: <742BABA7-C73A-490F-96EB-B65E44EAC236@icloud.com> References: <742BABA7-C73A-490F-96EB-B65E44EAC236@icloud.com> Message-ID: 21.11.17 11:34, Saeed Baig ????: > Serhiy asked, in relation to constants,??To do what? What problem do you > need to solve??. No problem in particular, to be honest. I just thought > they?d be nice?since they?d increase confidence that my > variables-intended-to-be-constants wouldn?t get reassigned, and just to > increase readability/explicitness to other programmers. For increasing readability/explicitness you can use a name convention (UPPER_CASE), comments and documentation. As for increasing confidence that your variables-intended-to-be-constants wouldn?t get reassigned, you can use read-only properties. This is a tricky to implement read-only properties for modules, but see PEP 562 which should help to do this. Please don't use GIGANTIC font in your messages. And please don't create a new thread when answer. And *please* don't include unrelated text in your messages. From storchaka at gmail.com Tue Nov 21 05:06:15 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 21 Nov 2017 12:06:15 +0200 Subject: [Python-ideas] Should Python have user-defined constants? In-Reply-To: References: <29DDA3B1-8ED1-46C6-BF4E-DBD4DE9240C3@icloud.com> Message-ID: 21.11.17 11:47, Paul Moore ????: > -1. I don't see how this would improve any programs I've written or > seen. Tools like mypy or linters might benefit from a feature to track > constants and ensure they don't get changed, but I don't think it's > needed as a language feature. Seriously, has anyone ever written > "math.pi = 5" and been surprised that their code broke? Or even > modified an application constant like BUFFER_LIMIT? Do you have any > evidence that this would avoid bugs in real-world code? Actually the possibility of modifying application constants like BUFFER_LIMIT can be useful. From sf at fermigier.com Tue Nov 21 05:40:14 2017 From: sf at fermigier.com (=?UTF-8?Q?St=C3=A9fane_Fermigier?=) Date: Tue, 21 Nov 2017 11:40:14 +0100 Subject: [Python-ideas] Should Python have user-defined constants? In-Reply-To: References: <742BABA7-C73A-490F-96EB-B65E44EAC236@icloud.com> Message-ID: On Tue, Nov 21, 2017 at 11:03 AM, Serhiy Storchaka wrote: > 21.11.17 11:34, Saeed Baig ????: > >> Serhiy asked, in relation to constants, ?To do what? What problem do you >> need to solve??. No problem in particular, to be honest. I just thought >> they?d be nice since they?d increase confidence that my >> variables-intended-to-be-constants wouldn?t get reassigned, and just to >> increase readability/explicitness to other programmers. >> > > For increasing readability/explicitness you can use a name convention > (UPPER_CASE), comments and documentation. > UPPER_CASE variable names are a reasonable convention for global constants. I wouldn't like to read something like: for i in range(0, 100): FOO = f(i) BAR = g(i, FOO) do_something_with(BAR) but rather: for i in range(0, 100): const foo = f(i) const bar = g(i, foo) do_something_with(bar) or: for i in range(0, 100): let foo = f(i) let bar = g(i, foo) do_something_with(bar) If we move on with this idea (which is a long shot of course), choosing between 'let' and 'const' is a matter of taste and being consistent with other languages. As for increasing confidence that your variables-intended-to-be-constants > wouldn?t get reassigned, you can use read-only properties. This is a tricky > to implement read-only properties for modules, but see PEP 562 which should > help to do this. > > Please don't use GIGANTIC font in your messages. > +1. S. > > And please don't create a new thread when answer. > > And *please* don't include unrelated text in your messages. > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier - http://linkedin.com/in/sfermigier Founder & CEO, Abilian - Enterprise Social Software - http://www.abilian.com/ Chairman, Free&OSS Group / Systematic Cluster - http://www.gt-logiciel-libre.org/ Co-Chairman, National Council for Free & Open Source Software (CNLL) - http://cnll.fr/ Founder & Organiser, PyData Paris - http://pydata.fr/ --- ?You never change things by ?ghting the existing reality. To change something, build a new model that makes the existing model obsolete.? ? R. Buckminster Fuller -------------- next part -------------- An HTML attachment was scrubbed... URL: From levkivskyi at gmail.com Tue Nov 21 05:41:14 2017 From: levkivskyi at gmail.com (Ivan Levkivskyi) Date: Tue, 21 Nov 2017 11:41:14 +0100 Subject: [Python-ideas] Should Python have user-defined constants? In-Reply-To: References: <29DDA3B1-8ED1-46C6-BF4E-DBD4DE9240C3@icloud.com> Message-ID: On 21 November 2017 at 10:47, Paul Moore wrote: > -1. I don't see how this would improve any programs I've written or > seen. Tools like mypy or linters might benefit from a feature to track > constants and ensure they don't get changed > It is actually likely that something like this will appear in ``typing``: from typing import Final, List x: Final = 42 x = 1 # Fails type check lst: Final[List[int]] = [] lst.append(5) # OK lst = [1, 2, 3] # Fails type check -- Ivan -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Tue Nov 21 05:44:38 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 21 Nov 2017 12:44:38 +0200 Subject: [Python-ideas] Rewriting the "roundrobin" recipe in the itertools documentation In-Reply-To: References: Message-ID: 21.11.17 11:44, Serhiy Storchaka ????: > The roundrobin() implementation in recipes has quadratic time for large > number of iterables. As well as all other proposed implementations. This > is a problem if you use it with hundreds or thousands of iterables. For > example: > > ??? list(roundrobin(*([[1]]*1000))) > ??? next(roundrobin(*([[]]*1000 + [[1]]]))) > > The following implementation has linear time. > > def roundrobin(*iterables): > ??? "roundrobin('ABC', 'D', 'EF') --> A D E B F C" > ??? nexts = [iter(it).__next__ for it in iterables] > ??? while nexts: > ??????? next_nexts = [] > ??????? append = next_nexts.append > ??????? for next in nexts: > ??????????? try: > ??????????????? yield next() > ??????????? except StopIteration: > ??????????????? pass > ??????????? else: > ??????????????? append(next) > ??????? nexts = next_nexts > > Actually it expands cycle() in Python. And this makes it slower for > smaller number of iterations. Yet one natural implementation with linear time is: from collections import deque def roundrobin(*iterables): "roundrobin('ABC', 'D', 'EF') --> A D E B F C" nexts = deque(iter(it).__next__ for it in iterables) popleft = nexts.popleft append = nexts.append while nexts: next = popleft() try: yield next() except StopIteration: pass else: append(next) As the previous example it doesn't have any relation to itertools. From steve at pearwood.info Tue Nov 21 05:53:38 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 21 Nov 2017 21:53:38 +1100 Subject: [Python-ideas] Star assignment in iterator way? In-Reply-To: References: Message-ID: <20171121105338.GL19802@ando.pearwood.info> On Tue, Nov 21, 2017 at 12:27:32PM +0300, Kirill Balunov wrote: > Backward compatibility is an important issue, but at the same time it is > the main brake on progress. "Progress just means bad things happen faster." -- Terry Pratchett, "Witches Abroad" [...] > And how do you look at something like this (deferred star evaluation)?: > > a, ?*b, c, d = something_iterable A waste of effort? How do you defer evaluating the second and subsequent items if you evaluate the final two? Given: def gen(): yield 999 for i in range(100): yield random.random() yield 999 yield 999 then a, ?*b, c, d = gen() has to evaluate all 100 random numbers in order to assign a, c, d all equal to 999. Making b an iterator instead of a list doesn't actually avoid evaluating anything, and it will still require as much storage as a list. The most likely implementation would: - store the evaluated items in a list; - assign iter(the list) as b. I suppose that there could be some way of delaying the calls to random.random() by returning a thunk, but that is likely to be more expensive in both memory and time than a simple list of floats. -- Steven From clint.hepner at gmail.com Tue Nov 21 06:08:50 2017 From: clint.hepner at gmail.com (Clint Hepner) Date: Tue, 21 Nov 2017 06:08:50 -0500 Subject: [Python-ideas] Should Python have user-defined constants? In-Reply-To: References: <742BABA7-C73A-490F-96EB-B65E44EAC236@icloud.com> Message-ID: <83DCA5E0-8749-4B38-BD01-EE647053AE08@gmail.com> > On Nov 21, 2017, at 5:40 AM, St?fane Fermigier wrote: > > for i in range(0, 100): > const foo = f(i) > const bar = g(i, foo) > do_something_with(bar) This wouldn?t work, since a for loop doesn?t introduce a new scope for variables, and allowing a constant to be rebound in the same scope would somewhat defeat the purpose. From elazarg at gmail.com Tue Nov 21 06:10:46 2017 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Tue, 21 Nov 2017 11:10:46 +0000 Subject: [Python-ideas] Should Python have user-defined constants? In-Reply-To: References: <29DDA3B1-8ED1-46C6-BF4E-DBD4DE9240C3@icloud.com> Message-ID: Note that no other feature in typing is about the reference - everything is about the objects themselves. Final makes less sense as a general type. We can't enforce A[T]().foo() not to reassign a Final T. Elazar ?????? ??? ??, 21 ????? 2017, 12:42, ??? Ivan Levkivskyi ?< levkivskyi at gmail.com>: > On 21 November 2017 at 10:47, Paul Moore wrote: > >> -1. I don't see how this would improve any programs I've written or >> seen. Tools like mypy or linters might benefit from a feature to track >> constants and ensure they don't get changed >> > > It is actually likely that something like this will appear in ``typing``: > > from typing import Final, List > > x: Final = 42 > x = 1 # Fails type check > > lst: Final[List[int]] = [] > lst.append(5) # OK > lst = [1, 2, 3] # Fails type check > > -- > Ivan > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sf at fermigier.com Tue Nov 21 06:12:44 2017 From: sf at fermigier.com (=?UTF-8?Q?St=C3=A9fane_Fermigier?=) Date: Tue, 21 Nov 2017 12:12:44 +0100 Subject: [Python-ideas] Should Python have user-defined constants? In-Reply-To: References: <29DDA3B1-8ED1-46C6-BF4E-DBD4DE9240C3@icloud.com> Message-ID: That's one way to do it with no changes to the language, though syntaxically I find it lacking a bit of elegance (maybe a matter of getting accustomed with it?). Also, I'm not sure "Final" really conveys what it means (at first glance, I thought it was about immutability, not constantness). Maybe "Const" would be better in this context ? (Or maybe you've discussed this question already and come to the conclusion that "Final" is better for some reason?) S. On Tue, Nov 21, 2017 at 11:41 AM, Ivan Levkivskyi wrote: > On 21 November 2017 at 10:47, Paul Moore wrote: > >> -1. I don't see how this would improve any programs I've written or >> seen. Tools like mypy or linters might benefit from a feature to track >> constants and ensure they don't get changed >> > > It is actually likely that something like this will appear in ``typing``: > > from typing import Final, List > > x: Final = 42 > x = 1 # Fails type check > > lst: Final[List[int]] = [] > lst.append(5) # OK > lst = [1, 2, 3] # Fails type check > > -- > Ivan > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier - http://linkedin.com/in/sfermigier Founder & CEO, Abilian - Enterprise Social Software - http://www.abilian.com/ Chairman, Free&OSS Group / Systematic Cluster - http://www.gt-logiciel-libre.org/ Co-Chairman, National Council for Free & Open Source Software (CNLL) - http://cnll.fr/ Founder & Organiser, PyData Paris - http://pydata.fr/ --- ?You never change things by ?ghting the existing reality. To change something, build a new model that makes the existing model obsolete.? ? R. Buckminster Fuller -------------- next part -------------- An HTML attachment was scrubbed... URL: From levkivskyi at gmail.com Tue Nov 21 06:17:38 2017 From: levkivskyi at gmail.com (Ivan Levkivskyi) Date: Tue, 21 Nov 2017 12:17:38 +0100 Subject: [Python-ideas] Should Python have user-defined constants? In-Reply-To: References: <29DDA3B1-8ED1-46C6-BF4E-DBD4DE9240C3@icloud.com> Message-ID: On 21 November 2017 at 12:12, St?fane Fermigier wrote: > That's one way to do it with no changes to the language, though > syntaxically I find it lacking a bit of elegance (maybe a matter of getting > accustomed with it?). > > Also, I'm not sure "Final" really conveys what it means (at first glance, > I thought it was about immutability, not constantness). > > Maybe "Const" would be better in this context ? (Or maybe you've discussed > this question already and come to the conclusion that "Final" is better for > some reason?) > It is not set in stone, but it looks like most people like Final (although the initial proposal was Const, see https://github.com/python/mypy/issues/1214) -- Ivan -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Tue Nov 21 06:31:01 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 21 Nov 2017 22:31:01 +1100 Subject: [Python-ideas] Should Python have user-defined constants? In-Reply-To: <742BABA7-C73A-490F-96EB-B65E44EAC236@icloud.com> References: <742BABA7-C73A-490F-96EB-B65E44EAC236@icloud.com> Message-ID: <20171121113101.GM19802@ando.pearwood.info> On Tue, Nov 21, 2017 at 08:34:31PM +1100, Saeed Baig wrote: > > Hi. Just to respond to some peoples? points? > Bernardo asked 'This would be "constant" as in Java constants right? Referential > constants, so that if an object were mutable you would still be able to > change its internal state.? > Um, I?m not entirely sure what you mean, since I?ve never used Java. > But if it means what I think it means, I would be INCLINED to say no. > If, for example, I saw code like this: > > x = [9, 5, 7, 1] > let y = x > y[0] = 42 > print(y) > > ? I would expect that code to raise an error, not print ?[42, 5, 7, > 1]?, since you?re trying to modify the ?constant? y in place, which? > well, would kind of go against the idea of y being a constant, I > thought (unless I?m misunderstanding what you meant by ?object", since > everything in Python?s an object). That depends on what we mean by constant. If you mean that the *values* are immutable and cannot be changed, then Python already has constants. The object 99 (an int) is always equal to 99, you cannot change its value. Operations on it return a new object, not the same one modified. The same applies to strings and tuples. For example, consider the difference between a *mutable* object like a list, and an immutable one like a tuple: a = b = [1, 2] a += [999] print(b) # prints [1, 2, 999] Clearly the value [1, 2] is not "constant" in the sense of being immutable, unlike tuples: a = b = (1, 2) a += (999,) print(b) # prints (1, 2) So if that's what you mean by "constant", then most Python values (strings, ints, floats, bools, tuples, frozensets...) are already constants -- and the ones which aren't, we *want* to be mutable. But what Bernardo refers to, "referential constants", are more like constants in languages like C, Pascal, Java, etc. The value itself may or may not be mutable, or immutable, but the *name binding* is fixed. That is, once you have defined a constant, e.g. in Pascal you use a const declaration: const a = 1; in other languages you may use a "let": let a = 1 a = 2 # an error from this point on, the variable name "a" always refers to the same value. In this case, that value will be 1, which is an immutable object but it could be a mutable object like a list: let b = [] b.append(999) # modifies the object # but the name still refers to the same object b = [1, 2, 3] # an error The language we might choose here is to say that constants are "bind-once". That is, we can bind an object to the name once: let c = some(expression) but any subsequent attempts to re-bind or unbind the name should fail: c = something_else # fails del c # also fails That is, I think, what Bernardo means by referential constants, and that's what I mean when I talk about constants in Python. The question of mutability and immutability is orthogonal to the concept of being able to re-bind a name. They refer to two different concepts: - if the value is a fixed object, unable to be modified, we call the object "immutable"; otherwise the object is mutable; - if the *name* is fixed to the same object, unable to be changed to a different object, we might call the *name* a constant; otherwise the name is a variable. Because they are independent concepts, a language might support any combination of: - constant name with immutable value; - constant name with mutable value; - variable name with immutable value; - variable name with mutable value. Constants in Pascal and C are both constant *names* and immutable *values*. The closest they have to constant names with mutable values would be if you set a constant to a pointer (if the compiler supports that). In Pascal, if it were allowed, that might look something like: const a = ^x; {a is set to the address of x} In C, it might be something similar to: const int *a = &x; (I think). In either case, if the compiler supported this, you wouldn't be able to modify the pointer a itself, but you could modify the area of memory that a points to (that is, x). I don't think there is anything interesting about "constants" which are nothing more than immutable objects. We already have that. The interesting (but is it useful?) concept is constant identifiers which cannot be re-bound or re-assigned once they are set the first time. P.S. Please do not reply to digests without trimming all the excessive quoted emails that are not relevant to the discussion. -- Steve From steve at pearwood.info Tue Nov 21 06:33:49 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 21 Nov 2017 22:33:49 +1100 Subject: [Python-ideas] Should Python have user-defined constants? In-Reply-To: References: <742BABA7-C73A-490F-96EB-B65E44EAC236@icloud.com> Message-ID: <20171121113349.GN19802@ando.pearwood.info> On Tue, Nov 21, 2017 at 11:40:14AM +0100, St?fane Fermigier wrote: > for i in range(0, 100): > const foo = f(i) > const bar = g(i, foo) > do_something_with(bar) I wouldn't expect that to work. You are binding some value f(0) to the constant name "foo" on the first loop, then on the second loop you try to rebind a new value to the same name "foo". I would expect that to be an error. -- Steve From turnbull.stephen.fw at u.tsukuba.ac.jp Tue Nov 21 06:47:52 2017 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Tue, 21 Nov 2017 20:47:52 +0900 Subject: [Python-ideas] Adding a thin wrapper class around the functions in stdlib.heapq In-Reply-To: References: Message-ID: <23060.4712.569411.970796@turnbull.sk.tsukuba.ac.jp> bunslow writes: > Perhaps such repetition is a sign that *something* needs to be > done... It's not. Most such repetition is due to new people who have not read the last 20 years of archives. :-) (In case the smiley isn't clear enough, parse that as "all the new people who aren't cray-cray".) That doesn't mean there's nothing to the idea, it just means that *mere repetition* of certain proposals is only a sign that "great[sic] minds think alike," while Python development is attracting new people at a high rate. Repetition of certain arguments *in the same thread*, on the other hand, is often a sign that a meeting of minds has not be reached, and that can be symptomatic of problems (though it's not necessarily an indication of the necessity of a proposal). Regards, Steve From kirillbalunov at gmail.com Tue Nov 21 06:53:07 2017 From: kirillbalunov at gmail.com (Kirill Balunov) Date: Tue, 21 Nov 2017 14:53:07 +0300 Subject: [Python-ideas] Star assignment in iterator way? In-Reply-To: <20171121105338.GL19802@ando.pearwood.info> References: <20171121105338.GL19802@ando.pearwood.info> Message-ID: May be the first thing which I should do is to improve my English:) My main point was that in many cases (in my experience) it is a waste of memory to store entire list for star variable (*b) instead of some kind of iterator or deferred evaluation. How do you defer evaluating the second and subsequent items if you > evaluate the final two? Given: > > def gen(): > yield 999 > for i in range(100): > yield random.random() > yield 999 > yield 999 > > then > > a, ?*b, c, d = gen() If I can not copy at Python level, I can 'tee' when 'star_pos' is reached. In my usual practice, the main use that I encounter when see an assignment to a star variable is as a storage which is used only if the other vars match some criterion. With kind regards, -gdg -------------- next part -------------- An HTML attachment was scrubbed... URL: From turnbull.stephen.fw at u.tsukuba.ac.jp Tue Nov 21 06:55:33 2017 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Tue, 21 Nov 2017 20:55:33 +0900 Subject: [Python-ideas] Allow additional separator character in variables In-Reply-To: References: <23059.34466.1253.737550@turnbull.sk.tsukuba.ac.jp> Message-ID: <23060.5173.899290.61724@turnbull.sk.tsukuba.ac.jp> Mikhail V writes: > BTW, as per Serhiy Storchaka's note: > > my?variable > my%G???%@variable > my%G? ?%@variable > my%G???%@variable > my%G???%@variable > my%G???%@variable > > ^ Is this good idea *for Python*? Given that 5 of 6 show up with the glyph for U+FFFD REPLACEMENT CHARACTER in my client, I'd say not (but then, I can always fix my mail client so don't mind me ;-). > I don't know how it is possible. Looks like a result of some > unlucky nuclear experment. Might be it will not cause any possible > confusion, or less than a hyphen and a minus. It depends on how familiar people and tools are with Unicode. For example, after almost clicking on something from "Apple.co.jp" where the "A" is from the Cyrillic block, my mail program now highlights confusables (there's a list at Unicode.org) and also places where languages I don't use are present in the message preview. It really helps in detecting spam! But most people neither have that knowledge, nor the source code to their mail clients and browsers, to help them with those distinctions. Personally, I think that Python probably should ban non-ASCII non-letter characters in identifiers and whitespace, and maybe add them later in response to requests from native speakers of the relevant languages. I don't know how easy that would be to do, though, since I think the rule is already that identifiers must be composed only of letters, numbers, and ASCII "_". Since Serhiy's examples are valid, we'd have to rule them out explicitly, rather than by reference to the Unicode database. Yuck. > If the new language thing would happen and gained popularity - it > would be the worst scenario - competing syntaxes, CO2 emissions, > community splittage, etc. I don't endorse such ideas. Think of it as evolution in action. So, languages evolve whether you do it yourself or not. It's not a question of you endorsing the idea. Eventually somebody will write a better language than Python. Why not you? The problem, if it's similar to an existing one, is the work it creates for the author of the new language. The worst possible case is something like Python 3. IIUC, Guido's opinion now is that looking back, Python 3 was the right thing to do at the time but he's never gonna do that again, too much work on explaining "why Python 3". The question would be, is it right for *you* to do it for a language with your favorite features? I don't say you *should*, just that you *could*. Regards, Steve From brent.bejot at gmail.com Tue Nov 21 07:24:11 2017 From: brent.bejot at gmail.com (brent bejot) Date: Tue, 21 Nov 2017 07:24:11 -0500 Subject: [Python-ideas] Should Python have user-defined constants? In-Reply-To: <20171121113349.GN19802@ando.pearwood.info> References: <742BABA7-C73A-490F-96EB-B65E44EAC236@icloud.com> <20171121113349.GN19802@ando.pearwood.info> Message-ID: Are there any 3rd party tools that already implement this? By my understanding, type annotations started as 3rd party tools that became popular and so are now incorporated into the language proper. I could see a 'const' syntax making its way into the language that way, but not directly. Certainly this idea has been hashed out multiple times before. -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Tue Nov 21 07:35:08 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 21 Nov 2017 14:35:08 +0200 Subject: [Python-ideas] Star assignment in iterator way? In-Reply-To: References: <20171121105338.GL19802@ando.pearwood.info> Message-ID: 21.11.17 13:53, Kirill Balunov ????: > If I can not copy at Python level, I can 'tee' when 'star_pos' is reached. And tee() uses a real RAM for saving items. From stephanh42 at gmail.com Tue Nov 21 07:50:03 2017 From: stephanh42 at gmail.com (Stephan Houben) Date: Tue, 21 Nov 2017 13:50:03 +0100 Subject: [Python-ideas] Allow additional separator character in variables In-Reply-To: <23060.5173.899290.61724@turnbull.sk.tsukuba.ac.jp> References: <23059.34466.1253.737550@turnbull.sk.tsukuba.ac.jp> <23060.5173.899290.61724@turnbull.sk.tsukuba.ac.jp> Message-ID: 2017-11-21 12:55 GMT+01:00 Stephen J. Turnbull < turnbull.stephen.fw at u.tsukuba.ac.jp>: > Personally, I think that Python probably should ban non-ASCII > non-letter characters in identifiers and whitespace, and maybe add > them later in response to requests from native speakers of the > relevant languages. That would be quite a backward-incompatible change since such identifiers have been legal since Python 3.0. > I don't know how easy that would be to do, > though, since I think the rule is already that identifiers must be > composed only of letters, numbers, and ASCII "_". See: https://www.python.org/dev/peps/pep-313 The identifier syntax is *. ID_Start is defined as all characters having one of the general categories uppercase letters (Lu), lowercase letters (Ll), titlecase letters (Lt), modifier letters (Lm), other letters (Lo), letter numbers (Nl), the underscore, and characters carrying the Other_ID_Start property. XID_Start then closes this set under normalization, by removing all characters whose NFKC normalization is not of the form ID_Start ID_Continue* anymore. ID_Continue is defined as all characters in ID_Start, plus nonspacing marks (Mn), spacing combining marks (Mc), decimal number (Nd), connector punctuations (Pc), and characters carryig the Other_ID_Continue property. Again, XID_Continue closes this set under NFKC-normalization; it also adds U+00B7 to support Catalan. Since Serhiy's > examples are valid, we'd have to rule them out explicitly, rather than > by reference to the Unicode database. Yuck. > If we take this thinking to its logical extreme we should ban ASCII 1 and l since they can be confused. Also 0 and O. Realistically, this is extremely unlikely to be an issue in practice. If you have people making such malignant code changes with checkin permission, you have bigger problems... Anyway, you can have your linter enforce ASCII or whatever character subset you deem safe. Stephan -------------- next part -------------- An HTML attachment was scrubbed... URL: From kirillbalunov at gmail.com Tue Nov 21 08:22:09 2017 From: kirillbalunov at gmail.com (Kirill Balunov) Date: Tue, 21 Nov 2017 16:22:09 +0300 Subject: [Python-ideas] Should Python have user-defined constants? In-Reply-To: References: <29DDA3B1-8ED1-46C6-BF4E-DBD4DE9240C3@icloud.com> Message-ID: > > It is not set in stone, but it looks like most people like Final (although > the initial proposal was Const, see https://github.com/python/ > mypy/issues/1214) > Ivan, you mean this thread "Can't index named tuple by defined constant" ? With kind regards, -gdg -------------- next part -------------- An HTML attachment was scrubbed... URL: From levkivskyi at gmail.com Tue Nov 21 08:24:17 2017 From: levkivskyi at gmail.com (Ivan Levkivskyi) Date: Tue, 21 Nov 2017 14:24:17 +0100 Subject: [Python-ideas] Should Python have user-defined constants? In-Reply-To: References: <29DDA3B1-8ED1-46C6-BF4E-DBD4DE9240C3@icloud.com> Message-ID: On 21 November 2017 at 14:22, Kirill Balunov wrote: > It is not set in stone, but it looks like most people like Final (although >> the initial proposal was Const, see https://github.com/python/mypy >> /issues/1214) >> > > > Ivan, you mean this thread "Can't index named tuple by defined constant" > ? > Discussions about Final/Const happened in several threads on both typing and mypy trackers, I just don't remember them all. -- Ivan -------------- next part -------------- An HTML attachment was scrubbed... URL: From kirillbalunov at gmail.com Tue Nov 21 09:37:36 2017 From: kirillbalunov at gmail.com (Kirill Balunov) Date: Tue, 21 Nov 2017 17:37:36 +0300 Subject: [Python-ideas] Make bytes __repr__ and __str__ representation different? Message-ID: Currently, __repr__ and __str__ representation of bytes is the same. Perhaps it is worth making them different, this will make it easier to visually perceive them as a container of integers from 0 to 255, instead of a mixture of printable and non-printable ascii characters. It is proposed: a) __str__ - leave unchanged b) __repr__ - represent as sequence of escaped hex >>> a = bytes([42,43,44,45,46]) >>> a # Current b'*+-./' >>> a # Proposed b'\x2a\x2b\x2d\x2e\x2f' As you can see, the second example is more easily perceived as a sequence, in which '\' is also perceived as ',' in list or tuple. In addition, 2020 is close, it allows the new Pythonistas not to take them as an ascii mixture strings. With kind regards, -gdg -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Tue Nov 21 10:16:23 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 22 Nov 2017 02:16:23 +1100 Subject: [Python-ideas] Make bytes __repr__ and __str__ representation different? In-Reply-To: References: Message-ID: <20171121151623.GO19802@ando.pearwood.info> On Tue, Nov 21, 2017 at 05:37:36PM +0300, Kirill Balunov wrote: > Currently, __repr__ and __str__ representation of bytes is the same. > Perhaps it is worth making them different, this will make it easier to > visually perceive them as a container of integers from 0 to 255, > instead of a mixture of printable and non-printable ascii characters. It is > proposed: > > a) __str__ - leave unchanged > b) __repr__ - represent as sequence of escaped hex > > >>> a = bytes([42,43,44,45,46]) > >>> a # Current > b'*+-./' > >>> a # Proposed > b'\x2a\x2b\x2d\x2e\x2f' I'd rather leave __str__ and __repr__ alone. Changing them will have huge backwards compatibility implications. I'd rather give bytes a hexdump() method that returns a string: '2a 2b 2d 2e 2f' (possibly with optional arguments to specify the formatting). > As you can see, the second example is more easily perceived as a sequence, > in which '\' is also perceived as ',' in list or tuple. I disagree. And if you perceive \ as a separator, why does the sequence start with a separator? And why are there so many x characters? > In addition, 2020 > is close, it allows the new Pythonistas not to take them as an ascii > mixture strings. The special role of ASCII is far too important for us to ever completely discard it. -- Steve From victor.stinner at gmail.com Tue Nov 21 10:38:46 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 21 Nov 2017 16:38:46 +0100 Subject: [Python-ideas] Make bytes __repr__ and __str__ representation different? In-Reply-To: References: Message-ID: Hi, While it may shock you, using bytes for "text" makes sense in some areas. Please read the Motivation of the PEP 461: https://www.python.org/dev/peps/pep-0461/#motivation Victor 2017-11-21 15:37 GMT+01:00 Kirill Balunov : > Currently, __repr__ and __str__ representation of bytes is the same. Perhaps > it is worth making them different, this will make it easier to visually > perceive them as a container of integers from 0 to 255, > instead of a mixture of printable and non-printable ascii characters. It is > proposed: > > a) __str__ - leave unchanged > b) __repr__ - represent as sequence of escaped hex > >>>> a = bytes([42,43,44,45,46]) >>>> a # Current > b'*+-./' >>>> a # Proposed > b'\x2a\x2b\x2d\x2e\x2f' > > As you can see, the second example is more easily perceived as a sequence, > in which '\' is also perceived as ',' in list or tuple. In addition, 2020 is > close, it allows the new Pythonistas not to take them as an ascii mixture > strings. > > With kind regards, -gdg > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From k7hoven at gmail.com Tue Nov 21 10:46:02 2017 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Tue, 21 Nov 2017 17:46:02 +0200 Subject: [Python-ideas] Star assignment in iterator way? In-Reply-To: References: <20171121105338.GL19802@ando.pearwood.info> Message-ID: FWIW, here's something for working with memory-efficient sequences (and generators), which should get more features in the future: pip install git+https://github.com/k7hoven/views Some examples of what it does: py> from views import seq py> seq[::range(3), None, ::"abc", "Hi!"] py> seq[::range(100)] py> from views import seq, gen py> seq.chain([1, 2, 3], [4, 5, 6]) py> list(gen.chain([1, 2, 3], [4, 5, 6])) [1, 2, 3, 4, 5, 6] py> from views import range py> range(5) range(0, ..., 4) py> range(1, 10, 3) range(1, ..., 7, step=3) py> range(1, ..., 5) range(1, ..., 5) py> range(1, 3, ..., 10) range(1, ..., 9, step=2) Sequences are perhaps more interesting than the generators, which are just there, because I don't want to implicitly try to convert generators/iterators into sequences. I do intend to add at least one *explicit* mechanism. Much of this is thread-safe, but the assumption in general is that one does not modify the original sequences. One problem is that there's no way to efficiently check if the originals have been mutated. Currently it just sometimes checks that the lengths match. This approach can also be a big performance boost because it avoids copying stuff around in memory etc. However, many possible optimizations have not been implemented yet, so there's overhead that can be significant for small sequences. For instance, itertools could be used to optimize some features. ??Koos On Tue, Nov 21, 2017 at 2:35 PM, Serhiy Storchaka wrote: > 21.11.17 13:53, Kirill Balunov ????: > >> If I can not copy at Python level, I can 'tee' when 'star_pos' is reached. >> > > And tee() uses a real RAM for saving items. > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- + Koos Zevenhoven + http://twitter.com/k7hoven + -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Tue Nov 21 10:47:05 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 21 Nov 2017 16:47:05 +0100 Subject: [Python-ideas] Should Python have user-defined constants? In-Reply-To: <29DDA3B1-8ED1-46C6-BF4E-DBD4DE9240C3@icloud.com> References: <29DDA3B1-8ED1-46C6-BF4E-DBD4DE9240C3@icloud.com> Message-ID: 2017-11-21 7:33 GMT+01:00 Saeed Baig : > Hey guys I am thinking of perhaps writing a PEP to introduce user-defined > constants to Python. Something along the lines of Swift?s ?let? syntax (e.g. > ?let pi = 3.14?). If you want to work on a PEP, you will have to write a strong rationale for it :-) > Do you guys think it would be a good idea? Why or why not? Do you think > there?s a better way to do it? I?d like to know what others think about this > idea before making any formal submission (I?ve already posted this same > question on python-list, but I just wanted to gauge opinion here too). Python has different kinds of namespaces: module globals, class attributes, function local variables, etc. The https://github.com/fijal/quill programming language looks like Python but makes module globals *mapping* "immutable": setattr(module, 'var', new_value). Only the mapping is immutable, a value can be mutable. I guess that the motivation here is to help the optimizer to emit more efficient code. See also previous attempts: "PEP 416 -- Add a frozendict builtin type" https://www.python.org/dev/peps/pep-0416/ => my motivation was to develop a sandbox for Python "PEP 351 -- The freeze protocol" https://www.python.org/dev/peps/pep-0351/ => I guess that the main motivation was to previous programming mistakes, misuse of an API The question is if you only want to have a technical solution to prevent modification of module globals, or if you would like to advertize that a variable is constant and use it somehow. Victor From mertz at gnosis.cx Tue Nov 21 11:16:03 2017 From: mertz at gnosis.cx (David Mertz) Date: Tue, 21 Nov 2017 08:16:03 -0800 Subject: [Python-ideas] Should Python have user-defined constants? In-Reply-To: References: <29DDA3B1-8ED1-46C6-BF4E-DBD4DE9240C3@icloud.com> Message-ID: ISTM that using the already widely used convention of ALL_CAPS for constants is plenty for linters to warn about rebinding names. Python believes "we're all adults here" after all. So even though it's worth *warning* users if BUFFER_LIMIT gets redefined, there can be reasons between consenting adults for doing so when you know what you are doing. It feels similar to the fact that I *can* call: instance._Klass__secret() But the spelling suggests a strong recommendation to use a more public API. Exact same story with redefining an implied constant. On Tue, Nov 21, 2017 at 7:47 AM, Victor Stinner wrote: > 2017-11-21 7:33 GMT+01:00 Saeed Baig : > > Hey guys I am thinking of perhaps writing a PEP to introduce user-defined > > constants to Python. Something along the lines of Swift?s ?let? syntax > (e.g. > > ?let pi = 3.14?). > > If you want to work on a PEP, you will have to write a strong > rationale for it :-) > > > Do you guys think it would be a good idea? Why or why not? Do you think > > there?s a better way to do it? I?d like to know what others think about > this > > idea before making any formal submission (I?ve already posted this same > > question on python-list, but I just wanted to gauge opinion here too). > > Python has different kinds of namespaces: module globals, class > attributes, function local variables, etc. > > The https://github.com/fijal/quill programming language looks like > Python but makes module globals *mapping* "immutable": setattr(module, > 'var', new_value). Only the mapping is immutable, a value can be > mutable. I guess that the motivation here is to help the optimizer to > emit more efficient code. > > See also previous attempts: > > "PEP 416 -- Add a frozendict builtin type" > https://www.python.org/dev/peps/pep-0416/ > => my motivation was to develop a sandbox for Python > > "PEP 351 -- The freeze protocol" > https://www.python.org/dev/peps/pep-0351/ > => I guess that the main motivation was to previous programming > mistakes, misuse of an API > > The question is if you only want to have a technical solution to > prevent modification of module globals, or if you would like to > advertize that a variable is constant and use it somehow. > > Victor > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Tue Nov 21 12:22:14 2017 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 21 Nov 2017 09:22:14 -0800 Subject: [Python-ideas] Make bytes __repr__ and __str__ representation different? In-Reply-To: References: Message-ID: On Tue, Nov 21, 2017 at 6:37 AM, Kirill Balunov wrote: > Currently, __repr__ and __str__ representation of bytes is the same. > Perhaps it is worth making them different, this will make it easier to > visually perceive them as a container of integers from 0 to 255, > instead of a mixture of printable and non-printable ascii characters. It > is proposed: > > a) __str__ - leave unchanged > b) __repr__ - represent as sequence of escaped hex > > >>> a = bytes([42,43,44,45,46]) > >>> a # Current > b'*+-./' > >>> a # Proposed > b'\x2a\x2b\x2d\x2e\x2f' > supposedly __repr__ is supposed to give an eval-able version -- which your proposal is. But the way you did your example indicates that: bytes((42, 43, 44, 45, 46)) would be an even better __repr__, if the goal is to make it clear and easy that it is a "container of integers from 0 to 255" I've been programming since quite some time ago, and hex has NEVER come naturally to me :-) But backward compatibility and all that :-( -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Tue Nov 21 12:34:33 2017 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 21 Nov 2017 09:34:33 -0800 Subject: [Python-ideas] Should Python have user-defined constants? In-Reply-To: <20171121113101.GM19802@ando.pearwood.info> References: <742BABA7-C73A-490F-96EB-B65E44EAC236@icloud.com> <20171121113101.GM19802@ando.pearwood.info> Message-ID: On Tue, Nov 21, 2017 at 3:31 AM, Steven D'Aprano wrote: > The > interesting (but is it useful?) concept is constant identifiers which > cannot be re-bound or re-assigned once they are set the first time. > This would actually be a substantial shift in what Python is about -- currently the concept of type is all about values, not names -- there are various rules about scope, and they can be adjusted with global and nonlocal, but it's only about scope -- a name is still a name, and can be bound to any type of object, etc. static languages, on the other hand often (always) assign type to the name itself -- so adding constants and the like makes sense. But changing the property of a name (this name can not be rebound) is a big shift in what names are about in Python -- and I don't think a good one. And what's the use-case, really? beyond the use case for all sorts of static typing... in case it's not obvious: -1 -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From sf at fermigier.com Tue Nov 21 13:04:17 2017 From: sf at fermigier.com (=?UTF-8?Q?St=C3=A9fane_Fermigier?=) Date: Tue, 21 Nov 2017 19:04:17 +0100 Subject: [Python-ideas] Should Python have user-defined constants? In-Reply-To: References: <742BABA7-C73A-490F-96EB-B65E44EAC236@icloud.com> <20171121113101.GM19802@ando.pearwood.info> Message-ID: On Tue, Nov 21, 2017 at 6:34 PM, Chris Barker wrote: > > And what's the use-case, really? beyond the use case for all sorts of > static typing... > Javascript is dynamically typed, and has 'const' variable declaration which were added in ES6. Static typing is not the only use case. Discouraging reuse of variables of favouring functional style over imperative is a strong use case for 'const' in JS. But I agree that the scoping rules are different in Javascript, and this argument is much less convincing in Python. S. -- Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier - http://linkedin.com/in/sfermigier Founder & CEO, Abilian - Enterprise Social Software - http://www.abilian.com/ Chairman, Free&OSS Group / Systematic Cluster - http://www.gt-logiciel-libre.org/ Co-Chairman, National Council for Free & Open Source Software (CNLL) - http://cnll.fr/ Founder & Organiser, PyData Paris - http://pydata.fr/ --- ?You never change things by ?ghting the existing reality. To change something, build a new model that makes the existing model obsolete.? ? R. Buckminster Fuller -------------- next part -------------- An HTML attachment was scrubbed... URL: From elazarg at gmail.com Tue Nov 21 13:05:17 2017 From: elazarg at gmail.com (=?UTF-8?B?15DXnNei15bXqA==?=) Date: Tue, 21 Nov 2017 18:05:17 +0000 Subject: [Python-ideas] Should Python have user-defined constants? In-Reply-To: References: <742BABA7-C73A-490F-96EB-B65E44EAC236@icloud.com> <20171121113101.GM19802@ando.pearwood.info> Message-ID: ?????? ??? ??, 21 ????? 2017, 19:36, ??? Chris Barker ?< chris.barker at noaa.gov>: > ... > And what's the use-case, really? beyond the use case for all sorts of > static typing... > I don't understand the question. The use case was explained before - people want to have better ways to reason about their programs. Statically. Why dismiss it as a non-usecase? It's helpful for both tools and humans. It was mentioned that there are related conventions in Python. Good point. But they are not as strong as the "convention" that tuples and strings cannot change, and this is useful in reasoning about code - in understanding the program, essentially. When I read "final int x = 5;" in Java, I don't have to think about it anymore - it's 5. When I read "X = 5" in Python, it might be a constant, but it might also be a misnomer, or something that used to be a constant, or a class reassigned. I still have to watch whether it will be changed, intentionally or accidentally, by this or other module. It does not even communicate intention in an unambiguous way. In my opinion, the inability to control mutation in Python is the biggest readability issue of the language (thank god monkey-patching is discouraged). I see why the proposal can be rejected, but I don't understand the "you don't gain anything" responses. There are significant gains there. Elazar -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Tue Nov 21 13:13:01 2017 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 21 Nov 2017 13:13:01 -0500 Subject: [Python-ideas] Star assignment in iterator way? In-Reply-To: References: Message-ID: On 11/21/2017 3:54 AM, Kirill Balunov wrote: > Hello. Currently during star assignement the new list is created. > What was the idea to produce new list instead of returning an iterator? > It seems to me that returning an iterator more suited to the spirit of > Python 3. > There are three cases: > > 1. a,b,c,*d = something_iterable > 2. *a,b,c,d = something_iterable > 3. a,*b,c,d = something_iterable > > The first one is obvious. Right, and easily dealt with with current Python. d = iter(something iterable) a,b,c = islice(d, 3) (or 3 next(d) calls) More typical is to pull one item off the iterator with next(), as is optionally done with csv readers. it = iter(iterable) header = next(it) # such as column names for item in it: process(item) > For the rest two we always need to iterate > through > entire iterable to achieve values for b,c,d (or c,d) binding. But this > can be done > more memory effiecient than currently (may be I'm wrong). For the general case, there is no choice but to save in a list. -- Terry Jan Reedy From mikhailwas at gmail.com Tue Nov 21 13:19:17 2017 From: mikhailwas at gmail.com (Mikhail V) Date: Tue, 21 Nov 2017 19:19:17 +0100 Subject: [Python-ideas] Allow additional separator character in variables In-Reply-To: <23060.5173.899290.61724@turnbull.sk.tsukuba.ac.jp> References: <23059.34466.1253.737550@turnbull.sk.tsukuba.ac.jp> <23060.5173.899290.61724@turnbull.sk.tsukuba.ac.jp> Message-ID: Serhiy Storchaka wrote: > Yes, it causes less confusion that changing meaning of a minus. If those chars are not used at all, then yes :) And I don't recall I was exactly propsing changing meaning of minus > But the name ????????? doesn't cause any confusion if used in an > appropriate context (for example in a lesson for young Ukrainian > children). I believe the above dot- and hyphen-like characters don't > cause confusion if used as letters in an appropriate language context. A single word written in local language should not. But its a perfect way to make whole code look like a mess. I think it is very interesting experience to use Cyrillic letters, since many are identical to Latin. So it would not be programming lessons in the first place, but rather constant changing of keyboard layout, and then trying to find unexplainable errors. Mikhail From AlonSnir at hotmail.com Tue Nov 21 08:50:38 2017 From: AlonSnir at hotmail.com (Alon Snir) Date: Tue, 21 Nov 2017 13:50:38 +0000 Subject: [Python-ideas] Rewriting the "roundrobin" recipe in the itertools documentation Message-ID: def roundrobin(*iterables): "roundrobin('ABC', 'D', 'EF') --> A D E B F C" nexts = [ iter(it).__next__ for it in iterables ] i = 0 while nexts: i %= len(nexts) try: yield nexts[i]() except StopIteration: del nexts[i] else: i += 1 Regards Alon Snir From stephanh42 at gmail.com Tue Nov 21 15:03:25 2017 From: stephanh42 at gmail.com (Stephan Houben) Date: Tue, 21 Nov 2017 21:03:25 +0100 Subject: [Python-ideas] Allow additional separator character in variables In-Reply-To: References: <23059.34466.1253.737550@turnbull.sk.tsukuba.ac.jp> <23060.5173.899290.61724@turnbull.sk.tsukuba.ac.jp> Message-ID: Hi all, If anybody is still worried about this, here is a 29-line proof-of-concept code checker which warns if your source file contains identifiers which are different but look the same. https://gist.github.com/stephanh42/61eceadc2890cf1b53ada5e48ef98ad1 Stephan 2017-11-21 19:19 GMT+01:00 Mikhail V : > Serhiy Storchaka wrote: > > > Yes, it causes less confusion that changing meaning of a minus. > > If those chars are not used at all, then yes :) > And I don't recall I was exactly propsing changing meaning of minus > > > > But the name ????????? doesn't cause any confusion if used in an > > appropriate context (for example in a lesson for young Ukrainian > > children). I believe the above dot- and hyphen-like characters don't > > cause confusion if used as letters in an appropriate language context. > > A single word written in local language should not. But its a perfect way > to make whole code look like a mess. > I think it is very interesting experience to use Cyrillic letters, since > many are identical to Latin. So it would not be programming lessons > in the first place, but rather constant changing of keyboard layout, > and then trying to find unexplainable errors. > > > > Mikhail > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kirillbalunov at gmail.com Tue Nov 21 15:27:06 2017 From: kirillbalunov at gmail.com (Kirill Balunov) Date: Tue, 21 Nov 2017 23:27:06 +0300 Subject: [Python-ideas] Make bytes __repr__ and __str__ representation different? In-Reply-To: References: Message-ID: 2017-11-21 20:22 GMT+03:00 Chris Barker wrote: > But the way you did your example indicates that: > > bytes((42, 43, 44, 45, 46)) > > would be an even better __repr__, if the goal is to make it clear and easy > that it is a "container of integers from 0 to 255" > > I've been programming since quite some time ago, and hex has NEVER come > naturally to me :-) > Yes, it is better, but it seemed too radical to me:) 2017-11-21 18:16 GMT+03:00 Steven D'Aprano wrote: > I'd rather give bytes a > hexdump() method that returns a string: > > '2a 2b 2d 2e 2f' > > (possibly with optional arguments to specify the formatting). Since Python 3.5 bytes has a .hex() method, the same as yours .hexdump() but without spaces. But still it is a string. 2017-11-21 18:38 GMT+03:00 Victor Stinner : > > While it may shock you, using bytes for "text" makes sense in some > areas. Please read the Motivation of the PEP 461: > https://www.python.org/dev/peps/pep-0461/#motivation It does not, because it is really useful feature. But rather, ascii was made so that it would fit into a byte, and not vice versa. Nevertheless, bytes are the strangest object in Python. It looks like a string (which contains only ascii), but it is not a string, because if you index, it does not return a byte -> bytes (b'123 '[0])! = Bytes (b'1'). May be it is closer to tuple, it is also immutable, but bytes(3) creates a sequence b '\ x00 \ x00 \ x00', but tuple not (and what the hell is b'\x00\x00\x00'?). Maybe it has some relationship to integers but int(b'1') == 1 when bytes([int(49)]) == b'1', i.e with integers it is not a friend either. It is bytes... With kind regards, -gdg -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Tue Nov 21 15:59:12 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 21 Nov 2017 22:59:12 +0200 Subject: [Python-ideas] Allow additional separator character in variables In-Reply-To: References: <23059.34466.1253.737550@turnbull.sk.tsukuba.ac.jp> <23060.5173.899290.61724@turnbull.sk.tsukuba.ac.jp> Message-ID: 21.11.17 22:03, Stephan Houben ????: > If anybody is still worried about this, > here is a 29-line proof-of-concept code checker which warns if your > source file > contains identifiers which are different but look the same. > > https://gist.github.com/stephanh42/61eceadc2890cf1b53ada5e48ef98ad1 Ha! I wanted to look at a 29-line code that is able to detect all homoglyphs. But actually it just imports a third-party module. :-( In any case this is a nice code. From greg.ewing at canterbury.ac.nz Tue Nov 21 16:49:32 2017 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 22 Nov 2017 10:49:32 +1300 Subject: [Python-ideas] Should Python have user-defined constants? In-Reply-To: <20171121113349.GN19802@ando.pearwood.info> References: <742BABA7-C73A-490F-96EB-B65E44EAC236@icloud.com> <20171121113349.GN19802@ando.pearwood.info> Message-ID: <5A149F6C.5020107@canterbury.ac.nz> Steven D'Aprano wrote: >>for i in range(0, 100): >> const foo = f(i) >> const bar = g(i, foo) >> do_something_with(bar) > > You are binding some value f(0) to the > constant name "foo" on the first loop, then on the second loop you try > to rebind a new value to the same name "foo". It could be made to kind of work if you were allowed to re-execute the same let statement, but not assign to the name with another statement in the same function. -- Greg From greg.ewing at canterbury.ac.nz Tue Nov 21 16:53:59 2017 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 22 Nov 2017 10:53:59 +1300 Subject: [Python-ideas] Should Python have user-defined constants? In-Reply-To: <20171121113101.GM19802@ando.pearwood.info> References: <742BABA7-C73A-490F-96EB-B65E44EAC236@icloud.com> <20171121113101.GM19802@ando.pearwood.info> Message-ID: <5A14A077.4060709@canterbury.ac.nz> Steven D'Aprano wrote: > In C, it might be something similar to: > > const int *a = &x; Not quite, that makes whatever a points to constant, not a itself. But you can do this: typedef int *ip; int x, y; const ip p = &x; void f(void) { *p = 42; /* ok */ p = &y; /* not ok */ } Compiling this gives: % gcc -c constptr.c constptr.c: In function ?f?: constptr.c:7: error: assignment of read-only variable ?p? BTW, this shows that gcc thinks of const-declared things as "read-only variables" rather than constants. Which is a bit of a silly term when you think about it! -- Greg From hugo.fisher at gmail.com Tue Nov 21 16:54:41 2017 From: hugo.fisher at gmail.com (Hugh Fisher) Date: Wed, 22 Nov 2017 08:54:41 +1100 Subject: [Python-ideas] Should Python have user-defined constants? Message-ID: > Message: 4 > Date: Tue, 21 Nov 2017 18:05:17 +0000 > From: ????? > To: Chris Barker > Cc: Python-Ideas > Subject: Re: [Python-ideas] Should Python have user-defined constants? [munch] > > It was mentioned that there are related conventions in Python. Good point. > But they are not as strong as the "convention" that tuples and strings > cannot change, and this is useful in reasoning about code - in > understanding the program, essentially. > > When I read "final int x = 5;" in Java, I don't have to think about it > anymore - it's 5. When I read "X = 5" in Python, it might be a constant, > but it might also be a misnomer, or something that used to be a constant, > or a class reassigned. I still have to watch whether it will be changed, > intentionally or accidentally, by this or other module. It does not even > communicate intention in an unambiguous way. Here I strongly disagree. The uppercase convention for things you really should not be assigning to has been in Python for ages and well understood. math.pi is the usual example brought up by people who want const/final, perhaps as a glaring exception to the rule. (Can math.PI be added to the standard library?) I think that adding const/let/final to Python would decrease readability and the ability to reason about the code for me, because now I would be reading foo += 1 and I could not easily decide whether this was wrong or not, because it might have been declared as immutable. Textual conventions are popular in many programming languages, not just Python, because they are easier to remember than individual type/attribute declarations. > In my opinion, the inability to control mutation in Python is the biggest > readability issue of the language (thank god monkey-patching is > discouraged). I think this is a separate issue to const/let/final. (And have you read PEP 351?) I don't miss simple scalar const/#define values in Python. In C/Java/Ada/... they are necessary to define array bounds, but everything in Python is dynamic and you can ask an array or whatever how big it is. In C/Ada/C#/... const protects again accidentally passing a value by reference/address and modifying it, but Python doesn't have pass by reference. No pointer aliasing in Python... As for monkey patching, or even just math.pi = 3.0 that is being unable to make dictionaries read-only. I agree, I get frustrated when I've accidentally misspelled an attribute name and Python just creates a new instance value instead of reporting an error. But this kind of dynamic object creation is so fundamental to how Python works, and also the source of so much power and flexibility, that I think it is worth living with the occasional glitch. I would not want to see Python allow module or class writers to declare "my code is perfect and not to be changed". But I would not mind if I could write code "I should not be changing this module/class/object , raise an exception if I do". -- cheers, Hugh Fisher From srkunze at mail.de Tue Nov 21 17:16:43 2017 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 21 Nov 2017 23:16:43 +0100 Subject: [Python-ideas] Adding a thin wrapper class around the functions in stdlib.heapq In-Reply-To: References: Message-ID: Maybe, that suffices: https://pypi.python.org/pypi/xheap On 21.11.2017 03:46, bunslow wrote: > Perhaps such repetition is a sign that *something* needs to be done... > > Thanks for the link though. I'm new enough to the community that it > didn't even occur to me to search for prior discussions. > > On Mon, Nov 20, 2017 at 8:38 PM, Sebastian Kreft > wrote: > > This has been brought up multiple times. Last time was on this > thread > https://mail.python.org/pipermail/python-ideas/2016-October/043024.html > . > > On Tue, Nov 21, 2017 at 3:13 AM, bunslow > wrote: > > Nothing so bombastic this time. The heapq functions are > basically all named "heapsomething", and basically all take a > "heap" for their first argument, with supplementary args > coming after. It's a textbook example of the (hypothetical) > Object Oriented Manifesto? where defining a class increases > type safety and programmers' conceptual clarity. There're > practically no drawbacks, and the code to be added would be > very simple. Updating the tests and docs would probably be > harder. > > In pure Python, such a class would look like this: > > class Heap(list): > > ? ? def __init__(self, iterable=None): > ? ? ? ? if iterable: > ? ? ? ? ? ? super().__init__(iterable) > ? ? ? ? else: > ? ? ? ? ? ? super().__init__() > > ? ? ? ? self.heapify() > > ? ? push = heapq.heappush > ? ? pop = heapq.heappop > ? ? pushpop = heapq.heappushpop > ? ? replace = heapq.heapreplace > ? ? heapify = heapq.heapify > > ? ? # This could be a simple wrapper as well, but I had the > following thoughts anyways, so here they are > ? ? def nsmallest(self, n, key=None): > ? ? ? ? # heapq.nsmallest makes a *max* heap of the first n > elements, > ? ? ? ? # while we know that self is already a min heap, so we can > ? ? ? ? # make the max heap construction faster > ? ? ? ? self[:n] = reversed(self[:n]) > ? ? ? ? return heapq.nsmallest(n, self, key) > > ? ? # do we define nlargest on a minheap?? > > > Wrapping around the C builtin functions (which aren't > descriptors) would be a bit harder, but not much so: > > from functools import partial > > class Heap(list): > ? ? def __init__(self, iterable=None): > ? ? ? ? if iterable: > ? ? ? ? ? ? super().__init__(iterable) > ? ? ? ? else: > ? ? ? ? ? ? super().__init__() > > ? ? ? ? self.heapify = partial(heapq.heapify, self) > ? ? ? ? self.push = partial(heapq.heappush, self) > ? ? ? ? ... > > ? ? ? ? self.heapify() > > > Thoughts? > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > > Code of Conduct: http://python.org/psf/codeofconduct/ > > > > > > -- > Sebastian Kreft > > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Tue Nov 21 17:11:27 2017 From: srkunze at mail.de (Sven R. Kunze) Date: Tue, 21 Nov 2017 23:11:27 +0100 Subject: [Python-ideas] Should Python have user-defined constants? In-Reply-To: References: <742BABA7-C73A-490F-96EB-B65E44EAC236@icloud.com> <20171121113101.GM19802@ando.pearwood.info> Message-ID: <065f7f25-9412-0997-3732-8e2861ef1678@mail.de> On 21.11.2017 19:05, ????? wrote: > I don't understand the question. The use case was explained before - > people want to have better ways to reason about their programs. > Statically. Why dismiss it as a non-usecase? It's helpful for both > tools and humans. Then type annotations are for them. But please don't clutter the core language with this. > When I read "final int x = 5;" in Java, I don't have to think about it > anymore - it's 5. When I read "X = 5" in Python, it might be a > constant, but it might also be a misnomer, or something that used to > be a constant, or a class reassigned. What about using real-world names instead of X? That could make the intentions crystal clear if you ask me. Cheers, Sven PS: it seems to me that this resembles the typing discussions here. Using non-sensical variable names to explain why type annotations are sorely needed. From steve at pearwood.info Tue Nov 21 17:35:55 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 22 Nov 2017 09:35:55 +1100 Subject: [Python-ideas] Make bytes __repr__ and __str__ representation different? In-Reply-To: References: <20171121151623.GO19802@ando.pearwood.info> Message-ID: <20171121223555.GP19802@ando.pearwood.info> On Tue, Nov 21, 2017 at 01:35:41PM -0200, Joao S. O. Bueno wrote: > On 21 November 2017 at 13:16, Steven D'Aprano wrote: > > I'd rather leave __str__ and __repr__ alone. Changing them will have > > huge backwards compatibility implications. I'd rather give bytes a > > hexdump() method that returns a string: > > > > '2a 2b 2d 2e 2f' > > Do you mean, like bytes.hex? Guido's Time Machine strikes again! -- Steve From prometheus235 at gmail.com Tue Nov 21 17:49:25 2017 From: prometheus235 at gmail.com (Nick Timkovich) Date: Tue, 21 Nov 2017 16:49:25 -0600 Subject: [Python-ideas] Make bytes __repr__ and __str__ representation different? In-Reply-To: References: Message-ID: On Tue, Nov 21, 2017 at 11:22 AM, Chris Barker wrote: > supposedly __repr__ is supposed to give an eval-able version -- which your > proposal is. But the way you did your example indicates that: > > bytes((42, 43, 44, 45, 46)) > > would be an even better __repr__, if the goal is to make it clear and easy > that it is a "container of integers from 0 to 255" > I wonder if for repr-synonyms, a format specifier to `repr()` could toggle how the object chooses to display itself would be handy: x = b'*+-./' repr(x) # b'*+-./' repr(x, bytes.REPR_HEX_STRING) # b'\x2a\x2b\x2c\x2d\x2e' repr(x, bytes.REPR_BYTES) # bytes([42, 43, 44, 45, 46]) repr(x, bytes.REPR_HEX_BYTES) # bytes([0x2A, 0x2B, 0x2C, 0x2D, 0x2E]) Kinda like `format()` but such that all of `eval(repr(x, ))` are equal. -------------- next part -------------- An HTML attachment was scrubbed... URL: From prometheus235 at gmail.com Tue Nov 21 17:56:27 2017 From: prometheus235 at gmail.com (Nick Timkovich) Date: Tue, 21 Nov 2017 16:56:27 -0600 Subject: [Python-ideas] Adding a thin wrapper class around the functions in stdlib.heapq In-Reply-To: References: Message-ID: On Tue, Nov 21, 2017 at 4:16 PM, Sven R. Kunze wrote: > Maybe, that suffices: https://pypi.python.org/pypi/xheap > I still think the heapq.heap* functions are atrocious and they should immediately be packaged with *no additional features* into a stdlib object for reasons along the line of https://mail.python.org/pipermail/python-ideas/2017-October/047514.html If the improvements in xheap are converging on boringly-stable, maybe bring them in at a later date. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Tue Nov 21 19:27:59 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 22 Nov 2017 11:27:59 +1100 Subject: [Python-ideas] Should Python have user-defined constants? In-Reply-To: References: <742BABA7-C73A-490F-96EB-B65E44EAC236@icloud.com> <20171121113101.GM19802@ando.pearwood.info> Message-ID: <20171122002759.GQ19802@ando.pearwood.info> On Tue, Nov 21, 2017 at 09:34:33AM -0800, Chris Barker wrote: > On Tue, Nov 21, 2017 at 3:31 AM, Steven D'Aprano > wrote: > > The > > interesting (but is it useful?) concept is constant identifiers which > > cannot be re-bound or re-assigned once they are set the first time. > > This would actually be a substantial shift in what Python is about -- > currently the concept of type is all about values, not names I didn't say anything about types. My comments were about names and values (objects): - Mutability is a property of values (objects): some objects are mutable, and other objects are not. - Constantness is a property of names or other identifiers: some names can only be bound once ("constants"), other names can be bound and rebound at will ("variables"). Types are literally irrelevant to this, except in the sense that in many languages, including Python, the distinction between mutable and immutable is usually controlled by the type of object. But that's not fundamental to the concept: we could, if we wanted, decouple the two. > -- there are > various rules about scope, and they can be adjusted with global and > nonlocal, but it's only about scope -- a name is still a name, and can be > bound to any type of object, etc. Sure, that's the way Python is now. But that's not fundamental to the concepts. Other languages allow other distinctions between names, not just scope: for instance, names can be static (they keep their value between function invocations), they can be read-only some languages allow you to specify whether they are kept in RAM or in a register. > static languages, on the other hand often (always) assign type to the name > itself -- so adding constants and the like makes sense. Constantness is orthogonal to the concept of static typing of variables. > But changing the > property of a name (this name can not be rebound) is a big shift in what > names are about in Python -- and I don't think a good one. It is a big shift, although not that big since we already have read-only properties. This would "merely" (hah!) extend that concept to other namespaces apart from instances. I'm undecided as to whether the benefit would justify the cost. -- Steve From steve at pearwood.info Tue Nov 21 19:45:06 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 22 Nov 2017 11:45:06 +1100 Subject: [Python-ideas] Adding a thin wrapper class around the functions in stdlib.heapq In-Reply-To: References: Message-ID: <20171122004506.GR19802@ando.pearwood.info> On Tue, Nov 21, 2017 at 04:56:27PM -0600, Nick Timkovich wrote: > On Tue, Nov 21, 2017 at 4:16 PM, Sven R. Kunze wrote: > > > Maybe, that suffices: https://pypi.python.org/pypi/xheap > > > I still think the heapq.heap* functions are atrocious and they should > immediately be packaged with *no additional features* into a stdlib object > for reasons along the line of > https://mail.python.org/pipermail/python-ideas/2017-October/047514.html I think you pasted the wrong URL. That link is about pip, and the discoverability of third-party libraries. It says nothing about why functions are "atrocious" and why wrapping them into an object is better. But generally, Python's APIs are not "pure object oriented" in the Java sense, and we don't needlessly create objects just for the sake of ensuring everything is an object. Functions are fine too, and if the only difference between a function and method is the syntax you use to call it: function(value, arguments) versus value.function(arguments) then that's a difference that makes no difference, and there is no advantage to using an object wrapper. See also: http://steve-yegge.blogspot.com.au/2006/03/execution-in-kingdom-of-nouns.html for a perspective on how the emphasis on objects is harmful. What advantage is there to making the heap functions into Heap methods? -- Steve From rosuav at gmail.com Tue Nov 21 19:47:28 2017 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 22 Nov 2017 11:47:28 +1100 Subject: [Python-ideas] Should Python have user-defined constants? In-Reply-To: <20171122002759.GQ19802@ando.pearwood.info> References: <742BABA7-C73A-490F-96EB-B65E44EAC236@icloud.com> <20171121113101.GM19802@ando.pearwood.info> <20171122002759.GQ19802@ando.pearwood.info> Message-ID: On Wed, Nov 22, 2017 at 11:27 AM, Steven D'Aprano wrote: > On Tue, Nov 21, 2017 at 09:34:33AM -0800, Chris Barker wrote: >> On Tue, Nov 21, 2017 at 3:31 AM, Steven D'Aprano >> wrote: > >> > The >> > interesting (but is it useful?) concept is constant identifiers which >> > cannot be re-bound or re-assigned once they are set the first time. >> >> This would actually be a substantial shift in what Python is about -- >> currently the concept of type is all about values, not names > > I didn't say anything about types. My comments were about names and > values (objects): > > - Mutability is a property of values (objects): some objects are > mutable, and other objects are not. > > - Constantness is a property of names or other identifiers: some > names can only be bound once ("constants"), other names can be > bound and rebound at will ("variables"). > > Types are literally irrelevant to this, except in the sense that in many > languages, including Python, the distinction between mutable and > immutable is usually controlled by the type of object. But that's not > fundamental to the concept: we could, if we wanted, decouple the two. There's a good reason for that - the type of an object determines what operations make sense (you can add two numbers together, but you can't add two open sockets), and mutability impacts the validity of operations. Can you give an example of where it would make sense to decouple mutability from object type? Everything I can think of in Python works by creating a separate type (eg frozenset for set), thus allowing new operations (eg hashing). The only way I can think of to decouple them would be to have a single type with all the operations available, and then raise exceptions where some operations aren't valid: class Foo: def __hash__(self): if self.mutable: raise TypeError return some_hash def __iadd__(self, other): if not self.mutable: return self + other # perform an in-place addition return self def __setitem__(self, key): if not self.mutable: raise TypeError # alter self This makes it hard to reason about the code - particularly in the case of __iadd__, where it'll sometimes behave like a list's += and sometimes like an int's. Where would you, preferably with some sort of real-world example, actually make use of this sort of thing? ChrisA From rosuav at gmail.com Tue Nov 21 19:51:16 2017 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 22 Nov 2017 11:51:16 +1100 Subject: [Python-ideas] Make bytes __repr__ and __str__ representation different? In-Reply-To: References: Message-ID: On Wed, Nov 22, 2017 at 9:49 AM, Nick Timkovich wrote: > On Tue, Nov 21, 2017 at 11:22 AM, Chris Barker > wrote: >> >> supposedly __repr__ is supposed to give an eval-able version -- which your >> proposal is. But the way you did your example indicates that: >> >> bytes((42, 43, 44, 45, 46)) >> >> would be an even better __repr__, if the goal is to make it clear and easy >> that it is a "container of integers from 0 to 255" > > > I wonder if for repr-synonyms, a format specifier to `repr()` could toggle > how the object chooses to display itself would be handy: > > x = b'*+-./' > repr(x) # b'*+-./' > repr(x, bytes.REPR_HEX_STRING) # b'\x2a\x2b\x2c\x2d\x2e' > repr(x, bytes.REPR_BYTES) # bytes([42, 43, 44, 45, 46]) > repr(x, bytes.REPR_HEX_BYTES) # bytes([0x2A, 0x2B, 0x2C, 0x2D, 0x2E]) > > Kinda like `format()` but such that all of `eval(repr(x, ))` are > equal. Methods are usually the best for that. Possibly with class methods to perform the reconstruction - which in this case you have: >>> b"asdf".hex() '61736466' >>> bytes.fromhex(_) b'asdf' ChrisA From rosuav at gmail.com Tue Nov 21 20:00:37 2017 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 22 Nov 2017 12:00:37 +1100 Subject: [Python-ideas] Adding a thin wrapper class around the functions in stdlib.heapq In-Reply-To: <20171122004506.GR19802@ando.pearwood.info> References: <20171122004506.GR19802@ando.pearwood.info> Message-ID: On Wed, Nov 22, 2017 at 11:45 AM, Steven D'Aprano wrote: > On Tue, Nov 21, 2017 at 04:56:27PM -0600, Nick Timkovich wrote: >> On Tue, Nov 21, 2017 at 4:16 PM, Sven R. Kunze wrote: >> >> > Maybe, that suffices: https://pypi.python.org/pypi/xheap >> > >> I still think the heapq.heap* functions are atrocious and they should >> immediately be packaged with *no additional features* into a stdlib object >> for reasons along the line of >> https://mail.python.org/pipermail/python-ideas/2017-October/047514.html > > I think you pasted the wrong URL. That link is about pip, and the > discoverability of third-party libraries. It says nothing about why > functions are "atrocious" and why wrapping them into an object is > better. > > But generally, Python's APIs are not "pure object oriented" in the Java > sense, and we don't needlessly create objects just for the sake of > ensuring everything is an object. Functions are fine too, and if the > only difference between a function and method is the syntax you use to > call it: > > function(value, arguments) > > versus > > value.function(arguments) > > > then that's a difference that makes no difference, and there is no > advantage to using an object wrapper. > > See also: > > http://steve-yegge.blogspot.com.au/2006/03/execution-in-kingdom-of-nouns.html > > for a perspective on how the emphasis on objects is harmful. > > What advantage is there to making the heap functions into Heap methods? A heap is an actual thing. We don't have dict methods implemented as functions operating on a "blob of memory" object; we have dict as a type. So the most simple and obvious way to work with a heap would be something like: from heaps import Heap h = Heap() h.push(blah) h.pop() So the question is more: why, with Python being the way it is, do the heap functions operate on a list? I think heapq.heapify is the answer: in linear time, it heapifies a list *in place*. I don't think there's any reason to have *both* interfaces to the heap functionality, but it certainly isn't illogical to try to treat a heap as a thing, and therefore have a Heap type. ChrisA From steve at pearwood.info Tue Nov 21 20:51:05 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 22 Nov 2017 12:51:05 +1100 Subject: [Python-ideas] Should Python have user-defined constants? In-Reply-To: References: <742BABA7-C73A-490F-96EB-B65E44EAC236@icloud.com> <20171121113101.GM19802@ando.pearwood.info> <20171122002759.GQ19802@ando.pearwood.info> Message-ID: <20171122015105.GS19802@ando.pearwood.info> On Wed, Nov 22, 2017 at 11:47:28AM +1100, Chris Angelico wrote: > On Wed, Nov 22, 2017 at 11:27 AM, Steven D'Aprano wrote: > > Types are literally irrelevant to this, except in the sense that in many > > languages, including Python, the distinction between mutable and > > immutable is usually controlled by the type of object. But that's not > > fundamental to the concept: we could, if we wanted, decouple the two. > > There's a good reason for that - the type of an object determines what > operations make sense (you can add two numbers together, but you can't > add two open sockets), and mutability impacts the validity of > operations. Can you give an example of where it would make sense to > decouple mutability from object type? I didn't say we should, or that I would, only that we could if we wanted to. But for the sake of the hypothetical argument, being able to freeze an object may be useful, as opposed to copying it into a new, frozen object. To make the difference clear: a = b = something() a = freeze(a) b would still be mutable; but: a = b = something() freeze(a) now b is immutable, since it is the same frozen object as a. But possibly even more interesting might be to have the concept of code permissions, so that you can grant software components either read/write or read-only permission on values you pass to them. (Immutable views could possibly do the same job.) At the moment every time you pass a list to a library function, you risk it modifying the list without your knowledge. In general that's more of a theoretical risk than an actual risk (we generally know which library functions mutate their arguments), but there may be circumstances where you'd like to pass a mutable object to (let's say) a callback function, or a thread, and trust that it can't modify the object at all. At the moment, the only obvious alternative is to make an arbitrarily expensive deepcopy of the object each time you pass it. Or just hope that the called function doesn't mess with your data. -- Steve From rosuav at gmail.com Tue Nov 21 20:58:07 2017 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 22 Nov 2017 12:58:07 +1100 Subject: [Python-ideas] Should Python have user-defined constants? In-Reply-To: <20171122015105.GS19802@ando.pearwood.info> References: <742BABA7-C73A-490F-96EB-B65E44EAC236@icloud.com> <20171121113101.GM19802@ando.pearwood.info> <20171122002759.GQ19802@ando.pearwood.info> <20171122015105.GS19802@ando.pearwood.info> Message-ID: On Wed, Nov 22, 2017 at 12:51 PM, Steven D'Aprano wrote: > On Wed, Nov 22, 2017 at 11:47:28AM +1100, Chris Angelico wrote: >> On Wed, Nov 22, 2017 at 11:27 AM, Steven D'Aprano wrote: > >> > Types are literally irrelevant to this, except in the sense that in many >> > languages, including Python, the distinction between mutable and >> > immutable is usually controlled by the type of object. But that's not >> > fundamental to the concept: we could, if we wanted, decouple the two. >> >> There's a good reason for that - the type of an object determines what >> operations make sense (you can add two numbers together, but you can't >> add two open sockets), and mutability impacts the validity of >> operations. Can you give an example of where it would make sense to >> decouple mutability from object type? > > I didn't say we should, or that I would, only that we could if we wanted > to. But for the sake of the hypothetical argument, being able to freeze > an object may be useful, as opposed to copying it into a new, frozen > object. Ah okay. Yeah, it's certainly plausible in theory, but I've never actually wanted it. ChrisA From wes.turner at gmail.com Tue Nov 21 21:11:23 2017 From: wes.turner at gmail.com (Wes Turner) Date: Tue, 21 Nov 2017 21:11:23 -0500 Subject: [Python-ideas] Rewriting the "roundrobin" recipe in the itertools documentation In-Reply-To: References: Message-ID: Here's toolz.itertoolz.interleave(): def interleave(seqs): """ Interleave a sequence of sequences >>> list(interleave([[1, 2], [3, 4]])) [1, 3, 2, 4] >>> ''.join(interleave(('ABC', 'XY'))) 'AXBYC' Both the individual sequences and the sequence of sequences may be infinite Returns a lazy iterator """ iters = itertools.cycle(map(iter, seqs)) while True: try: for itr in iters: yield next(itr) return except StopIteration: predicate = partial(operator.is_not, itr) iters = itertools.cycle(itertools.takewhile(predicate, iters)) At first glance, map should be e.g. six.moves.map for pythonic backward compatibility. Is this slower than the linear time implementations listed here? On Tuesday, November 21, 2017, Alon Snir wrote: > def roundrobin(*iterables): > "roundrobin('ABC', 'D', 'EF') --> A D E B F C" > nexts = [ iter(it).__next__ for it in iterables ] > i = 0 > while nexts: > i %= len(nexts) > try: > yield nexts[i]() > except StopIteration: > del nexts[i] > else: > i += 1 > > Regards > Alon Snir > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Tue Nov 21 23:55:51 2017 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 22 Nov 2017 17:55:51 +1300 Subject: [Python-ideas] Adding a thin wrapper class around the functions in stdlib.heapq In-Reply-To: References: <20171122004506.GR19802@ando.pearwood.info> Message-ID: <5A150357.8020004@canterbury.ac.nz> Chris Angelico wrote: > So the question is more: why, with Python being the way it is, do the > heap functions operate on a list? I think heapq.heapify is the answer: > in linear time, it heapifies a list *in place*. There's no reason a Heap object couldn't accomodate that case. E.g. the constructor could optionally accept an existing list to heapify and wrap. -- Greg From rosuav at gmail.com Wed Nov 22 01:11:53 2017 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 22 Nov 2017 17:11:53 +1100 Subject: [Python-ideas] Adding a thin wrapper class around the functions in stdlib.heapq In-Reply-To: <5A150357.8020004@canterbury.ac.nz> References: <20171122004506.GR19802@ando.pearwood.info> <5A150357.8020004@canterbury.ac.nz> Message-ID: On Wed, Nov 22, 2017 at 3:55 PM, Greg Ewing wrote: > Chris Angelico wrote: >> >> So the question is more: why, with Python being the way it is, do the >> heap functions operate on a list? I think heapq.heapify is the answer: >> in linear time, it heapifies a list *in place*. > > > There's no reason a Heap object couldn't accomodate that > case. E.g. the constructor could optionally accept an > existing list to heapify and wrap. > Yeah but if it's wrapping an existing list, it's not really constructing a new object. Every other constructor in Python will make something independent of its origin (if you call list() on a list, you get back a brand new copy, for instance), but to take advantage of in-place heapification, it would have to mutate the input. I think that would be even more surprising. ChrisA From prometheus235 at gmail.com Wed Nov 22 01:22:00 2017 From: prometheus235 at gmail.com (Nick Timkovich) Date: Wed, 22 Nov 2017 00:22:00 -0600 Subject: [Python-ideas] Adding a thin wrapper class around the functions in stdlib.heapq In-Reply-To: <20171122004506.GR19802@ando.pearwood.info> References: <20171122004506.GR19802@ando.pearwood.info> Message-ID: On Tue, Nov 21, 2017 at 6:45 PM, Steven D'Aprano wrote: > On Tue, Nov 21, 2017 at 04:56:27PM -0600, Nick Timkovich wrote: > > On Tue, Nov 21, 2017 at 4:16 PM, Sven R. Kunze wrote: > > > > > Maybe, that suffices: https://pypi.python.org/pypi/xheap > > > > > I still think the heapq.heap* functions are atrocious and they should > > immediately be packaged with *no additional features* into a stdlib > object > > for reasons along the line of > > https://mail.python.org/pipermail/python-ideas/2017-October/047514.html > > I think you pasted the wrong URL. That link is about pip, and the > discoverability of third-party libraries. > Probably straining the reference, but akin to "don't teach JSON with Python, teach XML-RPC because it supports that better in the stdlib", why not "don't teach heaps, just use lists with pop and insert+sort". > But generally, Python's APIs are not "pure object oriented" in the Java > sense, and we don't needlessly create objects just for the sake of > ensuring everything is an object. Functions are fine too... Functions are great. I'm a big fan of functions. However, 1. Once there are several that all have the same thing as an argument: thing_operation1(thing, arg), thing_operation2(thing, arg)...it's about time to bind them together. 2. And especially for the heap "soft-datatype": once it's heapified, naively modifying it with other methods will ruin the heap invariant. **The actual list you pass around can't be treated as a list.** -------------- next part -------------- An HTML attachment was scrubbed... URL: From srkunze at mail.de Wed Nov 22 05:16:56 2017 From: srkunze at mail.de (Sven R. Kunze) Date: Wed, 22 Nov 2017 11:16:56 +0100 Subject: [Python-ideas] Adding a thin wrapper class around the functions in stdlib.heapq In-Reply-To: References: <20171122004506.GR19802@ando.pearwood.info> Message-ID: <7f94efb9-68c5-766c-61e4-daa5ab9481e7@mail.de> On 22.11.2017 07:22, Nick Timkovich wrote: > Functions are great. I'm a big fan of functions. However, > > 1. Once there are several that all have the same thing as an argument: > thing_operation1(thing, arg), thing_operation2(thing, arg)...it's > about time to bind them together. > 2. And especially for the heap "soft-datatype": once it's heapified, > naively modifying it with other methods will ruin the heap invariant. > **The actual list you pass around can't be treated as a list.** Two important aspects to which I want to add some thoughts. The reason why I created xheap was because I actually needed a more complex datastructure which required more bookkeeping: 1) custom orders on the same data (list) 2) fast removal from within the heap The plain "Heap" implementation (object wrapper of heapq) was just for the sake of completion. Cheers, Sven PS: I tend to think that implementing xheap was easier BECAUSE the stdlib provided a function-only heap implementation. Nonetheless, both ways do have their merits. -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Wed Nov 22 06:06:45 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 22 Nov 2017 12:06:45 +0100 Subject: [Python-ideas] Adding a thin wrapper class around the functions in stdlib.heapq References: <20171122004506.GR19802@ando.pearwood.info> <5A150357.8020004@canterbury.ac.nz> Message-ID: <20171122120645.434eaebf@fsol> On Wed, 22 Nov 2017 17:11:53 +1100 Chris Angelico wrote: > On Wed, Nov 22, 2017 at 3:55 PM, Greg Ewing wrote: > > Chris Angelico wrote: > >> > >> So the question is more: why, with Python being the way it is, do the > >> heap functions operate on a list? I think heapq.heapify is the answer: > >> in linear time, it heapifies a list *in place*. > > > > > > There's no reason a Heap object couldn't accomodate that > > case. E.g. the constructor could optionally accept an > > existing list to heapify and wrap. > > > > Yeah but if it's wrapping an existing list, it's not really > constructing a new object. Every other constructor in Python will make > something independent of its origin (if you call list() on a list, you > get back a brand new copy, for instance), but to take advantage of > in-place heapification, it would have to mutate the input. I think > that would be even more surprising. In any case, copying a list a pretty cheap, so in-place heapification isn't a very important optimization. Regards Antoine. From solipsis at pitrou.net Wed Nov 22 06:09:23 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 22 Nov 2017 12:09:23 +0100 Subject: [Python-ideas] Adding a thin wrapper class around the functions in stdlib.heapq References: <20171122004506.GR19802@ando.pearwood.info> Message-ID: <20171122120923.27547682@fsol> On Wed, 22 Nov 2017 00:22:00 -0600 Nick Timkovich wrote: > > Functions are great. I'm a big fan of functions. However, > > 1. Once there are several that all have the same thing as an argument: > thing_operation1(thing, arg), thing_operation2(thing, arg)...it's about > time to bind them together. > 2. And especially for the heap "soft-datatype": once it's heapified, > naively modifying it with other methods will ruin the heap invariant. **The > actual list you pass around can't be treated as a list.** A third reason: documentation and discoverability. If I type help(some_heapified_list) at the prompt, I get the usual documentation for list methods, not the documentation of heapq functions... Regards Antoine. From stephanh42 at gmail.com Wed Nov 22 06:23:54 2017 From: stephanh42 at gmail.com (Stephan Houben) Date: Wed, 22 Nov 2017 12:23:54 +0100 Subject: [Python-ideas] Adding a thin wrapper class around the functions in stdlib.heapq In-Reply-To: <20171122120923.27547682@fsol> References: <20171122004506.GR19802@ando.pearwood.info> <20171122120923.27547682@fsol> Message-ID: I thought the purpose of heapq was to have a ready-made example for instructors on how an API can be improved by applying object-oriented techniques. ;-) I think adding a HeapQueue class would be a great idea. Obviously the existing functions would need to continue existing for backward compatibility. Stephan 2017-11-22 12:09 GMT+01:00 Antoine Pitrou : > On Wed, 22 Nov 2017 00:22:00 -0600 > Nick Timkovich > wrote: > > > > Functions are great. I'm a big fan of functions. However, > > > > 1. Once there are several that all have the same thing as an argument: > > thing_operation1(thing, arg), thing_operation2(thing, arg)...it's about > > time to bind them together. > > 2. And especially for the heap "soft-datatype": once it's heapified, > > naively modifying it with other methods will ruin the heap invariant. > **The > > actual list you pass around can't be treated as a list.** > > A third reason: documentation and discoverability. If I type > help(some_heapified_list) at the prompt, I get the usual documentation > for list methods, not the documentation of heapq functions... > > Regards > > Antoine. > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Wed Nov 22 17:02:21 2017 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 23 Nov 2017 11:02:21 +1300 Subject: [Python-ideas] Adding a thin wrapper class around the functions in stdlib.heapq In-Reply-To: References: <20171122004506.GR19802@ando.pearwood.info> <5A150357.8020004@canterbury.ac.nz> Message-ID: <5A15F3ED.6000306@canterbury.ac.nz> Chris Angelico wrote: > Yeah but if it's wrapping an existing list, it's not really > constructing a new object. That depends on what you consider the object to be. There are existing examples of objects that wrap other objects and mutate them, e.g. TextIOWrapper. If it would make anyone happier, there could be two classes, Heap and HeapWrapper (with one probably being implemented as a subclass of the other). -- Greg From AlonSnir at hotmail.com Wed Nov 22 06:28:20 2017 From: AlonSnir at hotmail.com (Alon Snir) Date: Wed, 22 Nov 2017 11:28:20 +0000 Subject: [Python-ideas] Rewriting the "roundrobin" recipe in the itertools documentation In-Reply-To: References: Message-ID: It would be faster with ?deque?: def roundrobin(*iterables): iters = deque(map(iter,iterables), len(iterables)) while iters: try: yield next(iters[0]) except StopIteration: iters.popleft() else: iters.rotate(-1) From: Wes Turner [mailto:wes.turner at gmail.com] Sent: Wednesday, November 22, 2017 04:11 To: Alon Snir Cc: python-ideas at python.org Subject: Re: [Python-ideas] Rewriting the "roundrobin" recipe in the itertools documentation Here's toolz.itertoolz.interleave(): def interleave(seqs): """ Interleave a sequence of sequences >>> list(interleave([[1, 2], [3, 4]])) [1, 3, 2, 4] >>> ''.join(interleave(('ABC', 'XY'))) 'AXBYC' Both the individual sequences and the sequence of sequences may be infinite Returns a lazy iterator """ iters = itertools.cycle(map(iter, seqs)) while True: try: for itr in iters: yield next(itr) return except StopIteration: predicate = partial(operator.is_not, itr) iters = itertools.cycle(itertools.takewhile(predicate, iters)) At first glance, map should be e.g. six.moves.map for pythonic backward compatibility. Is this slower than the linear time implementations listed here? On Tuesday, November 21, 2017, Alon Snir > wrote: def roundrobin(*iterables): "roundrobin('ABC', 'D', 'EF') --> A D E B F C" nexts = [ iter(it).__next__ for it in iterables ] i = 0 while nexts: i %= len(nexts) try: yield nexts[i]() except StopIteration: del nexts[i] else: i += 1 Regards Alon Snir _______________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Wed Nov 22 20:29:47 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 23 Nov 2017 12:29:47 +1100 Subject: [Python-ideas] Rewriting the "roundrobin" recipe in the itertools documentation In-Reply-To: References: Message-ID: <20171123012947.GT19802@ando.pearwood.info> On Wed, Nov 22, 2017 at 11:28:20AM +0000, Alon Snir wrote: > It would be faster with ?deque?: It isn't. According to my testing, your version with deque is approximately two times slower than the version from toolz.itertoolz that Wes quotes. -- Steve From ncoghlan at gmail.com Wed Nov 22 22:46:04 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 23 Nov 2017 13:46:04 +1000 Subject: [Python-ideas] Allow additional separator character in variables In-Reply-To: <23060.5173.899290.61724@turnbull.sk.tsukuba.ac.jp> References: <23059.34466.1253.737550@turnbull.sk.tsukuba.ac.jp> <23060.5173.899290.61724@turnbull.sk.tsukuba.ac.jp> Message-ID: On 21 November 2017 at 21:55, Stephen J. Turnbull < turnbull.stephen.fw at u.tsukuba.ac.jp> wrote: > Personally, I think that Python probably should ban non-ASCII > non-letter characters in identifiers and whitespace, and maybe add > them later in response to requests from native speakers of the > relevant languages. I don't know how easy that would be to do, > though, since I think the rule is already that identifiers must be > composed only of letters, numbers, and ASCII "_". Since Serhiy's > examples are valid, we'd have to rule them out explicitly, rather than > by reference to the Unicode database. Yuck. > We're not going to start second-guessing the Unicode Consortium on this point - human languages are complicated, and we don't have any special insight on this point that they don't. https://www.python.org/dev/peps/pep-3131/#specification-of-language-changes delegated this aspect of the language to them by way of the XID_Start and the XID_Continue categories, and we're not going to change that. Any hybrid Python 2/3 application or library is necessarily restricted to ASCII-only identifiers, since that's all that Python 2 supports. We've also explicitly retained the ASCII-only restriction for PyPI distribution names (see https://www.python.org/dev/peps/pep-0508/#names), but that doesn't restrict the names used for import packages, only the names used to publish and install those components. If we ever decide to lift that restriction, it will likely be by way of https://en.wikipedia.org/wiki/Punycode, similar to the way internationalized domain names work, as well as the way multi-phase extension module initialization locates init functions for extension modules with non-ASCII names. Beyond that, I'll note that these questions were all raised in the original PEP: https://www.python.org/dev/peps/pep-3131/#open-issues The reference interpreter really isn't the place to experiment with answering them - rather, they're more a question for opt-in code analysis, since that makes it possible for folks to choose settings that are right *for them* (e.g. by defining a set of "permitted scripts" [1], specifying the Unicode characters that should be allowed in identifiers beyond the core set of "Latin" code points allowed by ASCII) Cheers, Nick. [1] https://en.wikipedia.org/wiki/Script_(Unicode) -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Nov 22 23:05:05 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 23 Nov 2017 14:05:05 +1000 Subject: [Python-ideas] Adding a thin wrapper class around the functions in stdlib.heapq In-Reply-To: References: <20171122004506.GR19802@ando.pearwood.info> Message-ID: On 22 November 2017 at 11:00, Chris Angelico wrote: > So the question is more: why, with Python being the way it is, do the > heap functions operate on a list? I think heapq.heapify is the answer: > in linear time, it heapifies a list *in place*. > > I don't think there's any reason to have *both* interfaces to the heap > functionality, but it certainly isn't illogical to try to treat a heap > as a thing, and therefore have a Heap type. > Right, the parallel here is that we have a "heapq" module for the same reason that we have list.sort(), sorted(), and the bisect module, rather than a single "SortedList" type. https://code.activestate.com/recipes/577197-sortedcollection/ then provides an example of how to combine those into a "SortedCollection" type. That said, I'm still in favour of adding an object-oriented wrapper to either `collections` or the `heapq` module for all the classic OO reasons: - it makes it easier to reliably maintain the heap invariant (just drop your list reference after passing it to the heap wrapper) - it gives both human readers and static code analysers more information to work with ("this is a heap" rather than "this is a list") - it provides a hook for improved interactive help on heap instances I don't have any great concerns about potential confusion - the OO wrapper will be easy enough to use that anyone that's unsure will likely gravitate towards that, while the lower level `heapq` functions will remain available for folks writing their own heap implementations. This effect would likely be even more pronounced if the OO wrapper were made available as `collections.Heap` (`collections` already imports the `heapq` module for use in the Counter implementation). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From grant.jenks at gmail.com Thu Nov 23 00:11:16 2017 From: grant.jenks at gmail.com (Grant Jenks) Date: Wed, 22 Nov 2017 21:11:16 -0800 Subject: [Python-ideas] Adding a thin wrapper class around the functions in stdlib.heapq In-Reply-To: References: <20171122004506.GR19802@ando.pearwood.info> Message-ID: Honestly, I don't see the value in a thin object-oriented wrapper around heapq functions. I'm a big -1 on the idea. I'm the author of sortedcontainers ( https://pypi.python.org/pypi/sortedcontainers/) so I interact with a lot of people using sorted collections types. My observations show folk's needs tend to fit a bimodal distribution. At one end are those who get by with list.sort, bisect, or heapq and they seem to appreciate the simple function-based approach those modules provide. At the other end are those who want a SortedList data type and we have some good options on PyPI and some good building-blocks in the standard library. Personally, I think "sorted", "bisect" and "heapq" in the standard library are brilliant examples of the Python-way or "zen." I've learned a lot by studying their code and I encourage others to do the same. Just because something can be object-oriented doesn't mean it should be. There's a lot to be said for simplicity. I also think Nick's arguments are valid but I don't find them convincing. What I think would be sufficient is a "See also:" blurb like that under https://docs.python.org/3/library/bisect.html#bisect.insort which also references SortedContainers at http://www.grantjenks.com/docs/sortedcontainers/ and the same blurb on heapq. I think that would be a reasonable next-step before we include any new data type in the standard library. Grant On Wed, Nov 22, 2017 at 8:05 PM, Nick Coghlan wrote: > On 22 November 2017 at 11:00, Chris Angelico wrote: > >> So the question is more: why, with Python being the way it is, do the >> heap functions operate on a list? I think heapq.heapify is the answer: >> in linear time, it heapifies a list *in place*. >> >> I don't think there's any reason to have *both* interfaces to the heap >> functionality, but it certainly isn't illogical to try to treat a heap >> as a thing, and therefore have a Heap type. >> > > Right, the parallel here is that we have a "heapq" module for the same > reason that we have list.sort(), sorted(), and the bisect module, rather > than a single "SortedList" type. https://code.activestate.com/ > recipes/577197-sortedcollection/ then provides an example of how to > combine those into a "SortedCollection" type. > > That said, I'm still in favour of adding an object-oriented wrapper to > either `collections` or the `heapq` module for all the classic OO reasons: > > - it makes it easier to reliably maintain the heap invariant (just drop > your list reference after passing it to the heap wrapper) > - it gives both human readers and static code analysers more information > to work with ("this is a heap" rather than "this is a list") > - it provides a hook for improved interactive help on heap instances > > I don't have any great concerns about potential confusion - the OO wrapper > will be easy enough to use that anyone that's unsure will likely gravitate > towards that, while the lower level `heapq` functions will remain available > for folks writing their own heap implementations. > > This effect would likely be even more pronounced if the OO wrapper were > made available as `collections.Heap` (`collections` already imports the > `heapq` module for use in the Counter implementation). > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bunslow at gmail.com Thu Nov 23 00:12:31 2017 From: bunslow at gmail.com (bunslow) Date: Wed, 22 Nov 2017 23:12:31 -0600 Subject: [Python-ideas] Adding a thin wrapper class around the functions in stdlib.heapq In-Reply-To: References: <20171122004506.GR19802@ando.pearwood.info> Message-ID: On Wed, Nov 22, 2017 at 10:52 PM, bunslow wrote: > I'll just note the original proposal I made was specifically designed to > be the minimum possible improvement, to avoid controversy (and e.g. a PEP). > > I agree that there are significant flaws and room for improvement in the > current heapq module (which is why things like xheap exist), but even so > it's still very useful as is, and the minimal OOP wrapper would go a long > way towards usability of the current functionality (and of course "just use > pip" is often not a viable course of action for certain developers). It > seems that such a wrapper is widely (though not unanimously) approved. > > Next step I suppose is creating a bpo issue? > > On Wed, Nov 22, 2017 at 10:05 PM, Nick Coghlan wrote: > >> On 22 November 2017 at 11:00, Chris Angelico wrote: >> >>> So the question is more: why, with Python being the way it is, do the >>> heap functions operate on a list? I think heapq.heapify is the answer: >>> in linear time, it heapifies a list *in place*. >>> >>> I don't think there's any reason to have *both* interfaces to the heap >>> functionality, but it certainly isn't illogical to try to treat a heap >>> as a thing, and therefore have a Heap type. >>> >> >> Right, the parallel here is that we have a "heapq" module for the same >> reason that we have list.sort(), sorted(), and the bisect module, rather >> than a single "SortedList" type. https://code.activestate.com/r >> ecipes/577197-sortedcollection/ then provides an example of how to >> combine those into a "SortedCollection" type. >> >> That said, I'm still in favour of adding an object-oriented wrapper to >> either `collections` or the `heapq` module for all the classic OO reasons: >> >> - it makes it easier to reliably maintain the heap invariant (just drop >> your list reference after passing it to the heap wrapper) >> - it gives both human readers and static code analysers more information >> to work with ("this is a heap" rather than "this is a list") >> - it provides a hook for improved interactive help on heap instances >> >> I don't have any great concerns about potential confusion - the OO >> wrapper will be easy enough to use that anyone that's unsure will likely >> gravitate towards that, while the lower level `heapq` functions will remain >> available for folks writing their own heap implementations. >> >> This effect would likely be even more pronounced if the OO wrapper were >> made available as `collections.Heap` (`collections` already imports the >> `heapq` module for use in the Counter implementation). >> >> Cheers, >> Nick. >> >> -- >> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bunslow at gmail.com Thu Nov 23 00:22:40 2017 From: bunslow at gmail.com (bunslow) Date: Wed, 22 Nov 2017 23:22:40 -0600 Subject: [Python-ideas] Adding a thin wrapper class around the functions in stdlib.heapq In-Reply-To: References: <20171122004506.GR19802@ando.pearwood.info> Message-ID: Something *should* be object oriented with the functions in question all operate on the same data type, and in particular, those functions/verbs *are only well defined for that type*. heapq.heappush(list-not-heap, item) is perfectly valid code in the current interface, but doesn't make any sense at all. And list.sort is *not* function based, it's an object oriented method. sorted() provides the same functionality for other types, given that it's a well defined function for a wide variety of sequences (unlike heappush). (list.sort is a method because unlike sorted(), it operates inplace, and is thus only meaningful for mutable, "complete" (all in memory, not "lazy") sequences -- i.e., a list.) I've never used bisect, so I'll refrain from commenting on it. At the end of the day, the patch proposed is merely a wrapper around the functional approach; you are welcome to continue using it as you like, it's not going anywhere. I would propose that the docs put the OOP version first though. On Wed, Nov 22, 2017 at 11:11 PM, Grant Jenks wrote: > Honestly, I don't see the value in a thin object-oriented wrapper around > heapq functions. I'm a big -1 on the idea. > > I'm the author of sortedcontainers (https://pypi.python.org/pypi/ > sortedcontainers/) so I interact with a lot of people using sorted > collections types. My observations show folk's needs tend to fit a bimodal > distribution. At one end are those who get by with list.sort, bisect, or > heapq and they seem to appreciate the simple function-based approach those > modules provide. At the other end are those who want a SortedList data type > and we have some good options on PyPI and some good building-blocks in the > standard library. > > Personally, I think "sorted", "bisect" and "heapq" in the standard library > are brilliant examples of the Python-way or "zen." I've learned a lot by > studying their code and I encourage others to do the same. Just because > something can be object-oriented doesn't mean it should be. There's a lot > to be said for simplicity. I also think Nick's arguments are valid but I > don't find them convincing. > > What I think would be sufficient is a "See also:" blurb like that under > https://docs.python.org/3/library/bisect.html#bisect.insort which also > references SortedContainers at http://www.grantjenks.com/ > docs/sortedcontainers/ and the same blurb on heapq. I think that would be > a reasonable next-step before we include any new data type in the standard > library. > > Grant > > > On Wed, Nov 22, 2017 at 8:05 PM, Nick Coghlan wrote: > >> On 22 November 2017 at 11:00, Chris Angelico wrote: >> >>> So the question is more: why, with Python being the way it is, do the >>> heap functions operate on a list? I think heapq.heapify is the answer: >>> in linear time, it heapifies a list *in place*. >>> >>> I don't think there's any reason to have *both* interfaces to the heap >>> functionality, but it certainly isn't illogical to try to treat a heap >>> as a thing, and therefore have a Heap type. >>> >> >> Right, the parallel here is that we have a "heapq" module for the same >> reason that we have list.sort(), sorted(), and the bisect module, rather >> than a single "SortedList" type. https://code.activestate.com/r >> ecipes/577197-sortedcollection/ then provides an example of how to >> combine those into a "SortedCollection" type. >> >> That said, I'm still in favour of adding an object-oriented wrapper to >> either `collections` or the `heapq` module for all the classic OO reasons: >> >> - it makes it easier to reliably maintain the heap invariant (just drop >> your list reference after passing it to the heap wrapper) >> - it gives both human readers and static code analysers more information >> to work with ("this is a heap" rather than "this is a list") >> - it provides a hook for improved interactive help on heap instances >> >> I don't have any great concerns about potential confusion - the OO >> wrapper will be easy enough to use that anyone that's unsure will likely >> gravitate towards that, while the lower level `heapq` functions will remain >> available for folks writing their own heap implementations. >> >> This effect would likely be even more pronounced if the OO wrapper were >> made available as `collections.Heap` (`collections` already imports the >> `heapq` module for use in the Counter implementation). >> >> Cheers, >> Nick. >> >> -- >> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From turnbull.stephen.fw at u.tsukuba.ac.jp Thu Nov 23 00:24:16 2017 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Thu, 23 Nov 2017 14:24:16 +0900 Subject: [Python-ideas] Allow additional separator character in variables In-Reply-To: References: <23059.34466.1253.737550@turnbull.sk.tsukuba.ac.jp> <23060.5173.899290.61724@turnbull.sk.tsukuba.ac.jp> Message-ID: <23062.23424.677384.854676@turnbull.sk.tsukuba.ac.jp> Mikhail V writes: > A single word written in local language should not. But its a perfect way > to make whole code look like a mess. Alex Martelli wrote a couple of interesting posts about his experiences with multilingual comments back in the discussion of PEP 263. One of them involved a team from Israel, I think, or maybe South Africa. If you Google site:mail.python.org for Alex and those countries, the thread would probably come up. From ncoghlan at gmail.com Thu Nov 23 00:35:07 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 23 Nov 2017 15:35:07 +1000 Subject: [Python-ideas] Adding a thin wrapper class around the functions in stdlib.heapq In-Reply-To: References: <20171122004506.GR19802@ando.pearwood.info> Message-ID: On 23 November 2017 at 15:22, bunslow wrote: > Something *should* be object oriented with the functions in question all > operate on the same data type, and in particular, those functions/verbs > *are only well defined for that type*. heapq.heappush(list-not-heap, item) > is perfectly valid code in the current interface, but doesn't make any > sense at all. And list.sort is *not* function based, it's an object > oriented method. sorted() provides the same functionality for other types, > given that it's a well defined function for a wide variety of sequences > (unlike heappush). (list.sort is a method because unlike sorted(), it > operates inplace, and is thus only meaningful for mutable, "complete" (all > in memory, not "lazy") sequences -- i.e., a list.) > > I've never used bisect, so I'll refrain from commenting on it. > > At the end of the day, the patch proposed is merely a wrapper around the > functional approach; you are welcome to continue using it as you like, it's > not going anywhere. I would propose that the docs put the OOP version first > though. > Ensuring the docs are clearly separated is part of why I think "collections.Heap" would make more sense as a proposal than "heapq.Heap" - collections already imports heapq for use in the Counter implementation, and adding another primitive container type to collections is a smaller conceptual change than updating heapq to being more than just a heap algorithm library. I'd also note that a collections.Heap class that accepts the underlying list to use as its sole parameter would compose nicely with the list builtin to allow creation of a heap from an arbitrary iterable: heap = collections.Heap(list(iterable)) rather than the current: heap = list(iterable) heapq.heapify(heap) That's akin to the difference between: data = sorted(iterable) and: data = list(iterable) data.sort() I don't think it would be a disaster if we continued to leave this out, but I don't think it would be a major maintenance burden or future barrier to learning to add it, either. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From turnbull.stephen.fw at u.tsukuba.ac.jp Thu Nov 23 01:34:20 2017 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Thu, 23 Nov 2017 15:34:20 +0900 Subject: [Python-ideas] Allow additional separator character in variables In-Reply-To: References: <23059.34466.1253.737550@turnbull.sk.tsukuba.ac.jp> <23060.5173.899290.61724@turnbull.sk.tsukuba.ac.jp> Message-ID: <23062.27628.21997.896466@turnbull.sk.tsukuba.ac.jp> Nick Coghlan writes: > We're not going to start second-guessing the Unicode Consortium on this > point - human languages are complicated, and we don't have any special > insight on this point that they don't. Agreed. Python, however, is NOT a (natural) human language, and the Unicode Consortium definition of conformance does NOT prohibit subsetting appropriate to the purpose. We DO know more than the Unicode Consortium about Python. For example, I suspect that your catholic appetite for XID in identifiers does not apply to syntactic keywords or names of builtins. > https://www.python.org/dev/peps/pep-3131/#specification-of-language-changes > delegated this aspect of the language to them by way of the XID_Start and > the XID_Continue categories, and we're not going to change that. > The reference interpreter really isn't the place to experiment with > answering them - rather, they're more a question for opt-in code > analysis, I agree about experimentation. I'm not in a hurry, since I've only seen IDEOGRAPHIC SPACE and full-width ASCII break Python programs once or twice in the ten years I've been teaching my students to use it. Steve From ncoghlan at gmail.com Thu Nov 23 01:45:46 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 23 Nov 2017 16:45:46 +1000 Subject: [Python-ideas] Allow additional separator character in variables In-Reply-To: <23062.27628.21997.896466@turnbull.sk.tsukuba.ac.jp> References: <23059.34466.1253.737550@turnbull.sk.tsukuba.ac.jp> <23060.5173.899290.61724@turnbull.sk.tsukuba.ac.jp> <23062.27628.21997.896466@turnbull.sk.tsukuba.ac.jp> Message-ID: On 23 November 2017 at 16:34, Stephen J. Turnbull < turnbull.stephen.fw at u.tsukuba.ac.jp> wrote: > Nick Coghlan writes: > > > We're not going to start second-guessing the Unicode Consortium on this > > point - human languages are complicated, and we don't have any special > > insight on this point that they don't. > > Agreed. Python, however, is NOT a (natural) human language, and the > Unicode Consortium definition of conformance does NOT prohibit > subsetting appropriate to the purpose. We DO know more than the > Unicode Consortium about Python. For example, I suspect that your > catholic appetite for XID in identifiers does not apply to syntactic > keywords or names of builtins. > We already have a stricter ASCII-only naming policy for the standard library: https://www.python.org/dev/peps/pep-3131/#policy-specification That's different from placing additional constraints on end-user code, though, as that's where the line between "programming language" and "natural language" gets blurry (variable, function, attribute, and method names are often nouns and verbs in the author's language, and this is also the case for data-derived APIs like pandas column names) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From grant.jenks at gmail.com Thu Nov 23 02:13:59 2017 From: grant.jenks at gmail.com (Grant Jenks) Date: Wed, 22 Nov 2017 23:13:59 -0800 Subject: [Python-ideas] Adding a thin wrapper class around the functions in stdlib.heapq In-Reply-To: References: <20171122004506.GR19802@ando.pearwood.info> Message-ID: On Wed, Nov 22, 2017 at 9:22 PM, bunslow wrote: > Something *should* be object oriented with the functions in question all > operate on the same data type, and in particular, those functions/verbs > *are only well defined for that type*. > But here you are missing something that I think is important. These functions are valid on *any* data type that has the methods of a mutable sequence and has had heapq.heapify called on it. This is part of the brilliance of heapq. Suppose we wanted to create a list-like data-type backed by a filesystem or database. I have done exactly such a thing like: from collections import MutableSequence class MyFileSystemList(MutableSequence): ... data = MyFileSystemList('dir-path') heapq.heapify(data) heapq.heappush(data, 'text') And it will work! The heap algorithm is exposed through a high-level functional interface so that you can take advantage of duck-typing. This is an important Python feature. heapq.heappush(list-not-heap, item) is perfectly valid code in the current > interface, but doesn't make any sense at all. And list.sort is *not* > function based, it's an object oriented method. sorted() provides the same > functionality for other types, given that it's a well defined function for > a wide variety of sequences (unlike heappush). (list.sort is a method > because unlike sorted(), it operates inplace, and is thus only meaningful > for mutable, "complete" (all in memory, not "lazy") sequences -- i.e., a > list.) > Personally, I find it irritating that list.sort will not similarly work until I inherit from list. I would prefer if it worked on mutable sequences. But regardless, if you inherit from "list", define all your own methods to store data in a file system, database, whatever then "list.sort" will work just fine as if it were a function: class MyFileSystemList(list): ... # Define methods to read/write from file system. data = MyFileSystemList() list.sort(data) So it's not the case that list.sort is a method and must only be used as such. Rather, list.sort captures the algorithm known as sorting and works on any mutable sequence (with the unfortunate additional requirement that it inherits from list). I've never used bisect, so I'll refrain from commenting on it. > > At the end of the day, the patch proposed is merely a wrapper around the > functional approach; you are welcome to continue using it as you like, it's > not going anywhere. I would propose that the docs put the OOP version first > though. > > On Wed, Nov 22, 2017 at 11:11 PM, Grant Jenks wrote: > >> Honestly, I don't see the value in a thin object-oriented wrapper around >> heapq functions. I'm a big -1 on the idea. >> >> I'm the author of sortedcontainers (https://pypi.python.org/pypi/ >> sortedcontainers/) so I interact with a lot of people using sorted >> collections types. My observations show folk's needs tend to fit a bimodal >> distribution. At one end are those who get by with list.sort, bisect, or >> heapq and they seem to appreciate the simple function-based approach those >> modules provide. At the other end are those who want a SortedList data type >> and we have some good options on PyPI and some good building-blocks in the >> standard library. >> >> Personally, I think "sorted", "bisect" and "heapq" in the standard >> library are brilliant examples of the Python-way or "zen." I've learned a >> lot by studying their code and I encourage others to do the same. Just >> because something can be object-oriented doesn't mean it should be. There's >> a lot to be said for simplicity. I also think Nick's arguments are valid >> but I don't find them convincing. >> >> What I think would be sufficient is a "See also:" blurb like that under >> https://docs.python.org/3/library/bisect.html#bisect.insort which also >> references SortedContainers at http://www.grantjenks.com/d >> ocs/sortedcontainers/ and the same blurb on heapq. I think that would be >> a reasonable next-step before we include any new data type in the standard >> library. >> >> Grant >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Thu Nov 23 03:02:27 2017 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 23 Nov 2017 21:02:27 +1300 Subject: [Python-ideas] Adding a thin wrapper class around the functions in stdlib.heapq In-Reply-To: References: <20171122004506.GR19802@ando.pearwood.info> Message-ID: <5A168093.2070004@canterbury.ac.nz> Grant Jenks wrote: > The heap algorithm is exposed through a high-level > functional interface so that you can take advantage of duck-typing. This would also be true of a HeapWrapper class that wrapped an existing sequence. There's a difference between the heap functions and sorted(). The latter is just a single function that does its job and returns a result. But the heap interface is a collection of functions that have to be used together in the right way to maintain the heap invariants. That suggests some encapsulation is in order. -- Greg From gadgetsteve at live.co.uk Thu Nov 23 03:43:01 2017 From: gadgetsteve at live.co.uk (Steve Barnes) Date: Thu, 23 Nov 2017 08:43:01 +0000 Subject: [Python-ideas] Possible Enhancement to py.exe Python Launcher Message-ID: Following on from the discussions on pip I would like to suggest, (and possibly implement), a minor change to the current py.exe python launcher. Currently the launcher, when called without a version specifier, defaults to the highest version on the basis of 3>2, x.11 > x.9, -64 > -32 however this may not always be the most desirable behaviour. I would like to suggest that it take a configuration value for the default version to use, (and possibly a separate ones for pyw, & file associations), honouring the following with the later overriding: default - as current system setting - from registry user setting - from registry user setting - from config file maybe %appdata%\pylauncher\defaults.ini environment setting - from getenv local setting - from .pyconfig file in the current directory. Options would be the same format as the -X[.Y][-BB] format currently accepted on the command line plus a --No-Py-Default option which would always error out if the version to invoke was not specified. I see this as potentially adding quite a lot of value for people with multiple python versions installed and it could tie in quite well with the possible use of py.exe as an entry point for tools such as pip. It might also increase the awareness of the launcher as those who have to stick with python 2 for the moment or in a specific context could set the default to what they need but can always override. -- Steve (Gadget) Barnes Any opinions in this message are my personal opinions and do not reflect those of my employer. --- This email has been checked for viruses by AVG. http://www.avg.com From p.f.moore at gmail.com Thu Nov 23 04:53:48 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 23 Nov 2017 09:53:48 +0000 Subject: [Python-ideas] Possible Enhancement to py.exe Python Launcher In-Reply-To: References: Message-ID: If I understand your proposal correctly, this is already possible, see https://www.python.org/dev/peps/pep-0397/#python-version-qualifiers The details are likely a little different than what you're proposing, but if they don't cover what you're trying to do, maybe you could give a more specific example of a use case that isn't currently possible? Paul On 23 November 2017 at 08:43, Steve Barnes wrote: > Following on from the discussions on pip I would like to suggest, (and > possibly implement), a minor change to the current py.exe python launcher. > > Currently the launcher, when called without a version specifier, > defaults to the highest version on the basis of 3>2, x.11 > x.9, -64 > > -32 however this may not always be the most desirable behaviour. > > I would like to suggest that it take a configuration value for the > default version to use, (and possibly a separate ones for pyw, & file > associations), honouring the following with the later overriding: > > default - as current > system setting - from registry > user setting - from registry > user setting - from config file maybe %appdata%\pylauncher\defaults.ini > environment setting - from getenv > local setting - from .pyconfig file in the current directory. > > Options would be the same format as the -X[.Y][-BB] format currently > accepted on the command line plus a --No-Py-Default option which would > always error out if the version to invoke was not specified. > > I see this as potentially adding quite a lot of value for people with > multiple python versions installed and it could tie in quite well with > the possible use of py.exe as an entry point for tools such as pip. > > It might also increase the awareness of the launcher as those who have > to stick with python 2 for the moment or in a specific context could set > the default to what they need but can always override. > > -- > Steve (Gadget) Barnes > Any opinions in this message are my personal opinions and do not reflect > those of my employer. > > --- > This email has been checked for viruses by AVG. > http://www.avg.com > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From ncoghlan at gmail.com Thu Nov 23 05:18:27 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 23 Nov 2017 20:18:27 +1000 Subject: [Python-ideas] Adding a thin wrapper class around the functions in stdlib.heapq In-Reply-To: References: <20171122004506.GR19802@ando.pearwood.info> Message-ID: On 23 November 2017 at 17:13, Grant Jenks wrote: > And it will work! The heap algorithm is exposed through a high-level > functional interface so that you can take advantage of duck-typing. This is > an important Python feature. > Nobody is proposing taking the functional interface away (just as nobody is proposing to take the bisect module away). Instead, the argument is: - the heapq functions need to be combined with concrete storage to be useful - a simple wrapper class that combines them with a given list (or other mutable sequence) is short and easy to maintain - such a wrapper class is also useful in its own right for basic heap use cases - so let's provide one in the collections module The potential arguments against that are: - this means yet-another-data-type in the collections module, so it will make the collections module harder to learn - people will be confused as to whether they should use collections.Heap or the heapq functions The risks of both of those outcomes seem low, since the main current source of confusion appears to be folks looking for a concrete "Heap" container type, and being surprised when they get told they either need to be build a DIY one from a list + the heapq functions, or else install a 3rd party dependency to get a preassembled one. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikhailwas at gmail.com Thu Nov 23 09:10:28 2017 From: mikhailwas at gmail.com (Mikhail V) Date: Thu, 23 Nov 2017 15:10:28 +0100 Subject: [Python-ideas] Allow additional separator character in variables In-Reply-To: References: <23059.34466.1253.737550@turnbull.sk.tsukuba.ac.jp> <23060.5173.899290.61724@turnbull.sk.tsukuba.ac.jp> Message-ID: On Thu, Nov 23, 2017 at 4:46 AM, Nick Coghlan wrote: > On 21 November 2017 at 21:55, Stephen J. Turnbull > wrote: >> >> Personally, I think that Python probably should ban non-ASCII >> non-letter characters in identifiers and whitespace, and maybe add >> them later in response to requests from native speakers of the >> relevant languages. I don't know how easy that would be to do, >> though, since I think the rule is already that identifiers must be >> composed only of letters, numbers, and ASCII "_". Since Serhiy's >> examples are valid, we'd have to rule them out explicitly, rather than >> by reference to the Unicode database. Yuck. > Beyond that, I'll note that these questions were all raised in the original > PEP: https://www.python.org/dev/peps/pep-3131/#open-issues > > The reference interpreter really isn't the place to experiment with > answering them - rather, they're more a question for opt-in code analysis, > since that makes it possible for folks to choose settings that are right > *for them* (e.g. by defining a set of "permitted scripts" [1], specifying > the Unicode characters that should be allowed in identifiers beyond the core > set of "Latin" code points allowed by ASCII) Well, then there is some bitter irony in this, so it allows pretty much everything, but does not allow me to beautify code with hyphens. I can fully understand the wish to use non-latin scripts in strings or comments. As for identifiers, IMO, apart from latin letters and underscore, the first unicode candidate I would add is U+2010. And probably the LAST one I would add. Mikhail From rosuav at gmail.com Thu Nov 23 09:16:59 2017 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 24 Nov 2017 01:16:59 +1100 Subject: [Python-ideas] Allow additional separator character in variables In-Reply-To: References: <23059.34466.1253.737550@turnbull.sk.tsukuba.ac.jp> <23060.5173.899290.61724@turnbull.sk.tsukuba.ac.jp> Message-ID: On Fri, Nov 24, 2017 at 1:10 AM, Mikhail V wrote: > Well, then there is some bitter irony in this, so it allows pretty > much everything, > but does not allow me to beautify code with hyphens. > I can fully understand the wish to use non-latin scripts in strings or comments. > As for identifiers, IMO, apart from latin letters and underscore, the > first unicode candidate > I would add is U+2010. And probably the LAST one I would add. > Fortunately for the world, you're not the one who decided which characters were permitted in Python identifiers. The ability to use non-English words for function/variable names is of huge value; the ability to use a hyphen is of some value, but not nearly as much. Can this thread move to python-list? Or, better, to python-rants-about-unicode-list, to which I don't subscribe? ChrisA From carl.input at gmail.com Thu Nov 23 09:29:43 2017 From: carl.input at gmail.com (Carl Smith) Date: Thu, 23 Nov 2017 14:29:43 +0000 Subject: [Python-ideas] Allow additional separator character in variables In-Reply-To: References: <23059.34466.1253.737550@turnbull.sk.tsukuba.ac.jp> <23060.5173.899290.61724@turnbull.sk.tsukuba.ac.jp> Message-ID: Can't we just tell everyone to speak US English, and go back to ASCII? It would be a less painful migration. -- Carl Smith carl.input at gmail.com On 23 November 2017 at 14:16, Chris Angelico wrote: > On Fri, Nov 24, 2017 at 1:10 AM, Mikhail V wrote: > > Well, then there is some bitter irony in this, so it allows pretty > > much everything, > > but does not allow me to beautify code with hyphens. > > I can fully understand the wish to use non-latin scripts in strings or > comments. > > As for identifiers, IMO, apart from latin letters and underscore, the > > first unicode candidate > > I would add is U+2010. And probably the LAST one I would add. > > > > Fortunately for the world, you're not the one who decided which > characters were permitted in Python identifiers. The ability to use > non-English words for function/variable names is of huge value; the > ability to use a hyphen is of some value, but not nearly as much. > > Can this thread move to python-list? Or, better, to > python-rants-about-unicode-list, to which I don't subscribe? > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephanh42 at gmail.com Thu Nov 23 09:41:25 2017 From: stephanh42 at gmail.com (Stephan Houben) Date: Thu, 23 Nov 2017 15:41:25 +0100 Subject: [Python-ideas] Allow additional separator character in variables In-Reply-To: References: <23059.34466.1253.737550@turnbull.sk.tsukuba.ac.jp> <23060.5173.899290.61724@turnbull.sk.tsukuba.ac.jp> Message-ID: US English is too broad. I propose everybody talk like in the "Dallas" TV series and wear mandatory cowboy hats. (Or was this mail just a dream?) Stephan 2017-11-23 15:29 GMT+01:00 Carl Smith : > Can't we just tell everyone to speak US English, and go back to ASCII? It > would be a less painful migration. > > -- Carl Smith > carl.input at gmail.com > > On 23 November 2017 at 14:16, Chris Angelico wrote: > >> On Fri, Nov 24, 2017 at 1:10 AM, Mikhail V wrote: >> > Well, then there is some bitter irony in this, so it allows pretty >> > much everything, >> > but does not allow me to beautify code with hyphens. >> > I can fully understand the wish to use non-latin scripts in strings or >> comments. >> > As for identifiers, IMO, apart from latin letters and underscore, the >> > first unicode candidate >> > I would add is U+2010. And probably the LAST one I would add. >> > >> >> Fortunately for the world, you're not the one who decided which >> characters were permitted in Python identifiers. The ability to use >> non-English words for function/variable names is of huge value; the >> ability to use a hyphen is of some value, but not nearly as much. >> >> Can this thread move to python-list? Or, better, to >> python-rants-about-unicode-list, to which I don't subscribe? >> >> ChrisA >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Thu Nov 23 17:49:14 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 24 Nov 2017 09:49:14 +1100 Subject: [Python-ideas] Allow additional separator character in variables In-Reply-To: References: <23059.34466.1253.737550@turnbull.sk.tsukuba.ac.jp> <23060.5173.899290.61724@turnbull.sk.tsukuba.ac.jp> Message-ID: <20171123224914.GB22248@ando.pearwood.info> On Thu, Nov 23, 2017 at 02:29:43PM +0000, Carl Smith wrote: > Can't we just tell everyone to speak US English, and go back to ASCII? It > would be a less painful migration. I trust you're joking, but it makes me twitchy to see people saying that even in jest, because I've come across folks who *actually do* think that way. -- Steve From steve at pearwood.info Thu Nov 23 19:49:00 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 24 Nov 2017 11:49:00 +1100 Subject: [Python-ideas] Allow additional separator character in variables In-Reply-To: <23062.23424.677384.854676@turnbull.sk.tsukuba.ac.jp> References: <23059.34466.1253.737550@turnbull.sk.tsukuba.ac.jp> <23060.5173.899290.61724@turnbull.sk.tsukuba.ac.jp> <23062.23424.677384.854676@turnbull.sk.tsukuba.ac.jp> Message-ID: <20171124004900.GC22248@ando.pearwood.info> On Thu, Nov 23, 2017 at 02:24:16PM +0900, Stephen J. Turnbull wrote: > Alex Martelli wrote a couple of interesting posts about his > experiences with multilingual comments back in the discussion of PEP > 263. One of them involved a team from Israel, I think, or maybe South > Africa. If you Google site:mail.python.org for Alex and those > countries, the thread would probably come up. Either my google-fu is failing or your memory failed you. I've spent an hour and a half googling, and I'm getting nothing relevant. It doesn't help that Alex was a very prolific poster back in the day. Hell, I can't even find where PEP 263 was discussed, apart from a brief discussion here: https://mail.python.org/pipermail/python-dev/2002-July/026449.html which Alex didn't take part in. (And that was a year after the PEP was first created.) https://www.python.org/dev/peps/pep-0263/ -- Steve From AlonSnir at hotmail.com Thu Nov 23 07:48:02 2017 From: AlonSnir at hotmail.com (Alon Snir) Date: Thu, 23 Nov 2017 12:48:02 +0000 Subject: [Python-ideas] Rewriting the "roundrobin" recipe in the itertools documentation Message-ID: Sorry. Unfortunately I had a mistake. Alon Snir Date: Thu, 23 Nov 2017 12:29:47 +1100 From: Steven D'Aprano To: python-ideas at python.org Subject: Re: [Python-ideas] Rewriting the "roundrobin" recipe in the itertools documentation Message-ID: <20171123012947.GT19802 at ando.pearwood.info> Content-Type: text/plain; charset=utf-8 On Wed, Nov 22, 2017 at 11:28:20AM +0000, Alon Snir wrote: > It would be faster with ?deque?: It isn't. According to my testing, your version with deque is approximately two times slower than the version from toolz.itertoolz that Wes quotes. -- Steve From python at mrabarnett.plus.com Thu Nov 23 20:41:32 2017 From: python at mrabarnett.plus.com (MRAB) Date: Fri, 24 Nov 2017 01:41:32 +0000 Subject: [Python-ideas] Allow additional separator character in variables In-Reply-To: <20171124004900.GC22248@ando.pearwood.info> References: <23059.34466.1253.737550@turnbull.sk.tsukuba.ac.jp> <23060.5173.899290.61724@turnbull.sk.tsukuba.ac.jp> <23062.23424.677384.854676@turnbull.sk.tsukuba.ac.jp> <20171124004900.GC22248@ando.pearwood.info> Message-ID: <41905288-ff8c-5d0c-7679-201863bc7637@mrabarnett.plus.com> On 2017-11-24 00:49, Steven D'Aprano wrote: > On Thu, Nov 23, 2017 at 02:24:16PM +0900, Stephen J. Turnbull wrote: > >> Alex Martelli wrote a couple of interesting posts about his >> experiences with multilingual comments back in the discussion of PEP >> 263. One of them involved a team from Israel, I think, or maybe South >> Africa. If you Google site:mail.python.org for Alex and those >> countries, the thread would probably come up. > > Either my google-fu is failing or your memory failed you. I've spent an > hour and a half googling, and I'm getting nothing relevant. It doesn't > help that Alex was a very prolific poster back in the day. > > Hell, I can't even find where PEP 263 was discussed, apart from a > brief discussion here: > > https://mail.python.org/pipermail/python-dev/2002-July/026449.html > > which Alex didn't take part in. (And that was a year after the PEP was > first created.) > > https://www.python.org/dev/peps/pep-0263/ > I think your search might have been hindered by your ability to spell. :-) Multibyte Character Surport for Python http://grokbase.com/t/python/python-list/0258fms6xa/multibyte-character-surport-for-python From gadgetsteve at live.co.uk Fri Nov 24 01:25:01 2017 From: gadgetsteve at live.co.uk (Steve Barnes) Date: Fri, 24 Nov 2017 06:25:01 +0000 Subject: [Python-ideas] Possible Enhancement to py.exe Python Launcher In-Reply-To: References: Message-ID: Thanks Paul, I had missed that section of PEP-0397 - the only options that are not covered are loading a local configuration from the current directory and the option to set a prompt/error if the python version is not specified on the command line or in the shebang. Sorry for the noise! Steve On 23/11/2017 09:53, Paul Moore wrote: > If I understand your proposal correctly, this is already possible, see > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.python.org%2Fdev%2Fpeps%2Fpep-0397%2F%23python-version-qualifiers&data=02%7C01%7Cgadgetsteve%40live.co.uk%7C6e858560c6504120febe08d532581be5%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636470276358104937&sdata=mgZCUtbRoEU6A0JHMoI1kvC49kPkWy7TYP6LgmCUCn0%3D&reserved=0 > > The details are likely a little different than what you're proposing, > but if they don't cover what you're trying to do, maybe you could give > a more specific example of a use case that isn't currently possible? > > Paul > > On 23 November 2017 at 08:43, Steve Barnes wrote: >> Following on from the discussions on pip I would like to suggest, (and >> possibly implement), a minor change to the current py.exe python launcher. >> >> Currently the launcher, when called without a version specifier, >> defaults to the highest version on the basis of 3>2, x.11 > x.9, -64 > >> -32 however this may not always be the most desirable behaviour. >> >> I would like to suggest that it take a configuration value for the >> default version to use, (and possibly a separate ones for pyw, & file >> associations), honouring the following with the later overriding: >> >> default - as current >> system setting - from registry >> user setting - from registry >> user setting - from config file maybe %appdata%\pylauncher\defaults.ini >> environment setting - from getenv >> local setting - from .pyconfig file in the current directory. >> >> Options would be the same format as the -X[.Y][-BB] format currently >> accepted on the command line plus a --No-Py-Default option which would >> always error out if the version to invoke was not specified. >> >> I see this as potentially adding quite a lot of value for people with >> multiple python versions installed and it could tie in quite well with >> the possible use of py.exe as an entry point for tools such as pip. >> >> It might also increase the awareness of the launcher as those who have >> to stick with python 2 for the moment or in a specific context could set >> the default to what they need but can always override. >> >> -- >> Steve (Gadget) Barnes >> Any opinions in this message are my personal opinions and do not reflect >> those of my employer. >> >> --- >> This email has been checked for viruses by AVG. >> https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.avg.com&data=02%7C01%7Cgadgetsteve%40live.co.uk%7C6e858560c6504120febe08d532581be5%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636470276358104937&sdata=QPEFfMHbB9ez60d%2F7dMcf59kvooIz0WBfCpHFseti4M%3D&reserved=0 >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.python.org%2Fmailman%2Flistinfo%2Fpython-ideas&data=02%7C01%7Cgadgetsteve%40live.co.uk%7C6e858560c6504120febe08d532581be5%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636470276358104937&sdata=DacUrxgfyFgQCf0DLPy5IQUeN%2BhRrW3tVXjkjLKfXz0%3D&reserved=0 >> Code of Conduct: https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fpython.org%2Fpsf%2Fcodeofconduct%2F&data=02%7C01%7Cgadgetsteve%40live.co.uk%7C6e858560c6504120febe08d532581be5%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636470276358104937&sdata=dP1wkYgGYSpyP3M7QmRA7mlVt0O6HJIvIowh7UlX0vQ%3D&reserved=0 -- Steve (Gadget) Barnes Any opinions in this message are my personal opinions and do not reflect those of my employer. --- This email has been checked for viruses by AVG. http://www.avg.com From mikhailwas at gmail.com Sat Nov 25 19:49:06 2017 From: mikhailwas at gmail.com (Mikhail V) Date: Sun, 26 Nov 2017 01:49:06 +0100 Subject: [Python-ideas] Allow additional separator character in variables In-Reply-To: <23062.23424.677384.854676@turnbull.sk.tsukuba.ac.jp> References: <23059.34466.1253.737550@turnbull.sk.tsukuba.ac.jp> <23060.5173.899290.61724@turnbull.sk.tsukuba.ac.jp> <23062.23424.677384.854676@turnbull.sk.tsukuba.ac.jp> Message-ID: Just for the record, there is also another hyphen, called "soft hyphen", U+00AD. Main difference is that in some software it is an 'interpreted' symbol, and thus may simply disappear from the screen in such software, so it cannot be surely defined as a printable character. OTOH the benefit is that it is 100% present in any font, afaik. I have found a good technical summary about this character: http://jkorpela.fi/shy.html From turnbull.stephen.fw at u.tsukuba.ac.jp Sun Nov 26 23:42:54 2017 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Mon, 27 Nov 2017 13:42:54 +0900 Subject: [Python-ideas] Allow additional separator character in variables In-Reply-To: <20171124004900.GC22248@ando.pearwood.info> References: <23059.34466.1253.737550@turnbull.sk.tsukuba.ac.jp> <23060.5173.899290.61724@turnbull.sk.tsukuba.ac.jp> <23062.23424.677384.854676@turnbull.sk.tsukuba.ac.jp> <20171124004900.GC22248@ando.pearwood.info> Message-ID: <23067.38862.209271.403004@turnbull.sk.tsukuba.ac.jp> Steven D'Aprano writes: > Either my google-fu is failing or your memory failed you. I'm still pretty sure that it's the google-fu, but I confess my google-fu failed me too. In either case, you have my deepest apologies for the unprofitable hour-and-a-half. Steve From kirillbalunov at gmail.com Mon Nov 27 04:17:31 2017 From: kirillbalunov at gmail.com (Kirill Balunov) Date: Mon, 27 Nov 2017 12:17:31 +0300 Subject: [Python-ideas] How assignment should work with generators? Message-ID: Currently during assignment, when target list is a comma-separated list of targets (*without "starred" target*) the rule is that the object (rhs) must be an iterable with the same number of items as there are targets in the target list. That is, no check is performed on the number of targets present, and if something goes wrong the ValueError is raised. To show this on simple example: >>> from itertools import count, islice >>> it = count() >>> x, y = it >>> it count(3) Here the count was advanced two times but assignment did not happen. I found that in some cases it is too much restricting that rhs must have the same number of items as targets. It is proposed that if the rhs is a generator or an iterator (better some object that yields values on demand), the assignmenet should be lazy and dependent on the number of targets. I find this feature to be very convenient for interactive use, while it remains readable, expected, and expressed in a more compact code. There are some Pros: 1. No overhead 2. Readable and not so verbose code 3. Optimized case for x,y,*z = iterator 4. Clear way to assign values partially from infinite generators. Cons: 1. A special case of how assignment works 2. As with any implicit behavior, hard-to-find bugs There several cases with "undefined" behavior: 1. Because the items are assigned, from left to right to the corresponding targets, should rhs see side effects during assignment or not? 2. Should this work only for generators or for any iterators? 3. Is it Pythonic to distinguish what is on the rhs during assignment, or it contradicts with duck typing (goose typing)? In many cases it is possible to do this right now, but in too verbose way: >>> x, y = islice(gen(), 2) But it seems for me that: >>> x, y = gen() Looks the same, easy to follow and not so verbose. Another case, which is a pitfall, but will be possible: >>> x, y = iter([0,1,2,3,4]) # Now it is Ok >>> x 1 >>> y 2 But: >>> x, y = [0,1,2,3,4] # raises ValueError Any thoughts? With kind regards, -gdg -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Mon Nov 27 04:40:21 2017 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 27 Nov 2017 20:40:21 +1100 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: References: Message-ID: On Mon, Nov 27, 2017 at 8:17 PM, Kirill Balunov wrote: > In many cases it is possible to do this right now, but in too verbose way: > >>>> x, y = islice(gen(), 2) > > But it seems for me that: > >>>> x, y = gen() > > Looks the same, easy to follow and not so verbose. This, AIUI, is the nub of the proposal. I don't like the proposed solution, but if there's some alternative way to say "and then ignore the rest", that would be much safer. You said: > 3. Optimized case for x,y,*z = iterator It has to be semantically different from that, though. Consider these two generators: def gen1(): for _ in range(5): print("Yielding!") yield "spam" yield "ham" def gen2(): yield 1 yield 2 yield 3 while True: yield 4 If you use "x, y, *z = gen1()", you'll trigger all the prints and completely consume the generator. With gen2(), you'll get an infinite loop. Both of those are semantically different from the islice behaviour, which would consume only that part that you're looking for. What would be far safer is a special syntax that makes it clear that you are not assigning those extra values anywhere. Something along these lines has been proposed a number of times, and I think it's probably about time a PEP was written up, if only to get rejected. Here are a few syntaxes that I believe have been proposed at various times: x, y = islice(iter, 2) # status quo x, y = iter # your proposal x, y, = iter # omit last destination x, y, * = iter # unpack into nothing x, y, ... = iter # assigning to Ellipsis x, y, *... = iter # as above but clearly sequencing And there are a few others too. Every one of them has its downsides, none is perfect. (Personally, I think one of the Ellipsis options is likely the best, or perhaps the least-bad.) Want to spearhead the PEP? ChrisA From kirillbalunov at gmail.com Mon Nov 27 05:45:20 2017 From: kirillbalunov at gmail.com (Kirill Balunov) Date: Mon, 27 Nov 2017 13:45:20 +0300 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: References: Message-ID: 2017-11-27 12:40 GMT+03:00 Chris Angelico : > On Mon, Nov 27, 2017 at 8:17 PM, Kirill Balunov > wrote: > > In many cases it is possible to do this right now, but in too verbose > way: > > > >>>> x, y = islice(gen(), 2) > > > > But it seems for me that: > > > >>>> x, y = gen() > > > > Looks the same, easy to follow and not so verbose. > > This, AIUI, is the nub of the proposal. Yes, it is. For me, x,y = gen() underlines the lazy nature of generators. > I don't like the proposed > solution, but if there's some alternative way to say "and then ignore > the rest", that would be much safer. > Of course, everyone has a subjective assessment of what is good and what is bad. But here I am in something agree with you that at the present time some alternative way would be much safer. But if started from scratch, it seems to me natural to emphasize the nature of generators and iterators. > You said: > > > 3. Optimized case for x,y,*z = iterator > > It has to be semantically different from that, though. Consider these > two generators: > > def gen1(): > for _ in range(5): > print("Yielding!") > yield "spam" > yield "ham" > > def gen2(): > yield 1 > yield 2 > yield 3 > while True: > yield 4 > > If you use "x, y, *z = gen1()", you'll trigger all the prints and > completely consume the generator. With gen2(), you'll get an infinite > loop. Both of those are semantically different from the islice > behaviour, which would consume only that part that you're looking for. The idea is not to consume generator completely, but to get enough values to bind to x and y. It should be equivalent to islice(iter, 2), and perceive? as "bind x, y, where you don't need values for z at all". x, y = islice(iter, 2) # status quo > x, y = iter # your proposal > As I wrote above, I would like to see them equivalent x, y, = iter # omit last destination > I don't like this, it is hard to notice this nuance. It is valid to have trailing comma (and I find it to be a brilliant feature to have (x,y) equivalent to (x,y,)). This reminds me of ` in Python2, although I never used it and start my journey with Python3. > x, y, * = iter # unpack into nothing > Maybe, I like it. > x, y, ... = iter # assigning to Ellipsis > x, y, *... = iter # as above but clearly sequencing > Yes, it is nice to see Ellipsis as zero-length deques (or throw away container), it can be used also in the middle of list of targets. What I don't like about last three examples, that they imply by their form to be some kind of iteration which must completely consume the generator, which is opposite to the proposed idea. Moreover, this will not work with infinite generators, falling into infinite loop. > And there are a few others too. Every one of them has its downsides, > none is perfect. (Personally, I think one of the Ellipsis options is > likely the best, or perhaps the least-bad.) Want to spearhead the PEP? > To be honest, I'm quite new to the entire ecosystem of Python. Therefore, someone can perceive this as an ordinary attempt by a novice to change everything that is bad in his opinion. In addition, it seems to me that these changes, if approved, can not be made earlier than Python3.8, so I would like to get some feedback first. Nevertheless these words should not be taken as a renouncement, I would be happy and very interested in writing PEP and implementing it. But not to get stuck I need some support and guidelines from interested dev. With kind regards, -gdg -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Mon Nov 27 06:02:53 2017 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 27 Nov 2017 22:02:53 +1100 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: References: Message-ID: On Mon, Nov 27, 2017 at 9:45 PM, Kirill Balunov wrote: > 2017-11-27 12:40 GMT+03:00 Chris Angelico : >> If you use "x, y, *z = gen1()", you'll trigger all the prints and >> completely consume the generator. With gen2(), you'll get an infinite >> loop. Both of those are semantically different from the islice >> behaviour, which would consume only that part that you're looking for. > > > The idea is not to consume generator completely, but to get enough values to > bind to x and y. It should be equivalent to islice(iter, 2), and perceive? > as "bind x, y, where you don't need values for z at all". In terms of language proposals, you can't just say "don't need values for"; the semantics have to be EITHER "consume and discard" OR "don't consume". We already have a perfectly good way of spelling "consume and discard": x, y, _ = iter following the convention that a single underscore means "don't really care" (eg "for _ in range(3)"). So this proposal is about not consuming an iterator. >> x, y, * = iter # unpack into nothing > > Maybe, I like it. > >> x, y, ... = iter # assigning to Ellipsis >> x, y, *... = iter # as above but clearly sequencing > > > Yes, it is nice to see Ellipsis as zero-length deques (or throw away > container), it can be used also in the middle of list of targets. What I > don't like about last three examples, that they imply by their form to be > some kind of iteration which must completely consume the generator, which is > opposite to the proposed idea. Moreover, this will not work with infinite > generators, falling into infinite loop. Since this has to be about non-consumption of the generator/iterator, Ellipsis cannot be a zero-length deque. Thus this syntax would have to be restricted to the *last* entry, and it then means "don't check for more elements". The assignment "x, y = it" is roughly equivalent to: try: _iter = iter(it) x = next(_iter) y = next(_iter) except StopIteration: raise ValueError else: try: next(_iter) except StopIteration: pass # good, we got the right number of elements else: raise ValueError The proposed semantics, if I understand you correctly, are: try: _iter = iter(it) x = next(_iter) y = next(_iter) except StopIteration: raise ValueError # no "else" clause, we're done here And I think this would be a good thing to be able to spell conveniently. The only questions are: what spelling is the best, and is it sufficiently useful to justify the syntax? >> And there are a few others too. Every one of them has its downsides, >> none is perfect. (Personally, I think one of the Ellipsis options is >> likely the best, or perhaps the least-bad.) Want to spearhead the PEP? > > To be honest, I'm quite new to the entire ecosystem of Python. Therefore, > someone can perceive this as an ordinary attempt by a novice to change > everything that is bad in his opinion. In addition, it seems to me that > these changes, if approved, can not be made earlier than Python3.8, so I > would like to get some feedback first. Nevertheless these words should not > be taken as a renouncement, I would be happy and very interested in writing > PEP and implementing it. But not to get stuck I need some support and > guidelines from interested dev. Yes; now that we're into alphas for Python 3.7, it's not going to land until 3.8. That's fine. The PEP process is basically a way of gathering all the arguments for-and-against into a single, coherent document, rather than having them scattered all over the python-ideas email archive. The PEP author has the final say on what is the thrust of the proposal, and then Guido (or his delegate) will decide to accept or reject the PEP. If it's accepted, the change will be made; if it's not, the PEP remains in the repository as a permanent record of the proposal. That way, the *next* person to come along and suggest something can pick up from where this discussion left off, rather than starting fresh. Start by perusing PEP 1, and the template in PEP 12: https://www.python.org/dev/peps/pep-0001/ https://www.python.org/dev/peps/pep-0012/ The PEP editors (myself included) are here to help you; don't hesitate to reach out with questions. ChrisA From kirillbalunov at gmail.com Mon Nov 27 07:31:38 2017 From: kirillbalunov at gmail.com (Kirill Balunov) Date: Mon, 27 Nov 2017 15:31:38 +0300 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: References: Message-ID: > In terms of language proposals, you can't just say "don't need values > for"; the semantics have to be EITHER "consume and discard" OR "don't > consume". We already have a perfectly good way of spelling "consume > and discard": > > x, y, _ = iter > You mean ( x, y, *_ = iter ) ? Since this has to be about non-consumption of the generator/iterator, > Ellipsis cannot be a zero-length deque. Thus this syntax would have to > be restricted to the *last* entry, and it then means "don't check for > more elements". > Yes, you are right to the *last* entry. (*last* depends on proposed syntax (spelling)). > The proposed semantics, if I understand you correctly, are: > > try: > _iter = iter(it) > x = next(_iter) > y = next(_iter) > except StopIteration: > raise ValueError > # no "else" clause, we're done here > Yes, "roughly" this semantics is proposed, with some assumptions on _iter = iter(it). As I can see at the moment, these cases should behave differently: >>> x, y = [1,2,3,4] # must raise ValueError >>> x, y = iter([1,2,3,4]) # should work But at the same time, it violates current situation. So maybe, as you have said we need special syntax. I will think about it. > Start by perusing PEP 1, and the template in PEP 12: > > https://www.python.org/dev/peps/pep-0001/ > https://www.python.org/dev/peps/pep-0012/ > > The PEP editors (myself included) are here to help you; don't hesitate > to reach out with questions. > Thank you! With kind regards, -gdg -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Mon Nov 27 07:39:43 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 27 Nov 2017 12:39:43 +0000 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: References: Message-ID: On 27 November 2017 at 12:31, Kirill Balunov wrote: > As I can see at the moment, these cases should behave differently: > >>>> x, y = [1,2,3,4] # must raise ValueError >>>> x, y = iter([1,2,3,4]) # should work > > But at the same time, it violates current situation. So maybe, as you have > said we need special syntax. I will think about it. I would find this confusing. Consider where you don't have literals: def f(vals): x, y = vals data = [1,2,3,4] f(data) data = iter(data) f(data) Having the two calls behave differently would be a recipe for errors as someone refactors the calling code. Paul From kirillbalunov at gmail.com Mon Nov 27 07:59:38 2017 From: kirillbalunov at gmail.com (Kirill Balunov) Date: Mon, 27 Nov 2017 15:59:38 +0300 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: References: Message-ID: 2017-11-27 15:39 GMT+03:00 Paul Moore : > On 27 November 2017 at 12:31, Kirill Balunov > wrote: > > As I can see at the moment, these cases should behave differently: > > > >>>> x, y = [1,2,3,4] # must raise ValueError > >>>> x, y = iter([1,2,3,4]) # should work > > > > But at the same time, it violates current situation. So maybe, as you > have > > said we need special syntax. I will think about it. > > I would find this confusing. Consider where you don't have literals: > > def f(vals): > x, y = vals > > data = [1,2,3,4] > > f(data) > data = iter(data) > f(data) > > Having the two calls behave differently would be a recipe for errors > as someone refactors the calling code. > I can not completely disagree with you, but we all adults here. My first proposal was about generators only, but they are very similar to iterators in their behavior. Whatever it was with this syntax, there will be no difference: def f(vals): x, y = vals data = [1,2,3,4] f(data) data = (i for i in data) f(data) With kind regards, -gdg -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Mon Nov 27 08:47:45 2017 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 28 Nov 2017 00:47:45 +1100 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: References: Message-ID: On Mon, Nov 27, 2017 at 11:31 PM, Kirill Balunov wrote: > >> In terms of language proposals, you can't just say "don't need values >> for"; the semantics have to be EITHER "consume and discard" OR "don't >> consume". We already have a perfectly good way of spelling "consume >> and discard": >> >> x, y, _ = iter > > > You mean ( x, y, *_ = iter ) ? Uhh, yeah, that's what I meant. Sorry. Anyhow, point is, there IS a syntax for that, so we don't need another. >> The proposed semantics, if I understand you correctly, are: >> >> try: >> _iter = iter(it) >> x = next(_iter) >> y = next(_iter) >> except StopIteration: >> raise ValueError >> # no "else" clause, we're done here > > > Yes, "roughly" this semantics is proposed, with some assumptions on _iter = > iter(it). > As I can see at the moment, these cases should behave differently: > >>>> x, y = [1,2,3,4] # must raise ValueError >>>> x, y = iter([1,2,3,4]) # should work > > But at the same time, it violates current situation. So maybe, as you have > said we need special syntax. I will think about it. That's the part I disagree with, but if you're the PEP author, you can make the recommendation be anything you like. However, one very strong piece of advice: it's easier to get a proposal accepted if the backward compatibility section simply says "the proposed notation is a SyntaxError in current versions of Python". Changing the semantics of currently legal code requires that you demonstrate that the current semantics are, in some way, faulty or buggy. ChrisA From dmoisset at machinalis.com Mon Nov 27 08:44:28 2017 From: dmoisset at machinalis.com (Daniel Moisset) Date: Mon, 27 Nov 2017 10:44:28 -0300 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: References: Message-ID: On 27 November 2017 at 06:40, Chris Angelico wrote: > Here are a few syntaxes that I believe have been proposed at various > times: > > x, y = islice(iter, 2) # status quo > x, y = iter # your proposal > x, y, = iter # omit last destination > Just to clear the list, this one (trailing comma) would be ambiguous/backward incompatible for the 1 variable case: x, = iter which is a relatively common idiom and is expected to raise an error if the iterator has trailing elements. -- Daniel F. Moisset - UK Country Manager - Machinalis Limited www.machinalis.co.uk Skype: @dmoisset T: + 44 7398 827139 1 Fore St, London, EC2Y 9DT Machinalis Limited is a company registered in England and Wales. Registered number: 10574987. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon Nov 27 08:55:05 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 28 Nov 2017 00:55:05 +1100 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: References: Message-ID: <20171127135502.GF22248@ando.pearwood.info> On Mon, Nov 27, 2017 at 12:17:31PM +0300, Kirill Balunov wrote: > Currently during assignment, when target list is a comma-separated list of > targets (*without "starred" target*) the rule is that the object (rhs) must > be an iterable with the same number of items as there are targets in the > target list. That is, no check is performed on the number of targets > present, and if something goes wrong the ValueError is raised. That's a misleading description: ValueError is raised when the number of targets is different from the number of items. I consider that to be performing a check on the number of targets. > To show this on simple example: > > >>> from itertools import count, islice > >>> it = count() > >>> x, y = it > >>> it > count(3) For everyone else who was confused by this, as I was, that's not actually a copy and paste from the REPL. There should be a ValueError raised after the x, y assignment. As given, it is confusing because it looks like the assignment succeeded, when in fact it didn't. > Here the count was advanced two times but assignment did not happen. Correct, because there was an exception raised. > I found that in some cases it is too much restricting that rhs must > have the same number of items as targets. It is proposed that if the > rhs is a generator or an iterator (better some object that yields > values on demand), the assignmenet should be lazy and dependent on the > number of targets. I think that's problematic. How do you know what objects that yields values on demand? Not all lazy iterables are iterators: there are also lazy sequences like range. But even if we decide on a simple rule like "iterator unpacking depends on the number of targets, all other iterables don't", I think that will be a bug magnet. It will mean that you can't rely on this special behaviour unless you surround each call with a type check: if isinstance(it, collections.abc.Iterator): # special case for iterators x, y = it else: # sequences keep the old behaviour x, y = it[:2] > I find this feature to be very convenient for > interactive use, There are many things which would be convenient for interactive use that are a bad idea outside of the interactive environment. Errors which pass silently are one of them. Unpacking a sequence of 3 items into 2 assignment targets should be an error, unless you explicitly limit it to only two items. Sure, sometimes it would be convenient to unpack just two items out of some arbitrarily large iterator just be writing `x, y = it`. But other times that would be an error, even in the interactive interpreter. I don't want Python trying to *guess* whether I want to unpack the entire iteratable or just two items. Whatever tiny convenience there is from when Python guesses correctly will be outweighed by the nuisance value of when it guesses wrongly. > while it remains readable, expected, and expressed in a more compact > code. I don't think it is expected behaviour. It is different from the current behaviour, so it will be surprising to everyone used to the current behaviour, annoying to those who like the current behaviour, and a general inconvenience to those writing code that runs under multiple versions of Python. Personally, I would not expect this suggested behaviour. I would be very surprised, and annoyed, if a simple instruction like: x, y = some_iterable behaved differently for iterators and sequences. > There are some Pros: > 1. No overhead No overhead compared to what? > 2. Readable and not so verbose code > 3. Optimized case for x,y,*z = iterator The semantics of that are already set: the first two items are assigned to x and y, with all subsequent items assigned to z as a list. How will this change optimize this case? It still needs to run through the iterator to generate the list. > 4. Clear way to assign values partially from infinite generators. It isn't clear at all. If I have a non-generator lazy sequence like: # Toy example class EvenNumbers: def __getitem__(self, i): return 2*i it = EvenNumbers() # A lazy, infinite sequence then `x, y = it` will keep the current behaviour and raise an exception (since it isn't an iterator), but `x, y = iter(it)` will use the new behaviour. So in general, when I'm reading code and I see: x, y = some_iterable I have very little idea of which behaviour will apply. Will it be the special iterator behaviour that stops at two items, or the current sequence behaviour that raises if there are more than two items? > Cons: > 1. A special case of how assignment works > 2. As with any implicit behavior, hard-to-find bugs Right. Hard-to-find bugs beats any amount of convenience in the interactive interpreter. To use an analogy: "Sure, sometimes my car suddenly shifts into reverse while I'm driving at 60 kph, sometimes the engine falls out when I go around the corner, and occasionally the brakes catch fire, but gosh the cup holder makes it really convenient to drink coffee while I'm stopped at traffic lights!" > There several cases with "undefined" behavior: > 1. Because the items are assigned, from left to right to the corresponding > targets, should rhs see side effects during assignment or not? I don't understand what you mean by this. Surely the behaviour should be exactly the same as if you wrote: x, y = islice(it, 2) What would you do differently, and why? > 2. Should this work only for generators or for any iterators? I don't understand why you are even considering singling out *only* generators. A generator is a particular implementation of an iterator. I can write: def gen(): yield 1; yield 2; yield 3 it = gen() or I can write: it = iter([1, 2, 3]) and the behaviour of `it` should be identical. > 3. Is it Pythonic to distinguish what is on the rhs during assignment, or > it contradicts with duck typing (goose typing)? I don't understand this question. > In many cases it is possible to do this right now, but in too verbose way: > > >>> x, y = islice(gen(), 2) I don't think that is excessively verbose. But maybe we should consider allowing slice notation on arbitrary iterators: x, y = it[:2] I have not thought this through in any serious detail, but it seems to me that if the only problem here is the inconvenience of using islice(), we could add slicing to iterators. I think that would be better than having iterators and other iterables behave differently. Perhaps a better idea might be special syntax to tell the interpreter you don't want to run the right-hand side to completion. "Explicit is better than implicit" -- maybe something special like: x, y, * = iterable will attempt to extract exactly two items from iterable, without advancing past the second item. And it could work the same for sequences, iterators, lazy sequences like range, and any other iterable. I don't love having yet another meaning for * but that would be better than changing the standard behaviour of iterator unpacking. -- Steve From rosuav at gmail.com Mon Nov 27 09:01:01 2017 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 28 Nov 2017 01:01:01 +1100 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: References: Message-ID: On Tue, Nov 28, 2017 at 12:44 AM, Daniel Moisset wrote: > On 27 November 2017 at 06:40, Chris Angelico wrote: >> >> Here are a few syntaxes that I believe have been proposed at various >> times: >> >> x, y = islice(iter, 2) # status quo >> x, y = iter # your proposal >> x, y, = iter # omit last destination > > > Just to clear the list, this one (trailing comma) would be > ambiguous/backward incompatible for the 1 variable case: > > x, = iter > > which is a relatively common idiom and is expected to raise an error if the > iterator has trailing elements. Correct. For that and other reasons, I am not in favour of either of these two proposals. And the status quo is noisy and has duplicated information (you have to match the ", 2" to the number of assignment targets). I would support any syntax that (a) is currently illegal, (b) reads reasonably well, and (c) can't be TOO easily confused with something else. Assigning to Ellipsis (with or without a star) is my current preferred, but I'd happily support others that do the same job. ChrisA From steve at pearwood.info Mon Nov 27 09:13:14 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 28 Nov 2017 01:13:14 +1100 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: References: Message-ID: <20171127141314.GG22248@ando.pearwood.info> On Mon, Nov 27, 2017 at 03:31:38PM +0300, Kirill Balunov wrote: > As I can see at the moment, these cases should behave differently: > > >>> x, y = [1,2,3,4] # must raise ValueError > >>> x, y = iter([1,2,3,4]) # should work I *completely disagree* that they should behave differently. That would be a radical change to the current equivalency between iterators and other iterables. Of course iterators support next() (and a few other things), while iterables (sequences and others) support slicing, __getitem__, and so forth. But when it comes to iteration, they behave exactly the same in all ways that I can think of: for x in iterable: ... list(iterable) iter(iterable) func(*iterable) and most importantly for this discussion, iterable unpacking: a, b, c = *iterable They all work the same, regardless of whether `iterable` is an iterator, a generator, a list, a tuple, a range object, a custom lazy sequence. Sure, there are a few differences: iterators generally cannot be restarted or rewound, while lazy sequences might be, and eager sequences like lists can be. You can't peek into an arbitrary iterator without consuming the value. But as far as iteration itself goes, they are all the same. -- Steve From rosuav at gmail.com Mon Nov 27 09:14:40 2017 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 28 Nov 2017 01:14:40 +1100 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: <20171127135502.GF22248@ando.pearwood.info> References: <20171127135502.GF22248@ando.pearwood.info> Message-ID: On Tue, Nov 28, 2017 at 12:55 AM, Steven D'Aprano wrote: > But even if we decide on a simple rule like "iterator unpacking depends > on the number of targets, all other iterables don't", I think that will > be a bug magnet. It will mean that you can't rely on this special > behaviour unless you surround each call with a type check: > > > if isinstance(it, collections.abc.Iterator): > # special case for iterators > x, y = it > else: > # sequences keep the old behaviour > x, y = it[:2] Nah, far easier: x, y = iter(it) since that'll be a no-op in the first case, and trigger new behaviour in the second. However, I don't like this behaviour-switch. I'd much rather have actual syntax. > I don't want Python trying to *guess* whether I want to unpack the > entire iteratable or just two items. Whatever tiny convenience there is > from when Python guesses correctly will be outweighed by the nuisance > value of when it guesses wrongly. Exactly. >> There are some Pros: >> 1. No overhead > > No overhead compared to what? I think the point here (correct me if I'm wrong?) is that it takes work to probe the iterator to see if there's a third item, so grabbing just the first two items is simply *doing less work*. It's not doing MORE work (constructing an islice object, pumping it, then discarding it) - it's simply skipping the check that it would otherwise do. >> 2. Readable and not so verbose code >> 3. Optimized case for x,y,*z = iterator > > The semantics of that are already set: the first two items are assigned > to x and y, with all subsequent items assigned to z as a list. How will > this change optimize this case? It still needs to run through the > iterator to generate the list. Maybe 'optimized case for "x, y, *_ = iterator" where you then never use _ and it has no side effects'? But that could be worded better. >> In many cases it is possible to do this right now, but in too verbose way: >> >> >>> x, y = islice(gen(), 2) > > I don't think that is excessively verbose. > > But maybe we should consider allowing slice notation on arbitrary > iterators: > > x, y = it[:2] I do think islice is verbose, but the main problem is that you have to match the second argument to the number of assignment targets. Slice notation is an improvement, but it still has that same problem. But perhaps this should be added to the list of options for the PEP. > Perhaps a better idea might be special syntax to tell the interpreter > you don't want to run the right-hand side to completion. "Explicit is > better than implicit" -- maybe something special like: > > x, y, * = iterable > > will attempt to extract exactly two items from iterable, without > advancing past the second item. And it could work the same for > sequences, iterators, lazy sequences like range, and any other iterable. > > I don't love having yet another meaning for * but that would be better > than changing the standard behaviour of iterator unpacking. That's one of the options that I mentioned, as it's been proposed in the past. The problem is that it depends on internal whitespace to distinguish it from augmented assignment; granted, there's no way to use "*=" with multiple targets (or even in the single-target case, you can't do "x,*=it" with the comma in it), but that's still a readability problem. ChrisA From kirillbalunov at gmail.com Mon Nov 27 09:35:25 2017 From: kirillbalunov at gmail.com (Kirill Balunov) Date: Mon, 27 Nov 2017 17:35:25 +0300 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: References: <20171127135502.GF22248@ando.pearwood.info> Message-ID: > > > To show this on simple example: > > > > >>> from itertools import count, islice > > >>> it = count() > > >>> x, y = it > > >>> it > > count(3) > > For everyone else who was confused by this, as I was, that's not > actually a copy and paste from the REPL. There should be a ValueError > raised after the x, y assignment. As given, it is confusing because it > looks like the assignment succeeded, when in fact it didn't. > > > > Here the count was advanced two times but assignment did not happen. > > Correct, because there was an exception raised. > Sorry for that, I did not want to embarrass anyone, so I wrote below that the assignment did not happen. But probably the code should speak for itself, especially if it looks like a copy from REPL > if isinstance(it, collections.abc.Iterator): > # special case for iterators > x, y = it > else: > # sequences keep the old behaviour > x, y = it[:2] > No, it can be simply x, y = iter(it) > > Cons: > > 1. A special case of how assignment works > > 2. As with any implicit behavior, hard-to-find bugs > > Right. Hard-to-find bugs beats any amount of convenience in the > interactive interpreter. To use an analogy: > > "Sure, sometimes my car suddenly shifts into reverse while I'm driving > at 60 kph, sometimes the engine falls out when I go around the corner, > and occasionally the brakes catch fire, but gosh the cup holder makes it > really convenient to drink coffee while I'm stopped at traffic lights!" > :-) Perhaps a better idea might be special syntax to tell the interpreter > you don't want to run the right-hand side to completion. "Explicit is > better than implicit" -- maybe something special like: > > x, y, * = iterable > I wrote somewhere above, that "may be I like this form". But for me * '"starred" target implies -> collect something from iterable. So now I'm towards: x, y, ... = iterable But I have not summed up yet what pitfalls can be on this path. Thank you your remarks were very extensive! With kind regards, -gdg -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon Nov 27 09:46:24 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 28 Nov 2017 01:46:24 +1100 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: References: Message-ID: <20171127144624.GH22248@ando.pearwood.info> On Tue, Nov 28, 2017 at 01:01:01AM +1100, Chris Angelico wrote: > And the status quo is noisy and has duplicated > information (you have to match the ", 2" to the number of assignment > targets). Er, not really. How about: a, b = islice(iterable, 3, 10, 4) A somewhat unusual case, to be sure, but still legal. > I would support any syntax that (a) is currently illegal, > (b) reads reasonably well, and (c) can't be TOO easily confused with > something else. Honestly, I don't see this as such a big deal that Python needs to support it at all: maybe +0.25 on the idea of non-consuming iterable unpacking. islice does the job. If we have it at all, it is yet another special syntax to learn, and I'm not convinced that the benefit outways the cost of learning yet more Perlish magic syntax for a marginal use-case. Especially if the syntax looks like grit on Tim's monitor. On the other hand, if we did get this, I would prefer magic syntax over a backwards incompatible change. On balance, given that * is already used for at least 11 different things already[1], I'd actually prefer the grit on the monitor. Perhaps x, y, ... = iterable to indicate non-consuming iterable unpacking. Or maybe x, y, / = iterable since in some sense this is the opposite of unpacking -- the whole point is to NOT unpack the remaining items. And the slash kind of looks like cutting off the process from continuing: unpack two items, then stop. ... maybe +0.5 with the slash syntax *wink* [1] Multiplication, exponentiation, sequence replication, *varargs, **keyword varargs, regexes, globs, import wild cards, iterable unpacking in function calls, extended iterable unpacking in assignments, dict unpacking -- have I missed any? -- Steve From kirillbalunov at gmail.com Mon Nov 27 09:50:29 2017 From: kirillbalunov at gmail.com (Kirill Balunov) Date: Mon, 27 Nov 2017 17:50:29 +0300 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: References: <20171127135502.GF22248@ando.pearwood.info> Message-ID: 2017-11-27 17:14 GMT+03:00 Chris Angelico : > > > Nah, far easier: > > x, y = iter(it) > Yes, you are right. >> 2. Readable and not so verbose code > >> 3. Optimized case for x,y,*z = iterator > > > > The semantics of that are already set: the first two items are assigned > > to x and y, with all subsequent items assigned to z as a list. How will > > this change optimize this case? It still needs to run through the > > iterator to generate the list. > > Maybe 'optimized case for "x, y, *_ = iterator" where you then never > use _ and it has no side effects'? But that could be worded better. > Yes, you did not need to consume and then to throw out _, and in other cases not to hang in an endless loop. > I do think islice is verbose, but the main problem is that you have to > match the second argument to the number of assignment targets. Slice > notation is an improvement, but it still has that same problem. > > But perhaps this should be added to the list of options for the PEP. > Inconvenience is that in both cases: islice and iter[:2], you should specify the exact number of assignment targets. > That's one of the options that I mentioned, as it's been proposed in > the past. The problem is that it depends on internal whitespace to > distinguish it from augmented assignment; granted, there's no way to > use "*=" with multiple targets (or even in the single-target case, you > can't do "x,*=it" with the comma in it), but that's still a > readability problem. > Your suggestion using Ellipsis at the moment seems to me the most readable, like: x, ... = iterable x, y, ... = iterable But I have not summed up yet what pitfalls can be on this path. I really do not like to use "starred" targets in any way: x, y, * = iterable x, y, *... Because any "starred" target implies consuming or collecting, and contradicts with the proposed behavior. With kind regards, -gdg -------------- next part -------------- An HTML attachment was scrubbed... URL: From k7hoven at gmail.com Mon Nov 27 10:23:31 2017 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Mon, 27 Nov 2017 17:23:31 +0200 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: References: <20171127135502.GF22248@ando.pearwood.info> Message-ID: On Mon, Nov 27, 2017 at 4:50 PM, Kirill Balunov wrote: > > I really do not like to use "starred" targets in any way: > > x, y, * = iterable > ?This one won't work, because it looks like in-place multiplication (__imul__) which clashes with the above for a single name as assignment target: x *= 2 # "x = x * 2" > x, y, *... > > Because any "starred" target implies consuming or collecting, and > contradicts with the proposed behavior. > > Consuming does not contradict with your proposal, but maybe you mean *fully consuming*. I think you are proposing partial or incremental consumption of the rhs. ?Koos -- + Koos Zevenhoven + http://twitter.com/k7hoven + -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon Nov 27 10:35:50 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 28 Nov 2017 02:35:50 +1100 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: References: <20171127135502.GF22248@ando.pearwood.info> Message-ID: <20171127153550.GI22248@ando.pearwood.info> On Tue, Nov 28, 2017 at 01:14:40AM +1100, Chris Angelico wrote: > Nah, far easier: > > x, y = iter(it) > > since that'll be a no-op in the first case Not necessarily a no-op. __iter__ might have side-effects. Of course if you can guarantee that you ONLY have iterators, then there's no need to test for an iterator. But it isn't always appropriate to convert sequences to an iterator. And besides, if you're going to call iter(), that's not that much shorter than islice() (assuming you've already imported it, of course). You only save five characters or so. But the big problem here is that iterable unpacking would be conceptually split into "iterator unpacking" and "all other iterables unpacking". I think that's unnecessary complication. > I do think islice is verbose, but the main problem is that you have to > match the second argument to the number of assignment targets. Yes, there is a certain amount of redundancy in having to specify the number of items in a slice, using either syntax: a, b = sequence[:2] a, b = islice(iterable, 2) but it is minimal duplication, hardly the sort of thing DRY is concerned with: http://www.artima.com/intv/dry.html and there is an equally important principle that is satisfied by being explicit about the number of items you want. (In the face of ambiguity, refuse the temptation to guess.) If you get the number wrong, you will find out immediately. And the fix is trivial. In this case, there is a small but real benefit to counting the assignment targets and being explicit about the number of items to slice. Consider an extension to this "non-consuming" unpacking that allowed syntax like this to pass silently: a, b = x, y, z That ought to be a clear error, right? I would hope you don't think that Python should let that through. Okay, now we put x, y, z into a list, then unpack the list: L = [x, y, z] a, b = L That ought to still be an error, unless we explicity silence it. One way to do so is with an explicit slice: a, b = L[:2] This isn't repeating yourself, it isn't really duplicated or redundant information. It is telling the interpreter, and more importantly the reader, that you want exactly two items out of a list that could contain any arbitrary number of items and that you are planning to bind them to exactly two targets. Same goes if we use islice(iterable) instead. Another way would be special syntax to say you want non-consuming unpacking, swapping out the "ambiguity" Zen for the "explicitly silencing errors" Zen. So I really don't see this as anything more than, at best, a fairly minor piece of syntactic sugar. It doesn't really add anything to the language, or allow us to do anything cool we couldn't do before. -- Steve From rosuav at gmail.com Mon Nov 27 11:03:47 2017 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 28 Nov 2017 03:03:47 +1100 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: <20171127144624.GH22248@ando.pearwood.info> References: <20171127144624.GH22248@ando.pearwood.info> Message-ID: On Tue, Nov 28, 2017 at 1:46 AM, Steven D'Aprano wrote: > On Tue, Nov 28, 2017 at 01:01:01AM +1100, Chris Angelico wrote: > >> And the status quo is noisy and has duplicated >> information (you have to match the ", 2" to the number of assignment >> targets). > > Er, not really. How about: > > a, b = islice(iterable, 3, 10, 4) > > A somewhat unusual case, to be sure, but still legal. Well, sure. But the situation that's equivalent to the proposal here is simply islice(iterable, 2). There's no proposal to have something that assigns the way you're slicing there, and there's certainly no proposal to abolish islice. ChrisA From rosuav at gmail.com Mon Nov 27 11:05:39 2017 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 28 Nov 2017 03:05:39 +1100 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: <20171127153550.GI22248@ando.pearwood.info> References: <20171127135502.GF22248@ando.pearwood.info> <20171127153550.GI22248@ando.pearwood.info> Message-ID: On Tue, Nov 28, 2017 at 2:35 AM, Steven D'Aprano wrote: > In this case, there is a small but real benefit to counting the > assignment targets and being explicit about the number of items to > slice. Consider an extension to this "non-consuming" unpacking that > allowed syntax like this to pass silently: > > a, b = x, y, z > > That ought to be a clear error, right? I would hope you don't think that > Python should let that through. Okay, now we put x, y, z into a list, > then unpack the list: > > L = [x, y, z] > a, b = L > > That ought to still be an error, unless we explicity silence it. One way > to do so is with an explicit slice: > > a, b = L[:2] I absolutely agree with this for the default case. That's why I am ONLY in favour of the explicit options. So, for instance: a, b = x, y, z # error a, b, ... = x, y, z # valid (evaluates and ignores z) ChrisA From p.f.moore at gmail.com Mon Nov 27 11:23:46 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 27 Nov 2017 16:23:46 +0000 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: References: <20171127135502.GF22248@ando.pearwood.info> <20171127153550.GI22248@ando.pearwood.info> Message-ID: On 27 November 2017 at 16:05, Chris Angelico wrote: > On Tue, Nov 28, 2017 at 2:35 AM, Steven D'Aprano wrote: >> In this case, there is a small but real benefit to counting the >> assignment targets and being explicit about the number of items to >> slice. Consider an extension to this "non-consuming" unpacking that >> allowed syntax like this to pass silently: >> >> a, b = x, y, z >> >> That ought to be a clear error, right? I would hope you don't think that >> Python should let that through. Okay, now we put x, y, z into a list, >> then unpack the list: >> >> L = [x, y, z] >> a, b = L >> >> That ought to still be an error, unless we explicity silence it. One way >> to do so is with an explicit slice: >> >> a, b = L[:2] > > I absolutely agree with this for the default case. That's why I am > ONLY in favour of the explicit options. So, for instance: > > a, b = x, y, z # error > a, b, ... = x, y, z # valid (evaluates and ignores z) Agreed, only explicit options are even worth considering (because of backward compatibility if for no other reason). However, the unpacking syntax is already complex, and hard to search for. Making it more complex needs a strong justification. And good luck in doing a google search for "..." if you saw that code in a project you had to maintain. Seriously, has anyone done a proper investigation into how much benefit this proposal would provide? It should be reasonably easy to do a code search for something like "=.*islice", to find code that's a candidate for using the proposed syntax. I suspect there's very little code like that. I'm -1 on this proposal without a much better justification of the benefits it will bring. Paul From k7hoven at gmail.com Mon Nov 27 11:35:38 2017 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Mon, 27 Nov 2017 18:35:38 +0200 Subject: [Python-ideas] generator vs iterator etc. (was: How assignment should work with generators?) Message-ID: On Mon, Nov 27, 2017 at 3:55 PM, Steven D'Aprano wrote: > On Mon, Nov 27, 2017 at 12:17:31PM +0300, Kirill Balunov wrote: > ?? > > > 2. Should this work only for generators or for any iterators? > > I don't understand why you are even considering singling out *only* > generators. A generator is a particular implementation of an iterator. I > can write: > > def gen(): > yield 1; yield 2; yield 3 > > it = gen() > > or I can write: > > it = iter([1, 2, 3]) > > and the behaviour of `it` should be identical. > > > ?I can see where this is coming from. The thing is that "iterator" and "generator" are mostly synonymous, except two things: (1) Generators are iterators that are produced by a generator function (2) Generator functions are sometimes referred to as just "generators" The concept of "generator" thus overlaps with both "iterator" and "generator function". Then there's also "iterator" and "iterable", which are two different things: (3) If `obj` is an *iterable*, then `it = iter(obj)` is an *iterator* (over the contents of `obj`) ( ?4) ?Iterators yield values, for example on explicit calls to next(it). Personally I have leaned towards keeping a clear distinction between "generator function" and "generator"?, which leads to the situation that "generator" and "iterator" are mostly synonymous for me. Sometimes, for convenience, I use the term "generator" to refer to "iterators" more generally. This further seems to have a minor benefit that "generators" and "iterables" are less easily confused with each other than "iterators" and "iterables". I thought about this issue some time ago for the `views` package, which has a separation between sequences (seq) and other iterables (gen): https://github.com/k7hoven/views The functionality provided by `views.gen` is not that interesting?it's essentially a subset of itertools functionality, but with an API that parallels `views.seq` which works with sequences (iterable, sliceable, chainable, etc.). I used the name `gen`, because iterator/iterable variants of the functionality can be implemented with generator functions (although also with other kinds of iterators/iterables). Calling the thing `iter` would have conflicted with the builtin `iter`. HOWEVER, this naming can be confusing for those that lean more towards using "generator" to also mean "generator function", and for those that are comfortable with the term "iterator" despite its resemblance to "iterable". Now I'm actually seriously considering to consider renaming `views.gen` to ` views.iter` when I have time. After all, there's already `views.range` which "conflicts" with the builtin range. ?Anyway, the point is that the naming is suboptimal.? SOLUTION: Maybe (a) all iterators should be called iterators or (b) all iterators should be called generators, regardless of whether they are somehow a result of a generator function having been called in the past. (I'm not going into the distinction between things that can receive values via `send` or any other possible distinctions between different types of iterators and iterables.) ??Koos? ?(discussion originated from python-ideas, but cross-posted to python-dev in case there's more interest there)? -- + Koos Zevenhoven + http://twitter.com/k7hoven + -------------- next part -------------- An HTML attachment was scrubbed... URL: From k7hoven at gmail.com Mon Nov 27 11:53:23 2017 From: k7hoven at gmail.com (Koos Zevenhoven) Date: Mon, 27 Nov 2017 18:53:23 +0200 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: <20171127135502.GF22248@ando.pearwood.info> References: <20171127135502.GF22248@ando.pearwood.info> Message-ID: On Mon, Nov 27, 2017 at 3:55 PM, Steven D'Aprano wrote: > On Mon, Nov 27, 2017 at 12:17:31PM +0300, Kirill Balunov wrote: > ?? > > > 2. Should this work only for generators or for any iterators? > > I don't understand why you are even considering singling out *only* > generators. A generator is a particular implementation of an iterator. I > can write: > > def gen(): > yield 1; yield 2; yield 3 > > it = gen() > > or I can write: > > it = iter([1, 2, 3]) > > and the behaviour of `it` should be identical. > > > ?I can see where this is coming from, but I wrote about it in a new thread: "generator vs iterator etc. (was: How assignment should work with generators?)". > > But maybe we should consider allowing slice notation on arbitrary > iterators: > > x, y = it[:2] > > > I have not thought this through in any serious detail, but it seems to > me that if the only problem here is the inconvenience of using islice(), > we could add slicing to iterators. I think that would be better than > having iterators and other iterables behave differently. > > Making iterators behave like sequences (slicing etc.) introduces various issues including memory considerations and backwards compatibility. That's why the `views` package [1] keeps a clear separations between sequences and iterators. IterABLES are a bit fuzzy here, but they at least should be able to produce an iterator. I should have time to discuss this more at a later point, if needed. ??Koos? ?[1] ?https://github.com/k7hoven/views -- + Koos Zevenhoven + http://twitter.com/k7hoven + -------------- next part -------------- An HTML attachment was scrubbed... URL: From c at anthonyrisinger.com Mon Nov 27 12:48:35 2017 From: c at anthonyrisinger.com (C Anthony Risinger) Date: Mon, 27 Nov 2017 11:48:35 -0600 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: References: <20171127135502.GF22248@ando.pearwood.info> <20171127153550.GI22248@ando.pearwood.info> Message-ID: This proposal resonates with me. I've definitely wanted to use unpacking to crank an iterator a couple times and move on without exhausting the iterator. It's a very natural and intuitive meaning for unpacking as it relates to iterators. In my mind, this ask is aligned with, and has similar motivation to, lazy zip(), map(), and keys() in Python 3. Syntax support for unpacking as it stands today is not very conducive to iterables and conflicts with a widespread desire to use more of them. Couple thoughts: * Perhaps existence of `__len__` should influence unpacking? There is a semantic difference (and typically a visual one too) between 1-to-1 matching a fixed-width sequence/container on the RHS to identifiers on the LHS, even if they look similar (ie. "if RHS has a length make it fit, otherwise don't"). * (related to above) Perhaps the "rigidity"(?) of both RHS and LHS should influence unpacking? If they are both fixed-width, expect exact match. If either is variable-width, then lazily unravel until a different problem happens (eg. LHS is fixed-width but RHS ran out of values... basically we always unravel lazily, but keep track of when LHS or RHS become variable, and avoid checking length if they do). * Python 4 change as the language moves towards lazy iterators everywhere? If `__len__` influenced behavior like mentioned above, then a mechanical fix to code would simply be `LHS = tuple(*RHS)`, similar to keys(). While I like the original proposal, adding basic slice support to iterables is also a nice idea. Both are independently useful, eg. `gen.__getitem__(slice())` delegates to islice(). This achieves the goal of allowing meaningful unpacking of an iterator window, using normal syntax, without breaking existing code. The fixed/variable-width idea makes the most sense to me though. This enables things like: >>> a, b, c = (1, *range(2, 100), 3) (1, 2, 3) Since both sides are not explicitly sized unpacking is not forcibly sized either. Expecting LHS/RHS to exactly match 100% of the time is the special case here today, not the proposed general unpacking rules that will work well with iterators. This change also has a Lua-like multiple return value feel to it that appeals to me. Thanks, On Nov 27, 2017 10:23 AM, "Paul Moore" wrote: > On 27 November 2017 at 16:05, Chris Angelico wrote: > > On Tue, Nov 28, 2017 at 2:35 AM, Steven D'Aprano > wrote: > >> In this case, there is a small but real benefit to counting the > >> assignment targets and being explicit about the number of items to > >> slice. Consider an extension to this "non-consuming" unpacking that > >> allowed syntax like this to pass silently: > >> > >> a, b = x, y, z > >> > >> That ought to be a clear error, right? I would hope you don't think that > >> Python should let that through. Okay, now we put x, y, z into a list, > >> then unpack the list: > >> > >> L = [x, y, z] > >> a, b = L > >> > >> That ought to still be an error, unless we explicity silence it. One way > >> to do so is with an explicit slice: > >> > >> a, b = L[:2] > > > > I absolutely agree with this for the default case. That's why I am > > ONLY in favour of the explicit options. So, for instance: > > > > a, b = x, y, z # error > > a, b, ... = x, y, z # valid (evaluates and ignores z) > > Agreed, only explicit options are even worth considering (because of > backward compatibility if for no other reason). However, the unpacking > syntax is already complex, and hard to search for. Making it more > complex needs a strong justification. And good luck in doing a google > search for "..." if you saw that code in a project you had to > maintain. Seriously, has anyone done a proper investigation into how > much benefit this proposal would provide? It should be reasonably easy > to do a code search for something like "=.*islice", to find code > that's a candidate for using the proposed syntax. I suspect there's > very little code like that. > > I'm -1 on this proposal without a much better justification of the > benefits it will bring. > Paul > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Mon Nov 27 15:17:41 2017 From: brett at python.org (Brett Cannon) Date: Mon, 27 Nov 2017 20:17:41 +0000 Subject: [Python-ideas] Using an appropriate tone in emails (was: Adding a thin wrapper class around the functions in stdlib.heapq) In-Reply-To: References: Message-ID: On Tue, 21 Nov 2017 at 14:57 Nick Timkovich wrote: > On Tue, Nov 21, 2017 at 4:16 PM, Sven R. Kunze wrote: > >> Maybe, that suffices: https://pypi.python.org/pypi/xheap >> > I still think the heapq.heap* functions are atrocious and they should > immediately be packaged with *no additional features* into a stdlib object > for reasons along the line of > https://mail.python.org/pipermail/python-ideas/2017-October/047514.html > I wanted to address the general tone people should aim for on this list so that people don't inadvertently insult people. Now I'm going to preface this as I'm not explicitly calling out Nick here on this, I'm just using his email as an illustrative example as it's what happened to trigger me to write this email. So Nick said above that the design of the heapq module is "atrocious", to the extent that it should be "immediately" fixed with an OO wrapper. So obviously Nick doesn't like the design of the heapq module. ;) And that's okay! And he's totally within his rights to express the feeling that the heapq module as it stands doesn't meet his needs. But calling it "atrocious" and so bad that it needs to be fixed "immediately" as if it's a blight upon the stdlib is unnecessarily insulting to those that have worked on the module. To convey the feeling that you think an OO wrapper would be helpful as the current design doesn't work for you, you could just phrase it as I just did to get the same point across without insulting anyone. Basically if you wouldn't like your own work called "atrocious" by someone you respect, then it's probably best to not use that phrasing when talking about a stranger's code either. If you want more context as to why this all matters in order to keep open source sustainable, you can watch my 15 minute keynote from JupyterCon this year for a more in-depth, verbal explanation: https://www.youtube.com/watch?v=y19s6vPpGXA . And BTW, the heapq module is 15 years old, has a history, and there are explicit reasons it's designed the way it is, so from that perspective I would argue it has a good design. -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Nov 27 15:39:45 2017 From: guido at python.org (Guido van Rossum) Date: Mon, 27 Nov 2017 12:39:45 -0800 Subject: [Python-ideas] Using an appropriate tone in emails (was: Adding a thin wrapper class around the functions in stdlib.heapq) In-Reply-To: References: Message-ID: +1 Basically, if you're posting here, you probably want something to change. And that means that somebody has to make that change. And someone has to approve it. Etc. And you are trying to influence those people to make/approve/etc. that change, since for them it's less work if nothing change. And how do you influence people? Not by opening with an insult. (Also if you think that none of this matters because we're all rational actors, grow up. :-) --Guido On Mon, Nov 27, 2017 at 12:17 PM, Brett Cannon wrote: > > > On Tue, 21 Nov 2017 at 14:57 Nick Timkovich > wrote: > >> On Tue, Nov 21, 2017 at 4:16 PM, Sven R. Kunze wrote: >> >>> Maybe, that suffices: https://pypi.python.org/pypi/xheap >>> >> I still think the heapq.heap* functions are atrocious and they should >> immediately be packaged with *no additional features* into a stdlib object >> for reasons along the line of https://mail.python.org/ >> pipermail/python-ideas/2017-October/047514.html >> > > I wanted to address the general tone people should aim for on this list so > that people don't inadvertently insult people. Now I'm going to preface > this as I'm not explicitly calling out Nick here on this, I'm just using > his email as an illustrative example as it's what happened to trigger me to > write this email. > > So Nick said above that the design of the heapq module is "atrocious", to > the extent that it should be "immediately" fixed with an OO wrapper. So > obviously Nick doesn't like the design of the heapq module. ;) And that's > okay! And he's totally within his rights to express the feeling that the > heapq module as it stands doesn't meet his needs. > > But calling it "atrocious" and so bad that it needs to be fixed > "immediately" as if it's a blight upon the stdlib is unnecessarily > insulting to those that have worked on the module. To convey the feeling > that you think an OO wrapper would be helpful as the current design doesn't > work for you, you could just phrase it as I just did to get the same point > across without insulting anyone. Basically if you wouldn't like your own > work called "atrocious" by someone you respect, then it's probably best to > not use that phrasing when talking about a stranger's code either. > > If you want more context as to why this all matters in order to keep open > source sustainable, you can watch my 15 minute keynote from JupyterCon this > year for a more in-depth, verbal explanation: > https://www.youtube.com/watch?v=y19s6vPpGXA . > > And BTW, the heapq module is 15 years old, has a history, and there are > explicit reasons it's designed the way it is, so from that perspective I > would argue it has a good design. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Mon Nov 27 16:13:01 2017 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 28 Nov 2017 10:13:01 +1300 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: References: Message-ID: <5A1C7FDD.6080107@canterbury.ac.nz> Chris Angelico wrote: > x, y, * = iter # unpack into nothing I'm surprised this isn't already allowed. It seems like the One Obvious Way to me. -- Greg From greg.ewing at canterbury.ac.nz Mon Nov 27 16:18:50 2017 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 28 Nov 2017 10:18:50 +1300 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: References: <20171127135502.GF22248@ando.pearwood.info> Message-ID: <5A1C813A.3040904@canterbury.ac.nz> Chris Angelico wrote: > The problem is that it depends on internal whitespace to > distinguish it from augmented assignment; Ah, didn't spot that. I guess the ellipsis is the next best thing then. An alternative would be to require parens: (x, y, *) = z -- Greg From guido at python.org Mon Nov 27 16:18:48 2017 From: guido at python.org (Guido van Rossum) Date: Mon, 27 Nov 2017 13:18:48 -0800 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: <5A1C7FDD.6080107@canterbury.ac.nz> References: <5A1C7FDD.6080107@canterbury.ac.nz> Message-ID: On Mon, Nov 27, 2017 at 1:13 PM, Greg Ewing wrote: > Chris Angelico wrote: > > x, y, * = iter # unpack into nothing >> > > I'm surprised this isn't already allowed. It seems like the > One Obvious Way to me. I'm not that surprised. While it appears to rhyme with the use of a lone '*' in function signatures, it would actually mean the opposite: in signatures it means "don't allow any more". These opposite meanings would cause some confusion. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Nov 27 16:24:38 2017 From: guido at python.org (Guido van Rossum) Date: Mon, 27 Nov 2017 13:24:38 -0800 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: <5A1C813A.3040904@canterbury.ac.nz> References: <20171127135502.GF22248@ando.pearwood.info> <5A1C813A.3040904@canterbury.ac.nz> Message-ID: On Mon, Nov 27, 2017 at 1:18 PM, Greg Ewing wrote: > Chris Angelico wrote: > >> The problem is that it depends on internal whitespace to >> distinguish it from augmented assignment; >> > > Ah, didn't spot that. I guess the ellipsis is the next best > thing then. > > An alternative would be to require parens: > > (x, y, *) = z > But that would have the same issue. Is this problem really important enough that it requires dedicated syntax? Isn't the itertools-based solution good enough? (Or failing that, couldn't we add something to itertools to make it more readable rather than going straight to new syntax?) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Mon Nov 27 16:32:55 2017 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 28 Nov 2017 08:32:55 +1100 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: References: <20171127135502.GF22248@ando.pearwood.info> <5A1C813A.3040904@canterbury.ac.nz> Message-ID: On Tue, Nov 28, 2017 at 8:24 AM, Guido van Rossum wrote: > On Mon, Nov 27, 2017 at 1:18 PM, Greg Ewing > wrote: >> >> Chris Angelico wrote: >>> >>> The problem is that it depends on internal whitespace to >>> distinguish it from augmented assignment; >> >> >> Ah, didn't spot that. I guess the ellipsis is the next best >> thing then. >> >> An alternative would be to require parens: >> >> (x, y, *) = z > > > But that would have the same issue. > > Is this problem really important enough that it requires dedicated syntax? > Isn't the itertools-based solution good enough? (Or failing that, couldn't > we add something to itertools to make it more readable rather than going > straight to new syntax?) I don't think there's much that can be done without syntax; the biggest problem IMO is that you need to tell islice how many targets it'll be assigned into. It needs some interpreter support to express "grab as many as you have targets for, leaving everything else behind" without stating how many that actually is. So the question is whether that is sufficiently useful to justify extending the syntax. There are a number of potential advantages and several competing syntax options, and this suggestion keeps coming up, so I think a PEP is warranted. ChrisA From kirillbalunov at gmail.com Mon Nov 27 16:44:07 2017 From: kirillbalunov at gmail.com (Kirill Balunov) Date: Tue, 28 Nov 2017 00:44:07 +0300 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: References: <5A1C7FDD.6080107@canterbury.ac.nz> Message-ID: While I was more optimistic when I proposed this idea, moving on I gradually become less and less optimistic. I did not underestimate how much this would change the semantics. At present, if the lhs is comma-separated list of targets, the rhs is evaluated, and only then if they match the assignment happens. Here it depends on the number of targets in lhs, and should be evaluated lazily. So someone too perlish clever can assume that with the proposed syntax: >>> def gen(): >>> for i in ['a', 'b', 'c', 'd']: >>> v = x if 'x' in globals() else 'var ' >>> yield v + i >>> x, y = gen() >>> x var a >>> y var ab This is bad and I do not like it, but I do not see any serious reasons why it should not be allowed. In case of Ellipsis they also should trigger special behavior. While I like this feature of lazy assignment, may be it becomes more special than it deserves. With kind regards, -gdg -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Mon Nov 27 16:49:45 2017 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 28 Nov 2017 10:49:45 +1300 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: References: <20171127135502.GF22248@ando.pearwood.info> <20171127153550.GI22248@ando.pearwood.info> Message-ID: <5A1C8879.2050105@canterbury.ac.nz> C Anthony Risinger wrote: > * Perhaps existence of `__len__` should influence unpacking? There is a > semantic difference (and typically a visual one too) between 1-to-1 > matching a fixed-width sequence/container on the RHS to identifiers on > the LHS, even if they look similar (ie. "if RHS has a length make it > fit, otherwise don't"). -1. There's a convention that an iterator can implement __len__ to provide a hint about the number of items it will return (useful for preallocating space, etc.) It would be very annoying if such an iterator behaved differently from other iterators when unpacking. Another thing is that the proposed feature will be useful on non-iterator iterables as well, since it saves the overhead of unpacking the rest of the items only to throw them away. > While I like the original proposal, adding basic slice support to > iterables is also a nice idea. It's not as nice as it seems. You're asking for __getitem__ to be made part of the iterator protocol, which would be a huge change affecting all existing iterators. Otherwise, it would just be a piecemeal affair. Some iterators would support slicing and some wouldn't, so you couldn't rely on it. -- Greg From guido at python.org Mon Nov 27 16:49:25 2017 From: guido at python.org (Guido van Rossum) Date: Mon, 27 Nov 2017 13:49:25 -0800 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: References: <20171127135502.GF22248@ando.pearwood.info> <5A1C813A.3040904@canterbury.ac.nz> Message-ID: On Mon, Nov 27, 2017 at 1:32 PM, Chris Angelico wrote: > On Tue, Nov 28, 2017 at 8:24 AM, Guido van Rossum > wrote: > > On Mon, Nov 27, 2017 at 1:18 PM, Greg Ewing > > > wrote: > >> > >> Chris Angelico wrote: > >>> > >>> The problem is that it depends on internal whitespace to > >>> distinguish it from augmented assignment; > >> > >> > >> Ah, didn't spot that. I guess the ellipsis is the next best > >> thing then. > >> > >> An alternative would be to require parens: > >> > >> (x, y, *) = z > > > > > > But that would have the same issue. > > > > Is this problem really important enough that it requires dedicated > syntax? > > Isn't the itertools-based solution good enough? (Or failing that, > couldn't > > we add something to itertools to make it more readable rather than going > > straight to new syntax?) > > I don't think there's much that can be done without syntax; the > biggest problem IMO is that you need to tell islice how many targets > it'll be assigned into. It needs some interpreter support to express > "grab as many as you have targets for, leaving everything else behind" > without stating how many that actually is. So the question is whether > that is sufficiently useful to justify extending the syntax. There are > a number of potential advantages and several competing syntax options, > and this suggestion keeps coming up, so I think a PEP is warranted. > OK, that's reasonable, and at first blush the ellipsis proposal looks okay. My PEP queue for Python 3.7 is full though, so I would like to put this off until 3.8. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Mon Nov 27 16:52:50 2017 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 28 Nov 2017 08:52:50 +1100 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: References: <20171127135502.GF22248@ando.pearwood.info> <5A1C813A.3040904@canterbury.ac.nz> Message-ID: On Tue, Nov 28, 2017 at 8:49 AM, Guido van Rossum wrote: > My PEP queue for Python 3.7 is full though, so I would like to put this off > until 3.8. > Yeah, I don't think this could reasonably be raced into 3.7 even if it were critically important, and it's not. 3.8 will be fine. Kirill, do you want to spearhead the discussion? I'm happy to help out. ChrisA From kirillbalunov at gmail.com Mon Nov 27 16:54:18 2017 From: kirillbalunov at gmail.com (Kirill Balunov) Date: Tue, 28 Nov 2017 00:54:18 +0300 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: References: <20171127135502.GF22248@ando.pearwood.info> <20171127153550.GI22248@ando.pearwood.info> Message-ID: 2017-11-27 19:23 GMT+03:00 Paul Moore : > It should be reasonably easy > to do a code search for something like "=.*islice", to find code > that's a candidate for using the proposed syntax. I suspect there's > very little code like that. While isclice is something equivalent, it can be used in places like: x, y = seq[:2] x, y, z = seq[:3] With kind regards, -gdg -------------- next part -------------- An HTML attachment was scrubbed... URL: From prometheus235 at gmail.com Mon Nov 27 16:59:08 2017 From: prometheus235 at gmail.com (Nick Timkovich) Date: Mon, 27 Nov 2017 16:59:08 -0500 Subject: [Python-ideas] Using an appropriate tone in emails (was: Adding a thin wrapper class around the functions in stdlib.heapq) In-Reply-To: References: Message-ID: On Mon, Nov 27, 2017 at 8:17 PM, Brett Cannon wrote: > But calling it "atrocious" and so bad that it needs to be fixed > "immediately" as if it's a blight upon the stdlib is unnecessarily > insulting to those that have worked on the module. To convey the feeling > that you think an OO wrapper would be helpful as the current design doesn't > work for you, you could just phrase it as I just did to get the same point > across without insulting anyone. Basically if you wouldn't like your own > work called "atrocious" by someone you respect, then it's probably best to > not use that phrasing when talking about a stranger's code either. > Sorry for the curt tone, I did lose some sight on the code being designed by people rather than a faceless organization. My intention wasn't to disparage the original authors but sprung more out of my frustration and perception from that thread and those before that the status quo would not change and that if a contribution was proffered, would simply be dismissed or ignored. To motivate any change, there must be some argument levied against the status quo, but hopefully I can articulate it better. That little corner is something I'm interested in, and not having contributed to CPython before, I'm unsure how it "really works". The steps at https://devguide.python.org/stdlibchanges/ suggest trying to elicit community feedback from the lists as a step, so negative feedback tends to kill the enthusiasm to actually make the PR. In the absence of code, concrete arguments are almost impossible as we're discussing the shape of clouds. Nick -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Mon Nov 27 17:15:56 2017 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 28 Nov 2017 11:15:56 +1300 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: References: <5A1C7FDD.6080107@canterbury.ac.nz> Message-ID: <5A1C8E9C.7090706@canterbury.ac.nz> Guido van Rossum wrote: > While it appears to rhyme with the use of a lone > '*' in function signatures, it would actually mean the opposite: in > signatures it means "don't allow any more". That's the usage that's out of step with the rest -- all the others can be seen as some form of wildcard. So it's already confusing, and I don't think adding another wildcard meaning would make things any worse. -- Greg From p.f.moore at gmail.com Mon Nov 27 17:27:20 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 27 Nov 2017 22:27:20 +0000 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: References: <20171127135502.GF22248@ando.pearwood.info> <20171127153550.GI22248@ando.pearwood.info> Message-ID: On 27 November 2017 at 21:54, Kirill Balunov wrote: > 2017-11-27 19:23 GMT+03:00 Paul Moore : > >> >> It should be reasonably easy >> to do a code search for something like "=.*islice", to find code >> that's a candidate for using the proposed syntax. I suspect there's >> very little code like that. > > > While isclice is something equivalent, it can be used in places like: > > x, y = seq[:2] > x, y, z = seq[:3] But in those places, x, y, *_ = seq works fine at the moment. So if the programmer didn't feel inclined to use x, y, *_ = seq, there's no reason to assume that they would get any benefit from x, y, ... = seq either. Paul From storchaka at gmail.com Mon Nov 27 17:17:18 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 28 Nov 2017 00:17:18 +0200 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: References: <20171127135502.GF22248@ando.pearwood.info> <5A1C813A.3040904@canterbury.ac.nz> Message-ID: 27.11.17 23:24, Guido van Rossum ????: > Is this problem really important enough that it requires dedicated > syntax? Isn't the itertools-based solution good enough? (Or failing > that, couldn't we add something to itertools to make it more readable > rather than going straight to new syntax?) I want to remind PEP 204 and PEP 212. The special purposed syntaxes were proposed to help solving much more common problems. They were rejected in favor of builtins range() and enumerate(). And we are happy with these builtins. The function for solving this problem already exists. It's itertools.islice(). It isn't builtin, but this problem is much less common than use cases for range() and enumerate(). If we don't have special non-function syntax for range(), enumerate(), zip(), itertools.chain(), itertools.count(), itertools.repeat(), etc, I don't think we should have a special syntax for one particular case of using itertools.islice(). From p.f.moore at gmail.com Mon Nov 27 17:36:49 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 27 Nov 2017 22:36:49 +0000 Subject: [Python-ideas] Using an appropriate tone in emails (was: Adding a thin wrapper class around the functions in stdlib.heapq) In-Reply-To: References: Message-ID: On 27 November 2017 at 21:59, Nick Timkovich wrote: > On Mon, Nov 27, 2017 at 8:17 PM, Brett Cannon wrote: >> >> But calling it "atrocious" and so bad that it needs to be fixed >> "immediately" as if it's a blight upon the stdlib is unnecessarily insulting >> to those that have worked on the module. To convey the feeling that you >> think an OO wrapper would be helpful as the current design doesn't work for >> you, you could just phrase it as I just did to get the same point across >> without insulting anyone. Basically if you wouldn't like your own work >> called "atrocious" by someone you respect, then it's probably best to not >> use that phrasing when talking about a stranger's code either. > > > Sorry for the curt tone, I did lose some sight on the code being designed by > people rather than a faceless organization. My intention wasn't to disparage > the original authors but sprung more out of my frustration and perception > from that thread and those before that the status quo would not change and > that if a contribution was proffered, would simply be dismissed or ignored. > To motivate any change, there must be some argument levied against the > status quo, but hopefully I can articulate it better. > > That little corner is something I'm interested in, and not having > contributed to CPython before, I'm unsure how it "really works". The steps > at https://devguide.python.org/stdlibchanges/ suggest trying to elicit > community feedback from the lists as a step, so negative feedback tends to > kill the enthusiasm to actually make the PR. In the absence of code, > concrete arguments are almost impossible as we're discussing the shape of > clouds. In my experience (and this reiterates Brett's point) the proposals that get the best responses are those that are presented positively - instead of focusing on the (perceived) problems with the current situation, describe the benefits that will come from the proposed change. If you can't do that, then it's unlikely there is enough justification for a change. Certainly, negative feedback can be demotivating, and when you have a great idea and all you hear is "but what if...?" it's hard to remain positive. But you're not going to get better feedback if you criticise - at best, people will stop listening, and you'll have avoided some of the arguments, but at the cost of no-one being willing to support your proposal and so it dies. Your perception isn't wrong, by the way. It *is* hard to persuade people that the status quo needs to change. But that's not because there's no interest in change. Rather, it's because there's a strong sense among both the core developers and the frequent contributors on this list, of the significant impact any change will have - it's hard to conceive of just how many people will be affected by even the smallest change we make, and that's a big responsibility. So while it's often hard, focusing on the positives (and being willing to accept that the status quo is sufficient for many people) really is the only way to gain support. Paul From greg.ewing at canterbury.ac.nz Mon Nov 27 17:40:49 2017 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 28 Nov 2017 11:40:49 +1300 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: References: <20171127135502.GF22248@ando.pearwood.info> <5A1C813A.3040904@canterbury.ac.nz> Message-ID: <5A1C9471.9060803@canterbury.ac.nz> Guido van Rossum wrote: > Is this problem really important enough that it requires dedicated > syntax? Isn't the itertools-based solution good enough? Well, it works, but it feels very clumsy. It's annoying to have to specify the number of items in two places. Also, it seems perverse to have to tell Python to do *more* stuff to mitigate the effects of stuff it does that you didn't want it to do in the first place. Like I said, I'm actually surprised that this doesn't already work. To me it feels more like filling in a piece of functionality that was overlooked, rather than adding a new feature. Filling in a pothole in the road rather than bulding a new piece of road. (Pushing the road analogy maybe a bit too far, the current itertools solution is like digging *more* potholes to make the road bumpy enough that you don't notice the first pothole.) > (Or failing > that, couldn't we add something to itertools to make it more readable > rather than going straight to new syntax?) I'm not sure how we would do that. Even if we could, it would still feel clumsy having to use anything from itertools at all. -- Greg From steve at pearwood.info Mon Nov 27 18:44:49 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 28 Nov 2017 10:44:49 +1100 Subject: [Python-ideas] generator vs iterator etc. (was: How assignment should work with generators?) In-Reply-To: References: Message-ID: <20171127234449.GJ22248@ando.pearwood.info> On Mon, Nov 27, 2017 at 06:35:38PM +0200, Koos Zevenhoven wrote: > SOLUTION: Maybe (a) all iterators should be called iterators All iterators *are* called iterators. Just as all mammals are called "mammals". The subset of iterators which are created as generators are *also* called generators, just as the mammals which are dogs are called "dogs" when it is necessary to distinguish a dog from some other mammal. > or (b) all > iterators should be called generators, regardless of whether they are > somehow a result of a generator function having been called in the past. Absolutely not. That would be confusing -- it would be analogous to calling all sequences (lists, tuples, deques etc) "strings". What benefit is there to calling all iterators "generators", losing the distinction between those which are defined using def and yield and those created using iter()? Sometimes that distinction is important. You are right that sometimes the term "generator" is used as shorthand for "generator function". Most of the time the distinction doesn't actually matter, since you cannot (easily?) create a generator without first creating a generator function. Or if it does matter, it is clear in context which is meant. For those few times where it *does* matter, there is no substitute for precision in language, and that depends on the author, not the terminology. -- Steve From steve at pearwood.info Mon Nov 27 19:25:34 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 28 Nov 2017 11:25:34 +1100 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: References: <20171127135502.GF22248@ando.pearwood.info> Message-ID: <20171128002534.GK22248@ando.pearwood.info> On Mon, Nov 27, 2017 at 06:53:23PM +0200, Koos Zevenhoven wrote: > Making iterators behave like sequences (slicing etc.) introduces various > issues including memory considerations and backwards compatibility. Perhaps. I'm not going to actively champion the idea of supporting slicing for iterators, but the idea I had was for no more than having iterator[start:stop:step] to do *exactly* what itertools.islice(iterator, start, stop, step) does now. > That's > why the `views` package [1] keeps a clear separations between sequences and > iterators. IterABLES are a bit fuzzy here, but they at least should be able > to produce an iterator. I don't think the concept of iterable is fuzzy: they are a superset of iterators. To be precise, an iterable is something which supports iteration, which means it must either: - support the iterator protocol with __iter__ and __next__ raising StopIteration; - or support the classic sequence protocol with __getitem__ raising IndexError at the end of the sequence. I think that's the only two possibilities. That covers (all?) collections, sequences, lists, lazy computed sequences like range, iterators, generators, etc. https://docs.python.org/3/glossary.html#term-iterable -- Steve From steve at pearwood.info Mon Nov 27 19:31:56 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 28 Nov 2017 11:31:56 +1100 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: <5A1C8E9C.7090706@canterbury.ac.nz> References: <5A1C7FDD.6080107@canterbury.ac.nz> <5A1C8E9C.7090706@canterbury.ac.nz> Message-ID: <20171128003155.GL22248@ando.pearwood.info> On Tue, Nov 28, 2017 at 11:15:56AM +1300, Greg Ewing wrote: > Guido van Rossum wrote: > >While it appears to rhyme with the use of a lone > >'*' in function signatures, it would actually mean the opposite: in > >signatures it means "don't allow any more". > > That's the usage that's out of step with the rest -- all > the others can be seen as some form of wildcard. > > So it's already confusing, and I don't think adding another > wildcard meaning would make things any worse. How does "stop iterating here" equate to a wildcard? We already have a "wildcard" for iterable unpacking, using the extended iterable unpacking syntax: x, y, *z = iterable which unpacks the first two items into x and y and the rest into z. This is the opposite: *stop* unpacking, so that after x and y are unpacked the process stops. I don't see how this is conceptually a wild card. -- Steve From steve at pearwood.info Mon Nov 27 19:47:08 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 28 Nov 2017 11:47:08 +1100 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: <5A1C8879.2050105@canterbury.ac.nz> References: <20171127135502.GF22248@ando.pearwood.info> <20171127153550.GI22248@ando.pearwood.info> <5A1C8879.2050105@canterbury.ac.nz> Message-ID: <20171128004707.GM22248@ando.pearwood.info> On Tue, Nov 28, 2017 at 10:49:45AM +1300, Greg Ewing wrote: > C Anthony Risinger wrote: > >* Perhaps existence of `__len__` should influence unpacking? There is a > >semantic difference (and typically a visual one too) between 1-to-1 > >matching a fixed-width sequence/container on the RHS to identifiers on > >the LHS, even if they look similar (ie. "if RHS has a length make it > >fit, otherwise don't"). > > -1. There's a convention that an iterator can implement __len__ > to provide a hint about the number of items it will return > (useful for preallocating space, etc.) I think you mean __length_hint__ for giving a hint. If an iterator supported __len__ itself, I'd expect it to be exact, not a hint. I'm not sure that its a strong convention, but we certainly could create custom iterators that supported len(). https://www.python.org/dev/peps/pep-0424/ > >While I like the original proposal, adding basic slice support to > >iterables is also a nice idea. > > It's not as nice as it seems. You're asking for __getitem__ > to be made part of the iterator protocol, which would be a > huge change affecting all existing iterators. There are ways around that. As I said earlier, I'm not going to champion this idea, but for the sake of brainstorming, the interpreter could implement obj[slice] as: if obj has __getitem__: call obj.__getitem__(slice) elif iter(obj) is obj: call itertools.islice(obj, slice) else: raise TypeError or similar. Its not literally necessary to require iterators themselves to support a __getitem__ method in order to add slicing to the iterator protocol. (Similar to the way bool() falls back on testing for non-zero length in the event that __bool__ doesn't exist, or != falls back on calling and NOT'ing __eq__ if __ne__ doesn't exist.) So I think this is solvable if it needs to be solved. -- Steve From vano at mail.mipt.ru Mon Nov 27 21:12:36 2017 From: vano at mail.mipt.ru (Ivan Pozdeev) Date: Tue, 28 Nov 2017 05:12:36 +0300 Subject: [Python-ideas] Repurpose `assert' into a general-purpose check Message-ID: The `assert' statment was created the same as in previous languages like C/C++: a check to only do in debug mode, when you can't yet trust your code to manage and pass around internal data correctly. Examples are array bounds and object state integrity constraints. Unlike C, Python does the aforementioned checks all the time, i.e. it's effectively always in "debug mode". Furthermore, validation checks are an extremily common use case. I have been using assert in my in-house code for input validation for a long time, the only thing preventing me from doing the same in public code is the fact that it only raises AssertionError's while library code is expected to raise TypeError or ValueError on invalid input. The last drop that incited me to do this proposition was https://stackoverflow.com/questions/2142202/what-are-acceptable-use-cases-for-pythons-assert-statement where none of the _seven_ answer authors (beside me) managed to make a _single realistic use case_ for `assert' as a debug-only check. --- So, I'm hereby proposing to: * make `assert' stay in optimized mode * change its syntax to raise other types of exceptions E.g.: "assert condition, type, value" or "assert condition, type, exception_constructor_argument(s)" to maintain backward compatibility. -- Regards, Ivan From rosuav at gmail.com Mon Nov 27 21:19:12 2017 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 28 Nov 2017 13:19:12 +1100 Subject: [Python-ideas] Repurpose `assert' into a general-purpose check In-Reply-To: References: Message-ID: On Tue, Nov 28, 2017 at 1:12 PM, Ivan Pozdeev via Python-ideas wrote: > The `assert' statment was created the same as in previous languages like > C/C++: a check to only do in debug mode, when you can't yet trust your code > to manage and pass around internal data correctly. Examples are array bounds > and object state integrity constraints. > > Unlike C, Python does the aforementioned checks all the time, i.e. it's > effectively always in "debug mode". Furthermore, validation checks are an > extremily common use case. > > I have been using assert in my in-house code for input validation for a long > time, the only thing preventing me from doing the same in public code is the > fact that it only raises AssertionError's while library code is expected to > raise TypeError or ValueError on invalid input. Actually, Python does have a way of disabling assertions (the -O flag), so they should be treated the same way they are in C. Assertions should not be used as shorthands for "if cond: raise Exc" in the general case. ChrisA From vano at mail.mipt.ru Mon Nov 27 21:28:10 2017 From: vano at mail.mipt.ru (Ivan Pozdeev) Date: Tue, 28 Nov 2017 05:28:10 +0300 Subject: [Python-ideas] Repurpose `assert' into a general-purpose check In-Reply-To: References: Message-ID: On 28.11.2017 5:19, Chris Angelico wrote: > Actually, Python does have a way of disabling assertions (the -O > flag), so they should be treated the same way they are in C. > Assertions should not be used as shorthands for "if cond: raise Exc" > in the general case. I'm claiming, and provided evidence, that there are no use cases for this in Python, so no-one (of any significance) will suffer when the disabling is cut out. In any case, we will probably do it with a __future__ statement for a transitional period. -- Regards, Ivan From python-ideas at mgmiller.net Mon Nov 27 21:31:28 2017 From: python-ideas at mgmiller.net (Mike Miller) Date: Mon, 27 Nov 2017 18:31:28 -0800 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: <5A1C813A.3040904@canterbury.ac.nz> References: <20171127135502.GF22248@ando.pearwood.info> <5A1C813A.3040904@canterbury.ac.nz> Message-ID: <1c02c322-13bc-acf2-dc5d-88804878e539@mgmiller.net> Hmm, I didn't like the options below because they say to me, "consume everything:" x, y, * = iterable x, y, ... = iterable Believe the question behind the idea was, how to grab a couple items and then *stop?* If the syntax route is chosen, I'd expect something that tells me it is going to stop, like a "full stop" as the period/dot is called in jolly ol' England, e.g.: x, y, . = iterable Not sure about the second comma though. -Mike On 2017-11-27 13:18, Greg Ewing wrote: > Chris Angelico wrote: >> The problem is that it depends on internal whitespace to >> distinguish it from augmented assignment; > > Ah, didn't spot that. I guess the ellipsis is the next best > thing then. > > An alternative would be to require parens: > > ?? (x, y, *) = z From jjmaldonis at gmail.com Mon Nov 27 21:49:40 2017 From: jjmaldonis at gmail.com (Jason Maldonis) Date: Mon, 27 Nov 2017 20:49:40 -0600 Subject: [Python-ideas] Repurpose `assert' into a general-purpose check In-Reply-To: References: Message-ID: > > Assertions should not be used as shorthands for "if cond: raise Exc" > in the general case. > I'm just a lurker and usually I agree with why the suggested features shouldn't be implemented, but I actually might chime in to pitch this one a bit more -- and I think it can be done nicely without breaking backward compatibility. As a scientist, I like assert statements for two reasons: 1) you can disable them to speed up the code, and 2) it's how we think as scientists. Consider these two pieces of code, and which one you'd prefer to read: if not condition: raise ValueError assert condition: raise ValueError As a scientist, I prefer the second one because I naturally read it as: "condition is true, therefore ..." and I can predict the next step of the algorithm naturally by filling in the "..." in my mind. It makes the assumptions of the code a bit more explicit than the code `if not condition:`, which I must think about and to translate to "the condition must be true" before I can continue reading. That, in addition to being able to disable the assert statements, makes me like and use them a reasonable amount. However, every time I do use them, I always think "crap, when this breaks I'm going to have to come back here and read my code/comments because the error message isn't very helpful", and that makes me not want to write assert statements. So I like writing them because while I'm writing/reading the code, they make sense, but I don't like reading their error message output because it isn't useful to me as a user/developer. I realize this is very minor, but I actually really like it, and I think the below syntax would be pretty nice and backwards compatible: assert condition: raise ValueError Jason -------------- next part -------------- An HTML attachment was scrubbed... URL: From neatnate at gmail.com Mon Nov 27 22:21:04 2017 From: neatnate at gmail.com (Nathan Schneider) Date: Mon, 27 Nov 2017 22:21:04 -0500 Subject: [Python-ideas] Repurpose `assert' into a general-purpose check In-Reply-To: References: Message-ID: On Mon, Nov 27, 2017 at 9:28 PM, Ivan Pozdeev via Python-ideas < python-ideas at python.org> wrote: > On 28.11.2017 5:19, Chris Angelico wrote: > > Actually, Python does have a way of disabling assertions (the -O >> flag), so they should be treated the same way they are in C. >> Assertions should not be used as shorthands for "if cond: raise Exc" >> in the general case. >> > I'm claiming, and provided evidence, that there are no use cases for this > in Python, so no-one (of any significance) will suffer when the disabling > is cut out. > In any case, we will probably do it with a __future__ statement for a > transitional period. > > I think it would be interesting to investigate how assert statements are used in the wild. I can think of three kinds of uses: 1) Nonredundant checking: The assertion aims to be a concise way to raise exceptions under certain conditions that might arise even presuming that the code is internally correct. (For example, to check that the interface with another piece of code is as expected, or to validate user input.) This seems to be the use case the OP is targeting. It is probably not the use envisioned by the designers of the assert statement. But perhaps assert is frequently used in this way by people who don't know about the optimization flag or assume their program won't be run with optimization. 2) Redundant checking: The assertion acts as a test of code correctness. If the code is correct, it should have no effect; and presuming the code is well-tested, it can thus be ignored with optimization turned on. But it is useful as a sanity check of a tricky algorithm, and can help a reader to understand the implementation. Is it often the case that optimizing these assertions away brings significant performance savings? 3) Temporary debugging: I often write something like assert False,(x,y,z) as a quick way to force the program to terminate and print some values. Unlike print(x,y,z), this has the advantage of being blatantly obvious as debugging code that does not belong in the final version. It seems unlikely that one would purposefully turn on optimization in conjunction with temporary assertions, so this use case may be irrelevant to the proposal. Best, Nathan -------------- next part -------------- An HTML attachment was scrubbed... URL: From bunslow at gmail.com Mon Nov 27 22:22:59 2017 From: bunslow at gmail.com (bunslow) Date: Mon, 27 Nov 2017 21:22:59 -0600 Subject: [Python-ideas] Using an appropriate tone in emails (was: Adding a thin wrapper class around the functions in stdlib.heapq) In-Reply-To: References: Message-ID: On Mon, Nov 27, 2017 at 4:36 PM, Paul Moore wrote: > On 27 November 2017 at 21:59, Nick Timkovich > wrote: > > On Mon, Nov 27, 2017 at 8:17 PM, Brett Cannon wrote: > >> > >> But calling it "atrocious" and so bad that it needs to be fixed > >> "immediately" as if it's a blight upon the stdlib is unnecessarily > insulting > >> to those that have worked on the module. To convey the feeling that you > >> think an OO wrapper would be helpful as the current design doesn't work > for > >> you, you could just phrase it as I just did to get the same point across > >> without insulting anyone. Basically if you wouldn't like your own work > >> called "atrocious" by someone you respect, then it's probably best to > not > >> use that phrasing when talking about a stranger's code either. > > > > > > Sorry for the curt tone, I did lose some sight on the code being > designed by > > people rather than a faceless organization. My intention wasn't to > disparage > > the original authors but sprung more out of my frustration and perception > > from that thread and those before that the status quo would not change > and > > that if a contribution was proffered, would simply be dismissed or > ignored. > > To motivate any change, there must be some argument levied against the > > status quo, but hopefully I can articulate it better. > > > > That little corner is something I'm interested in, and not having > > contributed to CPython before, I'm unsure how it "really works". The > steps > > at https://devguide.python.org/stdlibchanges/ suggest trying to elicit > > community feedback from the lists as a step, so negative feedback tends > to > > kill the enthusiasm to actually make the PR. In the absence of code, > > concrete arguments are almost impossible as we're discussing the shape of > > clouds. > > In my experience (and this reiterates Brett's point) the proposals > that get the best responses are those that are presented positively - > instead of focusing on the (perceived) problems with the current > situation, describe the benefits that will come from the proposed > change. If you can't do that, then it's unlikely there is enough > justification for a change. Certainly, negative feedback can be > demotivating, and when you have a great idea and all you hear is "but > what if...?" it's hard to remain positive. But you're not going to get > better feedback if you criticise - at best, people will stop > listening, and you'll have avoided some of the arguments, but at the > cost of no-one being willing to support your proposal and so it dies. > My first submission to this list was predicated on what I'd read in PEPs -- and many of those, since they recommend major-enough changes to require a PEP, have sections (often lengthy) dedicated to "what's wrong with the status quo". My attempt to imitate that obviously crossed some boundaries in retrospect, and of course now that it's brought up here I see that spinning it as "what can be done to make it better" is psychologically much more effective than "why the current way sucks" (because semantically these are either approximately or exactly the same). But that's where it came from, at least with some of my earlier threads, and I suspect the author of the topic message of the OP will have a similar sentiment. (One major example I can point to is PEP 465 -- because it proposed such a major change to the language, literally half its text amounts to "what's wrong with the status quo", quantifiably and repeatedly. It was also a highly persuasive PEP due in no small part to its "why current things suck" section.) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ned at nedbatchelder.com Mon Nov 27 22:34:37 2017 From: ned at nedbatchelder.com (Ned Batchelder) Date: Mon, 27 Nov 2017 22:34:37 -0500 Subject: [Python-ideas] Repurpose `assert' into a general-purpose check In-Reply-To: References: Message-ID: <39071bb0-f93a-af09-9ca5-67b75d05234f@nedbatchelder.com> On 11/27/17 9:12 PM, Ivan Pozdeev via Python-ideas wrote: > The `assert' statment was created the same as in previous languages > like C/C++: a check to only do in debug mode, when you can't yet trust > your code to manage and pass around internal data correctly. Examples > are array bounds and object state integrity constraints. > > Unlike C, Python does the aforementioned checks all the time, i.e. > it's effectively always in "debug mode". Furthermore, validation > checks are an extremily common use case. > > I have been using assert in my in-house code for input validation for > a long time, the only thing preventing me from doing the same in > public code is the fact that it only raises AssertionError's while > library code is expected to raise TypeError or ValueError on invalid > input. > > The last drop that incited me to do this proposition was > https://stackoverflow.com/questions/2142202/what-are-acceptable-use-cases-for-pythons-assert-statement > where none of the _seven_ answer authors (beside me) managed to make a > _single realistic use case_ for `assert' as a debug-only check. > > --- > > So, I'm hereby proposing to: > > * make `assert' stay in optimized mode > * change its syntax to raise other types of exceptions > > E.g.: "assert condition, type, value" or "assert condition, type, > exception_constructor_argument(s)" to maintain backward compatibility. > You are proposing: ??? assert condition, type, value Why not just use Python as it is now, with this:? ??? if not condition: raise type(value) I don't see a reason to change the assert statement. You can do what you need without the change. --Ned. From vano at mail.mipt.ru Mon Nov 27 23:35:45 2017 From: vano at mail.mipt.ru (Ivan Pozdeev) Date: Tue, 28 Nov 2017 07:35:45 +0300 Subject: [Python-ideas] Repurpose `assert' into a general-purpose check In-Reply-To: <39071bb0-f93a-af09-9ca5-67b75d05234f@nedbatchelder.com> References: <39071bb0-f93a-af09-9ca5-67b75d05234f@nedbatchelder.com> Message-ID: <21a0d033-553c-b7dc-40b3-f957db7be1e0@mail.mipt.ru> On 28.11.2017 6:34, Ned Batchelder wrote: > > You are proposing: > > ??? assert condition, type, value Not specifically this, that's just an example. Actually, the way I'm using them, ??? assert condition, "error message", type would probably be the most expressive way. > > Why not just use Python as it is now, with this:? It's the most expressive way the language provides to write that logic. With Python's design focus on promoting expressive, readable and intuitive syntax, that's enough of a reason. > > ??? if not condition: raise type(value) > > I don't see a reason to change the assert statement. You can do what > you need without the change. I can do anything in any Turing-complete language without any changes to the language. That's no reason to never change anything, is it. The rationale basically is: * As it was intended, the statement has no practical use -- basically a rudiment, due to disappear eventually * It can instead be reused as syntax sugar to cover a very common use case -- Regards, Ivan From greg.ewing at canterbury.ac.nz Tue Nov 28 00:07:15 2017 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 28 Nov 2017 18:07:15 +1300 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: References: <5A1C7FDD.6080107@canterbury.ac.nz> Message-ID: <5A1CEF03.4030909@canterbury.ac.nz> Kirill Balunov wrote: > So someone too perlish > clever can assume that with the proposed syntax: > > >>> def gen(): > >>> for i in ['a', 'b', 'c', 'd']: > >>> v = x if 'x' in globals() else 'var ' > >>> yield v + i > > >>> x, y = gen() > >>> x > var a > >>> y > var ab There's no need for that to happen. It can still unpack all the values it needs before performing any assignments. -- Greg From greg.ewing at canterbury.ac.nz Tue Nov 28 00:15:47 2017 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 28 Nov 2017 18:15:47 +1300 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: <20171128003155.GL22248@ando.pearwood.info> References: <5A1C7FDD.6080107@canterbury.ac.nz> <5A1C8E9C.7090706@canterbury.ac.nz> <20171128003155.GL22248@ando.pearwood.info> Message-ID: <5A1CF103.7020804@canterbury.ac.nz> Steven D'Aprano wrote: > How does "stop iterating here" equate to a wildcard? The * means "I don't care what else the sequence has in it". Because I don't care, there's no need to iterate any further. -- Greg From steve at pearwood.info Tue Nov 28 00:41:35 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 28 Nov 2017 16:41:35 +1100 Subject: [Python-ideas] Repurpose `assert' into a general-purpose check In-Reply-To: References: Message-ID: <20171128054135.GN22248@ando.pearwood.info> On Tue, Nov 28, 2017 at 05:12:36AM +0300, Ivan Pozdeev via Python-ideas wrote: > The `assert' statment was created the same as in previous languages like > C/C++: a check to only do in debug mode, when you can't yet trust your > code to manage and pass around internal data correctly. That's not the sole use for assertions. Assertions are useful as checked comments, as "this can't happen" checks, as checks on your algorithm logic, as a poor man's form of design by contract, and more. > Unlike C, Python does the aforementioned checks all the time, i.e. it's > effectively always in "debug mode". Apart from -O which disables assertions. But in any case, the best use of assertions is not checking things which the interpreter is going to do anyway, but checking things which the interpreter can not and does not check automatically: your program logic. There is no way that the Python interpreter is going to do this check automatically, unless I write the assertion: assert 0 <= r < abs(y) That is copied straight out of one of my functions. The point is, if you are only using assertions to check for things which the interpreter already does, you're not making good use of assertions. > Furthermore, validation checks are an extremily common use case. > I have been using assert in my in-house code for input validation for a > long time, I maintain that using assert for input validation for public functions (even if "public" in this case remains in-house) is certainly an abuse of assert, and STRONGLY oppose any change that supports that bad habit. Its okay to use asserts to validate input to *private* functions and methods (although even that is a slight code smell) but not public functions. > So, I'm hereby proposing to: > > * make `assert' stay in optimized mode > * change its syntax to raise other types of exceptions Very strong -1 on this. We already have syntax for raising other sorts of exceptions: raise. if condition: raise Exception(message) is the right way to spell assert not condition, Exception, message The two require virtually the same amount of typing, so you don't even save any effort by using assert with an alternate exception type. Leave assert for assertions. It is a *good* thing that assertions are visually distinct from input error checking. As things stand now, assert is a clear signal to the reader that this is an internal check which may be disabled safely. -- Steve From steve at pearwood.info Tue Nov 28 00:59:56 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 28 Nov 2017 16:59:56 +1100 Subject: [Python-ideas] Repurpose `assert' into a general-purpose check In-Reply-To: <21a0d033-553c-b7dc-40b3-f957db7be1e0@mail.mipt.ru> References: <39071bb0-f93a-af09-9ca5-67b75d05234f@nedbatchelder.com> <21a0d033-553c-b7dc-40b3-f957db7be1e0@mail.mipt.ru> Message-ID: <20171128055956.GO22248@ando.pearwood.info> On Tue, Nov 28, 2017 at 07:35:45AM +0300, Ivan Pozdeev via Python-ideas wrote: > Actually, the way I'm using them, > > ??? assert condition, "error message", type > > would probably be the most expressive way. I disagree that is expressive -- I call it *misleading*. I see something which looks like an assertion (that is, a checked comment, a contract, a check on an internal piece of logic etc) but it is actually being used as a test. > I can do anything in any Turing-complete language without any changes to > the language. That's no reason to never change anything, is it. "We can change this" is not a reason to change this. There needs to be a *good* reason to change, and you have given no good reasons for this change. > The rationale basically is: > * As it was intended, the statement has no practical use -- basically a > rudiment, due to disappear eventually Nonsense. I make extensive use of assert as a way of checking assertions, and I will fight tooth and nail against any proposal to either remove it or to misuse it for public input tests instead of assertions. > * It can instead be reused as syntax sugar to cover a very common use case There is no need for such syntactic sugar. It would be harmful to use assert for something which is not an assertion. -- Steve From turnbull.stephen.fw at u.tsukuba.ac.jp Tue Nov 28 01:11:25 2017 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Tue, 28 Nov 2017 15:11:25 +0900 Subject: [Python-ideas] generator vs iterator etc. (was: How assignment should work with generators?) In-Reply-To: <20171127234449.GJ22248@ando.pearwood.info> References: <20171127234449.GJ22248@ando.pearwood.info> Message-ID: <23068.65037.184603.974838@turnbull.sk.tsukuba.ac.jp> Steven D'Aprano writes: > The subset of iterators which are created as generators are *also* > called generators, As long as we're being precise, I don't think that is precisely correct: >>> (x for x in range(1)) at 0x10dee5e08> >>> iter(range(1)) >>> iter((1,)) The two iterators have the same duck-type, the generator is different. A generator (object) is, of course, an interable. > You are right that sometimes the term "generator" is used as > shorthand for "generator function". I've always thought "generator factory" would be a better term, but "generator function" will do. I generally use "generator object" to make the distinction, though. > Most of the time the distinction doesn't actually matter, since you > cannot (easily?) create a generator without first creating a > generator function. At least you can create a generator (object) with the generator function created and called implicitly by using a generator expression. Reverting from pedantic mode. Hear, hear! this: > Or if it does matter, it is clear in context which is meant. > > For those few times where it *does* matter, there is no substitute for > precision in language, and that depends on the author, not the > terminology. Steve From steve at pearwood.info Tue Nov 28 01:15:23 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 28 Nov 2017 17:15:23 +1100 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: <1c02c322-13bc-acf2-dc5d-88804878e539@mgmiller.net> References: <20171127135502.GF22248@ando.pearwood.info> <5A1C813A.3040904@canterbury.ac.nz> <1c02c322-13bc-acf2-dc5d-88804878e539@mgmiller.net> Message-ID: <20171128061523.GP22248@ando.pearwood.info> On Mon, Nov 27, 2017 at 06:31:28PM -0800, Mike Miller wrote: > Believe the question behind the idea was, how to grab a couple items and > then *stop?* If the syntax route is chosen, I'd expect something that > tells me it is going to stop, like a "full stop" as the period/dot is > called in jolly ol' England, e.g.: > > x, y, . = iterable Sadly, that fails the "syntax should not look like grit on Tim's monitor" test. Ellipsis at least has three pieces of grit in sequence, which makes it more noticable. > Not sure about the second comma though. Without the comma, it will be visually too hard to distinguish from x, y , = iterable -- Steve From gadgetsteve at live.co.uk Tue Nov 28 01:00:58 2017 From: gadgetsteve at live.co.uk (Steve Barnes) Date: Tue, 28 Nov 2017 06:00:58 +0000 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: References: <20171127135502.GF22248@ando.pearwood.info> <5A1C813A.3040904@canterbury.ac.nz> Message-ID: On 27/11/2017 21:49, Guido van Rossum wrote: > On Mon, Nov 27, 2017 at 1:32 PM, Chris Angelico > wrote: > > On Tue, Nov 28, 2017 at 8:24 AM, Guido van Rossum > wrote: > > On Mon, Nov 27, 2017 at 1:18 PM, Greg Ewing > > > wrote: > >> > >> Chris Angelico wrote: > >>> > >>> The problem is that it depends on internal whitespace to > >>> distinguish it from augmented assignment; > >> > >> > >> Ah, didn't spot that. I guess the ellipsis is the next best > >> thing then. > >> > >> An alternative would be to require parens: > >> > >>? ? (x, y, *) = z > > > > > > But that would have the same issue. > > > > Is this problem really important enough that it requires dedicated syntax? > > Isn't the itertools-based solution good enough? (Or failing that, couldn't > > we add something to itertools to make it more readable rather than going > > straight to new syntax?) > > I don't think there's much that can be done without syntax; the > biggest problem IMO is that you need to tell islice how many targets > it'll be assigned into. It needs some interpreter support to express > "grab as many as you have targets for, leaving everything else behind" > without stating how many that actually is. So the question is whether > that is sufficiently useful to justify extending the syntax. There are > a number of potential advantages and several competing syntax options, > and this suggestion keeps coming up, so I think a PEP is warranted. > > > OK, that's reasonable, and at first blush the ellipsis proposal looks > okay. My PEP queue for Python 3.7 is full though, so I would like to put > this off until 3.8. > > -- > --Guido van Rossum (python.org/~guido Can we please take a note to ensure any future PEP clearly states which ellipsis (personally I prefer the first) of: - as 3 consecutive full stop characters (U+002E) i.e. ... - the Chicago system of 3 space separated full stops . . . - Unicode Horizontal ellipsis (U+2026) (at least there is a keyboard short cut for this) ? - Unicode Midline horizontal ellipsis (U+22EF) ? - any of the other ellipsis characters (https://en.wikipedia.org/wiki/Ellipsis#Computer_representations) As clarifying this early could save a lot of later discussion such as the recent minus, hyphen, underscore debate. -- Steve (Gadget) Barnes Any opinions in this message are my personal opinions and do not reflect those of my employer. --- This email has been checked for viruses by AVG. http://www.avg.com From electronnn at gmail.com Tue Nov 28 01:24:16 2017 From: electronnn at gmail.com (electronnn at gmail.com) Date: Tue, 28 Nov 2017 09:54:16 +0330 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: <20171128061523.GP22248@ando.pearwood.info> References: <20171127135502.GF22248@ando.pearwood.info> <5A1C813A.3040904@canterbury.ac.nz> <1c02c322-13bc-acf2-dc5d-88804878e539@mgmiller.net> <20171128061523.GP22248@ando.pearwood.info> Message-ID: How about x, y ?= iterable with ?= being the lazy assignment operator? On Tue, Nov 28, 2017 at 9:45 AM, Steven D'Aprano wrote: > On Mon, Nov 27, 2017 at 06:31:28PM -0800, Mike Miller wrote: > > > Believe the question behind the idea was, how to grab a couple items and > > then *stop?* If the syntax route is chosen, I'd expect something that > > tells me it is going to stop, like a "full stop" as the period/dot is > > called in jolly ol' England, e.g.: > > > > x, y, . = iterable > > Sadly, that fails the "syntax should not look like grit on Tim's > monitor" test. Ellipsis at least has three pieces of grit in sequence, > which makes it more noticable. > > > > Not sure about the second comma though. > > Without the comma, it will be visually too hard to distinguish from > > x, y , = iterable > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Tue Nov 28 01:22:39 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 28 Nov 2017 17:22:39 +1100 Subject: [Python-ideas] generator vs iterator etc. (was: How assignment should work with generators?) In-Reply-To: <23068.65037.184603.974838@turnbull.sk.tsukuba.ac.jp> References: <20171127234449.GJ22248@ando.pearwood.info> <23068.65037.184603.974838@turnbull.sk.tsukuba.ac.jp> Message-ID: <20171128062238.GQ22248@ando.pearwood.info> On Tue, Nov 28, 2017 at 03:11:25PM +0900, Stephen J. Turnbull wrote: > Steven D'Aprano writes: > > > The subset of iterators which are created as generators are *also* > > called generators, > > As long as we're being precise, I don't think that is precisely correct: > > >>> (x for x in range(1)) > at 0x10dee5e08> > >>> iter(range(1)) > > >>> iter((1,)) > > > The two iterators have the same duck-type, the generator is different. How is the generator different? It quacks like a range_iterator and tuple_iterator, it swims like them, it flies like them. Is there some iterator method or protocol that generators don't support? > A generator (object) is, of course, an interable. And also an iterator: py> collections.abc py> isinstance((x+1 for x in range(5)), collections.abc.Iterator) True > > Most of the time the distinction doesn't actually matter, since you > > cannot (easily?) create a generator without first creating a > > generator function. > > At least you can create a generator (object) with the generator > function created and called implicitly by using a generator > expression. Ah yes, I forget about generator expressions, thanks. -- Steve From steve at pearwood.info Tue Nov 28 01:31:51 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 28 Nov 2017 17:31:51 +1100 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: <5A1CF103.7020804@canterbury.ac.nz> References: <5A1C7FDD.6080107@canterbury.ac.nz> <5A1C8E9C.7090706@canterbury.ac.nz> <20171128003155.GL22248@ando.pearwood.info> <5A1CF103.7020804@canterbury.ac.nz> Message-ID: <20171128063151.GR22248@ando.pearwood.info> On Tue, Nov 28, 2017 at 06:15:47PM +1300, Greg Ewing wrote: > Steven D'Aprano wrote: > >How does "stop iterating here" equate to a wildcard? > > The * means "I don't care what else the sequence has in it". > > Because I don't care, there's no need to iterate any further. I'll grant you that. But I don't see how that relates to being a wildcard. I'm not seeing the connection. I mean, you wouldn't interpret from module import * to mean "I don't care what's in the module, so don't bother importing anything" would you? So the concept of wildcard here seems to be the opposite to its use here: - in imports, it means "import everything the module offers"; - in extended iterable unpacking, it means "collect everything"; both of which (to me) seem related to the concept of a wildcard; but in this proposed syntax, we have x, y, * = iterable which means the opposite to "collect everything", instead meaning "collect nothing and stop". Anyway, given that * = is visually ambiguous with *= I don't think this syntax is feasible (even though * = is currently a syntax error, or at least it is in 3.5). -- Steve From njs at pobox.com Tue Nov 28 01:55:49 2017 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 27 Nov 2017 22:55:49 -0800 Subject: [Python-ideas] Using an appropriate tone in emails (was: Adding a thin wrapper class around the functions in stdlib.heapq) In-Reply-To: References: Message-ID: On Mon, Nov 27, 2017 at 7:22 PM, bunslow wrote: > My first submission to this list was predicated on what I'd read in PEPs -- > and many of those, since they recommend major-enough changes to require a > PEP, have sections (often lengthy) dedicated to "what's wrong with the > status quo". My attempt to imitate that obviously crossed some boundaries in > retrospect, and of course now that it's brought up here I see that spinning > it as "what can be done to make it better" is psychologically much more > effective than "why the current way sucks" (because semantically these are > either approximately or exactly the same). But that's where it came from, at > least with some of my earlier threads, and I suspect the author of the topic > message of the OP will have a similar sentiment. To quote Brett's original email: > So obviously Nick doesn't like the design of the heapq module. ;) And that's okay! And he's totally within his rights to express the feeling that the heapq module as it stands doesn't meet his needs. > But calling it "atrocious" and so bad that it needs to be fixed "immediately" as if it's a blight upon the stdlib is unnecessarily insulting to those that have worked on the module. You can and should talk about problems with the status quo! But it's totally possible to do this without insulting anyone. Brett's talking about tone, not content. > (One major example I can point to is PEP 465 -- because it proposed such a > major change to the language, literally half its text amounts to "what's > wrong with the status quo", quantifiably and repeatedly. It was also a > highly persuasive PEP due in no small part to its "why current things suck" > section.) Maybe, but you won't find the word "suck" anywhere in that section :-). And of course, the nice thing about PEP 465 is that it's complaining about a missing feature, which sort of by definition means that it's not complaining about anyone in particular's work. Nonetheless, an earlier draft of PEP 465 did inadvertently talk about an old PEP in an overly-flippant manner, and I ended up apologizing to the author and fixing it. (Which of course also made the PEP stronger.) It's cool, no-one's perfect. If you think you've made a mistake, then apologize and try to do better, that's all. -n -- Nathaniel J. Smith -- https://vorpus.org From c at anthonyrisinger.com Tue Nov 28 02:06:05 2017 From: c at anthonyrisinger.com (C Anthony Risinger) Date: Tue, 28 Nov 2017 01:06:05 -0600 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: <20171128063151.GR22248@ando.pearwood.info> References: <5A1C7FDD.6080107@canterbury.ac.nz> <5A1C8E9C.7090706@canterbury.ac.nz> <20171128003155.GL22248@ando.pearwood.info> <5A1CF103.7020804@canterbury.ac.nz> <20171128063151.GR22248@ando.pearwood.info> Message-ID: On Nov 28, 2017 12:32 AM, "Steven D'Aprano" wrote: On Tue, Nov 28, 2017 at 06:15:47PM +1300, Greg Ewing wrote: > Steven D'Aprano wrote: > >How does "stop iterating here" equate to a wildcard? > > The * means "I don't care what else the sequence has in it". > > Because I don't care, there's no need to iterate any further. I'll grant you that. But I don't see how that relates to being a wildcard. I'm not seeing the connection. I mean, you wouldn't interpret from module import * to mean "I don't care what's in the module, so don't bother importing anything" would you? So the concept of wildcard here seems to be the opposite to its use here: - in imports, it means "import everything the module offers"; - in extended iterable unpacking, it means "collect everything"; both of which (to me) seem related to the concept of a wildcard; but in this proposed syntax, we have x, y, * = iterable which means the opposite to "collect everything", instead meaning "collect nothing and stop". Anyway, given that * = is visually ambiguous with *= I don't think this syntax is feasible (even though * = is currently a syntax error, or at least it is in 3.5). If not already considered, what if the RHS had to be explicitly unpacked? Something like: a, b, c = *iterator Which would essentially be: a, b, c = (*iterator,) This enables lazy assignment by default but `*` can force complete expansion (and exact matching) of the RHS. It's a breaking change, but it does have a straightforward fix (simply wrap and unpack any relevant RHS). Thanks, -- C Anthony -------------- next part -------------- An HTML attachment was scrubbed... URL: From vano at mail.mipt.ru Tue Nov 28 02:11:46 2017 From: vano at mail.mipt.ru (Ivan Pozdeev) Date: Tue, 28 Nov 2017 10:11:46 +0300 Subject: [Python-ideas] Repurpose `assert' into a general-purpose check In-Reply-To: <20171128055956.GO22248@ando.pearwood.info> References: <39071bb0-f93a-af09-9ca5-67b75d05234f@nedbatchelder.com> <21a0d033-553c-b7dc-40b3-f957db7be1e0@mail.mipt.ru> <20171128055956.GO22248@ando.pearwood.info> Message-ID: <5fb6604e-d412-7b26-fb28-564bfaf00b43@mail.mipt.ru> On 28.11.2017 8:59, Steven D'Aprano wrote: > On Tue, Nov 28, 2017 at 07:35:45AM +0300, Ivan Pozdeev via Python-ideas wrote: > >> Actually, the way I'm using them, >> >> ??? assert condition, "error message", type >> >> would probably be the most expressive way. > I disagree that is expressive -- I call it *misleading*. I see something > which looks like an assertion (that is, a checked comment, a contract, a > check on an internal piece of logic etc) but it is actually being used > as a test. > > >> I can do anything in any Turing-complete language without any changes to >> the language. That's no reason to never change anything, is it. > "We can change this" is not a reason to change this. There needs to be a > *good* reason to change, and you have given no good reasons for this > change. > > >> The rationale basically is: >> * As it was intended, the statement has no practical use -- basically a >> rudiment, due to disappear eventually > Nonsense. I make extensive use of assert as a way of checking > assertions, and I will fight tooth and nail against any proposal to > either remove it or to misuse it for public input tests instead of > assertions. I invite you to show me a single use case for those "assertions" because after ~20 years of experience in coding (that included fairly large projects), I've yet to see one. Any, every check that you make at debug time either * belongs in production as well (all the more because it's harder to diagnose there), or * belongs in a test -- something coded independently from the program (if your code as a whole cannot be trusted, how any specific part of it can?), or * isn't needed at all because a fault will inevitably surface somewhere down the line (as some exception or an incorrect result that a test will catch). Finally, I've got much experience using existing code outside its original use cases, where the original author's assumptions may no longer hold but the specific logic can be gauded to produce the desired result. Coding these assumptions in would undermine that goal. So, I see "debug assertions" as either intentionally compromizing correctness for performance (a direct opposite of Python's design principles), or as an inferiour, faulty, half-measure rudiment from times when CI wasn't a thing (thus not something that should be taught and promoted as a best practice any longer). > >> * It can instead be reused as syntax sugar to cover a very common use case > There is no need for such syntactic sugar. It would be harmful > to use assert for something which is not an assertion. > > > -- Regards, Ivan From elazarg at gmail.com Tue Nov 28 02:22:35 2017 From: elazarg at gmail.com (Elazar) Date: Tue, 28 Nov 2017 07:22:35 +0000 Subject: [Python-ideas] Repurpose `assert' into a general-purpose check In-Reply-To: <5fb6604e-d412-7b26-fb28-564bfaf00b43@mail.mipt.ru> References: <39071bb0-f93a-af09-9ca5-67b75d05234f@nedbatchelder.com> <21a0d033-553c-b7dc-40b3-f957db7be1e0@mail.mipt.ru> <20171128055956.GO22248@ando.pearwood.info> <5fb6604e-d412-7b26-fb28-564bfaf00b43@mail.mipt.ru> Message-ID: Just a note : in typechecked code (such as mypy's source code) assert is used to guide the checker: assert isinstance(x, CallableType) return x.args # checker knows it's valid So the assert becomes a kind of type annotation. The runtime check helps during tests, but is not that important - failure will be caught relatively soon. And I believe that the ability to remove the check at runtime is important, since isinstance calls have non-negligible impact on performance in mypy. (but other contributors here can correct me on this). Elazar ?????? ??? ??, 28 ????? 2017, 09:12, ??? Ivan Pozdeev via Python-ideas ?< python-ideas at python.org>: > On 28.11.2017 8:59, Steven D'Aprano wrote: > > On Tue, Nov 28, 2017 at 07:35:45AM +0300, Ivan Pozdeev via Python-ideas > wrote: > > > >> Actually, the way I'm using them, > >> > >> assert condition, "error message", type > >> > >> would probably be the most expressive way. > > I disagree that is expressive -- I call it *misleading*. I see something > > which looks like an assertion (that is, a checked comment, a contract, a > > check on an internal piece of logic etc) but it is actually being used > > as a test. > > > > > >> I can do anything in any Turing-complete language without any changes to > >> the language. That's no reason to never change anything, is it. > > "We can change this" is not a reason to change this. There needs to be a > > *good* reason to change, and you have given no good reasons for this > > change. > > > > > >> The rationale basically is: > >> * As it was intended, the statement has no practical use -- basically a > >> rudiment, due to disappear eventually > > Nonsense. I make extensive use of assert as a way of checking > > assertions, and I will fight tooth and nail against any proposal to > > either remove it or to misuse it for public input tests instead of > > assertions. > I invite you to show me a single use case for those "assertions" because > after ~20 years of experience in coding (that included fairly large > projects), I've yet to see one. > > Any, every check that you make at debug time either > * belongs in production as well (all the more because it's harder to > diagnose there), or > * belongs in a test -- something coded independently from the program > (if your code as a whole cannot be trusted, how any specific part of it > can?), or > * isn't needed at all because a fault will inevitably surface somewhere > down the line (as some exception or an incorrect result that a test will > catch). > > Finally, I've got much experience using existing code outside its > original use cases, where the original author's assumptions may no > longer hold but the specific logic can be gauded to produce the desired > result. Coding these assumptions in would undermine that goal. > > So, I see "debug assertions" as either intentionally compromizing > correctness for performance (a direct opposite of Python's design > principles), or as an inferiour, faulty, half-measure rudiment from > times when CI wasn't a thing (thus not something that should be taught > and promoted as a best practice any longer). > > > >> * It can instead be reused as syntax sugar to cover a very common use > case > > There is no need for such syntactic sugar. It would be harmful > > to use assert for something which is not an assertion. > > > > > > > > -- > Regards, > Ivan > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kirillbalunov at gmail.com Tue Nov 28 02:48:52 2017 From: kirillbalunov at gmail.com (Kirill Balunov) Date: Tue, 28 Nov 2017 10:48:52 +0300 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: References: <20171127135502.GF22248@ando.pearwood.info> <5A1C813A.3040904@canterbury.ac.nz> Message-ID: 2017-11-28 0:52 GMT+03:00 Chris Angelico : > On Tue, Nov 28, 2017 at 8:49 AM, Guido van Rossum > wrote: > > My PEP queue for Python 3.7 is full though, so I would like to put this > off > > until 3.8. > > > > Yeah, I don't think this could reasonably be raced into 3.7 even if it > were critically important, and it's not. 3.8 will be fine. > > Kirill, do you want to spearhead the discussion? I'm happy to help out. > Yes of course! With kind regards, -gdg -------------- next part -------------- An HTML attachment was scrubbed... URL: From vano at mail.mipt.ru Tue Nov 28 02:58:34 2017 From: vano at mail.mipt.ru (Ivan Pozdeev) Date: Tue, 28 Nov 2017 10:58:34 +0300 Subject: [Python-ideas] Repurpose `assert' into a general-purpose check In-Reply-To: References: <39071bb0-f93a-af09-9ca5-67b75d05234f@nedbatchelder.com> <21a0d033-553c-b7dc-40b3-f957db7be1e0@mail.mipt.ru> <20171128055956.GO22248@ando.pearwood.info> <5fb6604e-d412-7b26-fb28-564bfaf00b43@mail.mipt.ru> Message-ID: <46f336fa-ea1a-bc51-73dd-f0c4535e8913@mail.mipt.ru> On 28.11.2017 10:22, Elazar wrote: > Just a note : in typechecked code (such as mypy's source code) assert > is used to guide the checker: > > assert isinstance(x, CallableType) > return x.args? # checker knows it's valid > > So the assert becomes a kind of type annotation. The runtime check > helps during tests, but is not that important - failure will be caught > relatively soon. And I believe that the ability to remove the check at > runtime is important, since isinstance calls have non-negligible > impact on performance in mypy. > (but other contributors here can correct me on this). > This results in two different interfaces. In normal mode, it enforces types while in -O, accepts anything that passes the duck test. > > Elazar > > > ?????? ??? ??, 28 ????? 2017, 09:12, ??? Ivan Pozdeev via Python-ideas > ?>: > > On 28.11.2017 8:59, Steven D'Aprano wrote: > > On Tue, Nov 28, 2017 at 07:35:45AM +0300, Ivan Pozdeev via > Python-ideas wrote: > > > >> Actually, the way I'm using them, > >> > >>? ??? assert condition, "error message", type > >> > >> would probably be the most expressive way. > > I disagree that is expressive -- I call it *misleading*. I see > something > > which looks like an assertion (that is, a checked comment, a > contract, a > > check on an internal piece of logic etc) but it is actually > being used > > as a test. > > > > > >> I can do anything in any Turing-complete language without any > changes to > >> the language. That's no reason to never change anything, is it. > > "We can change this" is not a reason to change this. There needs > to be a > > *good* reason to change, and you have given no good reasons for this > > change. > > > > > >> The rationale basically is: > >> * As it was intended, the statement has no practical use -- > basically a > >> rudiment, due to disappear eventually > > Nonsense. I make extensive use of assert as a way of checking > > assertions, and I will fight tooth and nail against any proposal to > > either remove it or to misuse it for public input tests instead of > > assertions. > I invite you to show me a single use case for those "assertions" > because > after ~20 years of experience in coding (that included fairly large > projects), I've yet to see one. > > Any, every check that you make at debug time either > * belongs in production as well (all the more because it's harder to > diagnose there), or > * belongs in a test -- something coded independently from the program > (if your code as a whole cannot be trusted, how any specific part > of it > can?), or > * isn't needed at all because a fault will inevitably surface > somewhere > down the line (as some exception or an incorrect result that a > test will > catch). > > Finally, I've got much experience using existing code outside its > original use cases, where the original author's assumptions may no > longer hold but the specific logic can be gauded to produce the > desired > result. Coding these assumptions in would undermine that goal. > > So, I see "debug assertions" as either intentionally compromizing > correctness for performance (a direct opposite of Python's design > principles), or as an inferiour, faulty, half-measure rudiment from > times when CI wasn't a thing (thus not something that should be taught > and promoted as a best practice any longer). > > > >> * It can instead be reused as syntax sugar to cover a very > common use case > > There is no need for such syntactic sugar. It would be harmful > > to use assert for something which is not an assertion. > > > > > > > > -- > Regards, > Ivan > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Regards, Ivan -------------- next part -------------- An HTML attachment was scrubbed... URL: From bunslow at gmail.com Tue Nov 28 03:26:10 2017 From: bunslow at gmail.com (bunslow) Date: Tue, 28 Nov 2017 02:26:10 -0600 Subject: [Python-ideas] Using an appropriate tone in emails (was: Adding a thin wrapper class around the functions in stdlib.heapq) In-Reply-To: References: Message-ID: I certainly didn't take away the right lesson! And lesson well learned, hopefully. On Tue, Nov 28, 2017 at 12:55 AM, Nathaniel Smith wrote: > On Mon, Nov 27, 2017 at 7:22 PM, bunslow wrote: > > My first submission to this list was predicated on what I'd read in PEPs > -- > > and many of those, since they recommend major-enough changes to require a > > PEP, have sections (often lengthy) dedicated to "what's wrong with the > > status quo". My attempt to imitate that obviously crossed some > boundaries in > > retrospect, and of course now that it's brought up here I see that > spinning > > it as "what can be done to make it better" is psychologically much more > > effective than "why the current way sucks" (because semantically these > are > > either approximately or exactly the same). But that's where it came > from, at > > least with some of my earlier threads, and I suspect the author of the > topic > > message of the OP will have a similar sentiment. > > To quote Brett's original email: > > So obviously Nick doesn't like the design of the heapq module. ;) And > that's okay! And he's totally within his rights to express the feeling that > the heapq module as it stands doesn't meet his needs. > > But calling it "atrocious" and so bad that it needs to be fixed > "immediately" as if it's a blight upon the stdlib is unnecessarily > insulting to those that have worked on the module. > > You can and should talk about problems with the status quo! But it's > totally possible to do this without insulting anyone. Brett's talking > about tone, not content. > > > (One major example I can point to is PEP 465 -- because it proposed such > a > > major change to the language, literally half its text amounts to "what's > > wrong with the status quo", quantifiably and repeatedly. It was also a > > highly persuasive PEP due in no small part to its "why current things > suck" > > section.) > > Maybe, but you won't find the word "suck" anywhere in that section > :-). And of course, the nice thing about PEP 465 is that it's > complaining about a missing feature, which sort of by definition means > that it's not complaining about anyone in particular's work. > > Nonetheless, an earlier draft of PEP 465 did inadvertently talk about > an old PEP in an overly-flippant manner, and I ended up apologizing to > the author and fixing it. (Which of course also made the PEP > stronger.) It's cool, no-one's perfect. If you think you've made a > mistake, then apologize and try to do better, that's all. > > -n > > -- > Nathaniel J. Smith -- https://vorpus.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Tue Nov 28 03:29:08 2017 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 28 Nov 2017 09:29:08 +0100 Subject: [Python-ideas] Repurpose `assert' into a general-purpose check In-Reply-To: References: Message-ID: <6e216b81-7001-8eb5-78d3-80b431bff54a@egenix.com> On 28.11.2017 03:28, Ivan Pozdeev via Python-ideas wrote: > On 28.11.2017 5:19, Chris Angelico wrote: > >> Actually, Python does have a way of disabling assertions (the -O >> flag), so they should be treated the same way they are in C. >> Assertions should not be used as shorthands for "if cond: raise Exc" >> in the general case. > I'm claiming, and provided evidence, that there are no use cases for > this in Python, so no-one (of any significance) will suffer when the > disabling is cut out. ... except those who have relied on this behavior for around two decades to make their code run faster in production environments and those who run on memory constrained systems using -OO to remove doc-strings as well. asserts are meant as debug tool and runtime way to document application internal expectations of code following them. Once all tests pass they are no longer needed. If you find that you still need them, you should recode the asserts as proper if statements. As example use case, asserts testing user provided data are not a good idea, since you cannot test all possible user inputs. For those cases, an if statement is the right way to implement your checks. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Nov 28 2017) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From turnbull.stephen.fw at u.tsukuba.ac.jp Tue Nov 28 03:53:02 2017 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Tue, 28 Nov 2017 17:53:02 +0900 Subject: [Python-ideas] Advocating your ideas [was: Using an appropriate tone in emails] In-Reply-To: References: Message-ID: <23069.9198.135448.205256@turnbull.sk.tsukuba.ac.jp> This discussion started on Python-Ideas (q.v.), and is also somewhat applicable to Python-Dev. I think further discussion belongs on Core-Mentorship, though. Cc'd and reply-to set. bunslow writes: > My first submission to this list was predicated on what I'd read in > PEPs -- and many of those, since they recommend major-enough > changes to require a PEP, have sections (often lengthy) dedicated > to "what's wrong with the status quo". That's a good point! However, even a for modest enhancement, you do need to advocate from "what's wrong with the status quo". But that is most persuasive when it can be phrased as "you can't do X", or at least "you can't do X without Y", and this change allow that feature. And Y usually should not be something that most Python programmers do in the ordinary course of writing code, such as defining functions or classes. This is quite a fine point though. I note that Nick defended a *very* short context manager (the "null" context manager) on the grounds that typical Python programmers think of context managers as being a bit magical. They use them all the time in the recommended idioms for file handling and the like, but they very rarely write them. Again, "writing loop statements" is usually considered something that doesn't qualify as a "Y", but we got comprehensions and then generators. I'm not sure what the lesson is here. Maybe it's that crossing the statement/expression boundary is a Y. Or maybe it's that comprehensions and generators are imports of features successful in other languages, so were relatively easy to accept. Regards, Steve From turnbull.stephen.fw at u.tsukuba.ac.jp Tue Nov 28 03:54:12 2017 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Tue, 28 Nov 2017 17:54:12 +0900 Subject: [Python-ideas] Repurpose `assert' into a general-purpose check In-Reply-To: References: Message-ID: <23069.9268.799460.752975@turnbull.sk.tsukuba.ac.jp> Ivan Pozdeev via Python-ideas writes: > The `assert' statment was created the same as in previous languages like > C/C++: a check to only do in debug mode, when you can't yet trust your > code to manage and pass around internal data correctly. Examples are > array bounds and object state integrity constraints. I use assert in Python as I use it in C: as a statement that something should NEVER happen, algorithmically. I do occasionally have expensive assertions in inner loops, and do use -O to disable them when I assess that speed is important enough to offset the increased risk. So, -1. Oh, yeah: I have had programs stop because I got the algorithm wrong, and because I implemented it incorrectly. I understand the point made elsewhere about scientists' thinking about "assert", but as a scientist myself, I disagree. It is very useful to me to distinguish between validating data and validating algorithms. "assert" is how we do the latter. From steve at pearwood.info Tue Nov 28 06:30:26 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 28 Nov 2017 22:30:26 +1100 Subject: [Python-ideas] Repurpose `assert' into a general-purpose check In-Reply-To: <23069.9268.799460.752975@turnbull.sk.tsukuba.ac.jp> References: <23069.9268.799460.752975@turnbull.sk.tsukuba.ac.jp> Message-ID: <20171128113026.GS22248@ando.pearwood.info> On Tue, Nov 28, 2017 at 05:54:12PM +0900, Stephen J. Turnbull wrote: > I understand the point made elsewhere about scientists' thinking about > "assert", but as a scientist myself, I disagree. It is very useful to > me to distinguish between validating data and validating algorithms. > "assert" is how we do the latter. I'm not sure which point about scientists you are referring to -- I don't seem to have that email. Is that in this thread? -- Steve From ncoghlan at gmail.com Tue Nov 28 07:14:39 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 28 Nov 2017 22:14:39 +1000 Subject: [Python-ideas] generator vs iterator etc. (was: How assignment should work with generators?) In-Reply-To: <23068.65037.184603.974838@turnbull.sk.tsukuba.ac.jp> References: <20171127234449.GJ22248@ando.pearwood.info> <23068.65037.184603.974838@turnbull.sk.tsukuba.ac.jp> Message-ID: On 28 November 2017 at 16:11, Stephen J. Turnbull wrote: > Steven D'Aprano writes: > > > The subset of iterators which are created as generators are *also* > > called generators, > > As long as we're being precise, I don't think that is precisely correct: > > >>> (x for x in range(1)) > at 0x10dee5e08> > >>> iter(range(1)) > > >>> iter((1,)) > > > The two iterators have the same duck-type, the generator is different. > A generator (object) is, of course, an interable. While it's not obvious with the genexp (since they're anonymous), the main reason for the difference in the repr layouts here is just because generator iterators can have names: >>> def g(): yield ... >>> g() So the statement that "generator iterators are iterators" is correct. The functions that create them are called generator functions because they really are functions: >>> g What's more unfortunate here is that the usage of "generator" in the generator-iterator representation doesn't actually align with the preferred terminology in the documentation: https://docs.python.org/3/glossary.html#term-generator So I can understand the confusion here. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Tue Nov 28 07:52:50 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 28 Nov 2017 22:52:50 +1000 Subject: [Python-ideas] Using an appropriate tone in emails (was: Adding a thin wrapper class around the functions in stdlib.heapq) In-Reply-To: References: Message-ID: On 28 November 2017 at 13:22, bunslow wrote: > My first submission to this list was predicated on what I'd read in PEPs -- > and many of those, since they recommend major-enough changes to require a > PEP, have sections (often lengthy) dedicated to "what's wrong with the > status quo". My attempt to imitate that obviously crossed some boundaries in > retrospect, and of course now that it's brought up here I see that spinning > it as "what can be done to make it better" is psychologically much more > effective than "why the current way sucks" (because semantically these are > either approximately or exactly the same). But that's where it came from, at > least with some of my earlier threads, and I suspect the author of the topic > message of the OP will have a similar sentiment. Yeah, by the time someone reaches the point of writing a PEP, there's usually some level of existing awareness along the lines of "The status quo might not be OK any more, so let's explicitly document the risks and benefits associated with a possible change". That means part of the role of the PEP is to summarise the relevant problems with the status quo, such that future readers can understand why any change is being proposed at all. In cases where the proposed change is relatively simple at a technical level, like PEP 479 (converting an unhandled StopIteration to RuntimeError), PEP 538 (coercing the C locale to a UTF-8 based locale), or PEP 565 (tweaking the way we handle DeprecationWarning), the motivation & rationale may end up being the majority of the PEP, since the actual change to be made is relatively minor, but the potential consequences aren't necessarily obvious. By contrast, python-ideas threads usually start at a point earlier in the decision making process: asking ourselves the question "Is the status quo still OK?". In most cases the answer is "Yeah, it's still fine", but we keep asking, because sometimes the answer is "Actually, we could probably improve it by doing...". The easiest trap to fall into on that front is to think to ourselves "The status quo doesn't solve my problems, therefore it doesn't solve anyone's problems", which usually isn't a productive mindset. A more productive framing is typically "The problems that the status quo solves are not the problems that I currently have". It may seem like a small change, but in the second version, we're thinking: - the status quo solves problems for someone, just not for me - whatever I propose should try to avoid making the status quo worse at what it already does - I need to explain the problem I have, not just the potential solution I see and that ends up coming through in the way we write. I'll also note that nobody expects perfection on this front - most of us are thoroughly familiar with the lure of "But someone is *wrong* on the internet" [1], and we know that sometimes it's too late by the time we finally think "Oh, I really should have toned that down a bit before hitting Send...". We just strive to ensure our typical approach is to be respectful of each other, and of past contributors. Cheers, Nick. [1] https://xkcd.com/386/ P.S. For a longer version of the "What problem does it solve?" question in relation to the respective APIs of the requests and urllib modules, folks may be interested in https://www.curiousefficiency.org/posts/2016/08/what-problem-does-it-solve.html -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Tue Nov 28 08:36:50 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 28 Nov 2017 23:36:50 +1000 Subject: [Python-ideas] Repurpose `assert' into a general-purpose check In-Reply-To: <20171128054135.GN22248@ando.pearwood.info> References: <20171128054135.GN22248@ando.pearwood.info> Message-ID: On 28 November 2017 at 15:41, Steven D'Aprano wrote: > On Tue, Nov 28, 2017 at 05:12:36AM +0300, Ivan Pozdeev via Python-ideas wrote: >> Unlike C, Python does the aforementioned checks all the time, i.e. it's >> effectively always in "debug mode". > > Apart from -O which disables assertions. But in any case, the best use > of assertions is not checking things which the interpreter is going to > do anyway, but checking things which the interpreter can not and does > not check automatically: your program logic. There is no way that the > Python interpreter is going to do this check automatically, unless I > write the assertion: > > assert 0 <= r < abs(y) > > That is copied straight out of one of my functions. I'll make the same observation I usually do each time one of these threads comes up: * I'm opposed to making assert substantially different from the way it works now * I'm in favour of adding a new "ensure()" builtin that encapsulates the check-and-raise logic The reasons I prefer this approach: - assert is a statement *solely* so that the compiler can optimise it out. If it's not optional, it doesn't need to be a statement any more - if the existing assert statements are left alone, there are no performance or compatibility concerns for folks that rely on the current behaviour - if it's a function, it doesn't need to be called "assert", it can use the more imperative term "ensure" (meaning "ensure this condition is true before continuing") - if it's a function, it can easily be emulated on old versions via compatibility libraries - "ensure() is required, assert is optional" is a better answer to complaints about assertions being optional than suggesting "if cond: raise AssertionError(msg)" as a reasonable alternative to "assert cond, msg" - if it's a function, we get access to all the regular function machinery, so we're not restricted to positional-only arguments the way the assert statement is My initial proposed behaviour for the function: def ensure(cond, msg=None, exc_type=RuntimeError): """Raise an exception if the given condition is not true""" if not cond: if msg is None: frame = sys._getframe(1) line = frame.f_lineno modname = frame.f_globals.get("__name__", "") msg = f"Condition not met on line {line:d} in {modname!r}" raise exc_type(msg) Cheers, Nick. P.S. No, I'm not offering to write that PEP myself, I'd just be in favour of the idea if someone else were to write it :) -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Tue Nov 28 08:46:59 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 28 Nov 2017 13:46:59 +0000 Subject: [Python-ideas] Repurpose `assert' into a general-purpose check In-Reply-To: References: <20171128054135.GN22248@ando.pearwood.info> Message-ID: On 28 November 2017 at 13:36, Nick Coghlan wrote: > On 28 November 2017 at 15:41, Steven D'Aprano wrote: >> On Tue, Nov 28, 2017 at 05:12:36AM +0300, Ivan Pozdeev via Python-ideas wrote: >>> Unlike C, Python does the aforementioned checks all the time, i.e. it's >>> effectively always in "debug mode". >> >> Apart from -O which disables assertions. But in any case, the best use >> of assertions is not checking things which the interpreter is going to >> do anyway, but checking things which the interpreter can not and does >> not check automatically: your program logic. There is no way that the >> Python interpreter is going to do this check automatically, unless I >> write the assertion: >> >> assert 0 <= r < abs(y) >> >> That is copied straight out of one of my functions. > > I'll make the same observation I usually do each time one of these > threads comes up: > > * I'm opposed to making assert substantially different from the way it works now > * I'm in favour of adding a new "ensure()" builtin that encapsulates > the check-and-raise logic > > The reasons I prefer this approach: > > - assert is a statement *solely* so that the compiler can optimise it > out. If it's not optional, > it doesn't need to be a statement any more > - if the existing assert statements are left alone, there are no > performance or compatibility > concerns for folks that rely on the current behaviour > - if it's a function, it doesn't need to be called "assert", it can use the more > imperative term "ensure" (meaning "ensure this condition is true > before continuing") > - if it's a function, it can easily be emulated on old versions via > compatibility libraries > - "ensure() is required, assert is optional" is a better answer to > complaints about > assertions being optional than suggesting "if cond: raise AssertionError(msg)" > as a reasonable alternative to "assert cond, msg" > - if it's a function, we get access to all the regular function > machinery, so we're > not restricted to positional-only arguments the way the assert statement is > > My initial proposed behaviour for the function: > > def ensure(cond, msg=None, exc_type=RuntimeError): > """Raise an exception if the given condition is not true""" > if not cond: > if msg is None: > frame = sys._getframe(1) > line = frame.f_lineno > modname = frame.f_globals.get("__name__", "") > msg = f"Condition not met on line {line:d} in {modname!r}" > raise exc_type(msg) > > Cheers, > Nick. > > P.S. No, I'm not offering to write that PEP myself, I'd just be in > favour of the idea if someone else were to write it :) +1 on everything Nick said. Paul From fakedme+py at gmail.com Tue Nov 28 09:02:31 2017 From: fakedme+py at gmail.com (Soni L.) Date: Tue, 28 Nov 2017 12:02:31 -0200 Subject: [Python-ideas] Repurpose `assert' into a general-purpose check In-Reply-To: References: <20171128054135.GN22248@ando.pearwood.info> Message-ID: On 2017-11-28 11:36 AM, Nick Coghlan wrote: > On 28 November 2017 at 15:41, Steven D'Aprano wrote: >> On Tue, Nov 28, 2017 at 05:12:36AM +0300, Ivan Pozdeev via Python-ideas wrote: >>> Unlike C, Python does the aforementioned checks all the time, i.e. it's >>> effectively always in "debug mode". >> Apart from -O which disables assertions. But in any case, the best use >> of assertions is not checking things which the interpreter is going to >> do anyway, but checking things which the interpreter can not and does >> not check automatically: your program logic. There is no way that the >> Python interpreter is going to do this check automatically, unless I >> write the assertion: >> >> assert 0 <= r < abs(y) >> >> That is copied straight out of one of my functions. > I'll make the same observation I usually do each time one of these > threads comes up: > > * I'm opposed to making assert substantially different from the way it works now > * I'm in favour of adding a new "ensure()" builtin that encapsulates > the check-and-raise logic > > The reasons I prefer this approach: > > - assert is a statement *solely* so that the compiler can optimise it > out. If it's not optional, > it doesn't need to be a statement any more > - if the existing assert statements are left alone, there are no > performance or compatibility > concerns for folks that rely on the current behaviour > - if it's a function, it doesn't need to be called "assert", it can use the more > imperative term "ensure" (meaning "ensure this condition is true > before continuing") > - if it's a function, it can easily be emulated on old versions via > compatibility libraries > - "ensure() is required, assert is optional" is a better answer to > complaints about > assertions being optional than suggesting "if cond: raise AssertionError(msg)" > as a reasonable alternative to "assert cond, msg" > - if it's a function, we get access to all the regular function > machinery, so we're > not restricted to positional-only arguments the way the assert statement is > > My initial proposed behaviour for the function: > > def ensure(cond, msg=None, exc_type=RuntimeError): > """Raise an exception if the given condition is not true""" > if not cond: > if msg is None: > frame = sys._getframe(1) > line = frame.f_lineno > modname = frame.f_globals.get("__name__", "") > msg = f"Condition not met on line {line:d} in {modname!r}" > raise exc_type(msg) > > Cheers, > Nick. > > P.S. No, I'm not offering to write that PEP myself, I'd just be in > favour of the idea if someone else were to write it :) > I'd prefer (cond, exc_type=RuntimeError, msg=None). For example: ensure(x != 0, ZeroDivisionError) ensure(x > 0, IndexError, "list index out of range") ensure(x != FAIL, msg="failing") I'm not sure how you'd neatly handle no-args constructors and multiple-arguments constructors, tho. From ethan at stoneleaf.us Tue Nov 28 09:32:24 2017 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 28 Nov 2017 06:32:24 -0800 Subject: [Python-ideas] Repurpose `assert' into a general-purpose check In-Reply-To: <20171128113026.GS22248@ando.pearwood.info> References: <23069.9268.799460.752975@turnbull.sk.tsukuba.ac.jp> <20171128113026.GS22248@ando.pearwood.info> Message-ID: <5A1D7378.9030705@stoneleaf.us> On 11/28/2017 03:30 AM, Steven D'Aprano wrote: > On Tue, Nov 28, 2017 at 05:54:12PM +0900, Stephen J. Turnbull wrote: > >> I understand the point made elsewhere about scientists' thinking about >> "assert", but as a scientist myself, I disagree. It is very useful to >> me to distinguish between validating data and validating algorithms. >> "assert" is how we do the latter. > > I'm not sure which point about scientists you are referring to -- I > don't seem to have that email. Is that in this thread? Privately forwarded. (Yes, it is in this thread.) -- ~Ethan~ From vano at mail.mipt.ru Tue Nov 28 11:03:35 2017 From: vano at mail.mipt.ru (Ivan Pozdeev) Date: Tue, 28 Nov 2017 19:03:35 +0300 Subject: [Python-ideas] Repurpose `assert' into a general-purpose check In-Reply-To: References: <20171128054135.GN22248@ando.pearwood.info> Message-ID: <4c619667-9f4f-a360-479d-b3b21e9a6c21@mail.mipt.ru> On 28.11.2017 16:36, Nick Coghlan wrote: > On 28 November 2017 at 15:41, Steven D'Aprano wrote: >> On Tue, Nov 28, 2017 at 05:12:36AM +0300, Ivan Pozdeev via Python-ideas wrote: >>> Unlike C, Python does the aforementioned checks all the time, i.e. it's >>> effectively always in "debug mode". >> Apart from -O which disables assertions. But in any case, the best use >> of assertions is not checking things which the interpreter is going to >> do anyway, but checking things which the interpreter can not and does >> not check automatically: your program logic. There is no way that the >> Python interpreter is going to do this check automatically, unless I >> write the assertion: >> >> assert 0 <= r < abs(y) >> >> That is copied straight out of one of my functions. > I'll make the same observation I usually do each time one of these > threads comes up: > > * I'm opposed to making assert substantially different from the way it works now > * I'm in favour of adding a new "ensure()" builtin that encapsulates > the check-and-raise logic > > The reasons I prefer this approach: > > - assert is a statement *solely* so that the compiler can optimise it > out. If it's not optional, > it doesn't need to be a statement any more Another benefit of a statement vs function is only evaluating the error-related arguments when there's an error > - if the existing assert statements are left alone, there are no > performance or compatibility > concerns for folks that rely on the current behaviour > - if it's a function, it doesn't need to be called "assert", it can use the more > imperative term "ensure" (meaning "ensure this condition is true > before continuing") > - if it's a function, it can easily be emulated on old versions via > compatibility libraries > - "ensure() is required, assert is optional" is a better answer to > complaints about > assertions being optional than suggesting "if cond: raise AssertionError(msg)" > as a reasonable alternative to "assert cond, msg" > - if it's a function, we get access to all the regular function > machinery, so we're > not restricted to positional-only arguments the way the assert statement is > > My initial proposed behaviour for the function: > > def ensure(cond, msg=None, exc_type=RuntimeError): > """Raise an exception if the given condition is not true""" > if not cond: > if msg is None: > frame = sys._getframe(1) > line = frame.f_lineno > modname = frame.f_globals.get("__name__", "") > msg = f"Condition not met on line {line:d} in {modname!r}" > raise exc_type(msg) > > Cheers, > Nick. > > P.S. No, I'm not offering to write that PEP myself, I'd just be in > favour of the idea if someone else were to write it :) > -- Regards, Ivan From mertz at gnosis.cx Tue Nov 28 11:05:25 2017 From: mertz at gnosis.cx (David Mertz) Date: Tue, 28 Nov 2017 08:05:25 -0800 Subject: [Python-ideas] Repurpose `assert' into a general-purpose check In-Reply-To: References: <39071bb0-f93a-af09-9ca5-67b75d05234f@nedbatchelder.com> <21a0d033-553c-b7dc-40b3-f957db7be1e0@mail.mipt.ru> <20171128055956.GO22248@ando.pearwood.info> <5fb6604e-d412-7b26-fb28-564bfaf00b43@mail.mipt.ru> Message-ID: I've used Python for 20 years, and programming in general for about 40 now. I think almost EVERY large code base I've seen used asserts in their intended way. I frequently use them myself in a way differentiated (subtly) from 'if cond: raise SomeException'. I honestly don't know how Ivan has avoided seeing these. Giving up this useful construct would feel about the same as depreciating 'finally'. Yes, Python would be Turing complete without it, and you could work around it being missing... But why?! On Nov 27, 2017 11:12 PM, "Ivan Pozdeev via Python-ideas" < python-ideas at python.org> wrote: On 28.11.2017 8:59, Steven D'Aprano wrote: > On Tue, Nov 28, 2017 at 07:35:45AM +0300, Ivan Pozdeev via Python-ideas > wrote: > > Actually, the way I'm using them, >> >> assert condition, "error message", type >> >> would probably be the most expressive way. >> > I disagree that is expressive -- I call it *misleading*. I see something > which looks like an assertion (that is, a checked comment, a contract, a > check on an internal piece of logic etc) but it is actually being used > as a test. > > > I can do anything in any Turing-complete language without any changes to >> the language. That's no reason to never change anything, is it. >> > "We can change this" is not a reason to change this. There needs to be a > *good* reason to change, and you have given no good reasons for this > change. > > > The rationale basically is: >> * As it was intended, the statement has no practical use -- basically a >> rudiment, due to disappear eventually >> > Nonsense. I make extensive use of assert as a way of checking > assertions, and I will fight tooth and nail against any proposal to > either remove it or to misuse it for public input tests instead of > assertions. > I invite you to show me a single use case for those "assertions" because after ~20 years of experience in coding (that included fairly large projects), I've yet to see one. Any, every check that you make at debug time either * belongs in production as well (all the more because it's harder to diagnose there), or * belongs in a test -- something coded independently from the program (if your code as a whole cannot be trusted, how any specific part of it can?), or * isn't needed at all because a fault will inevitably surface somewhere down the line (as some exception or an incorrect result that a test will catch). Finally, I've got much experience using existing code outside its original use cases, where the original author's assumptions may no longer hold but the specific logic can be gauded to produce the desired result. Coding these assumptions in would undermine that goal. So, I see "debug assertions" as either intentionally compromizing correctness for performance (a direct opposite of Python's design principles), or as an inferiour, faulty, half-measure rudiment from times when CI wasn't a thing (thus not something that should be taught and promoted as a best practice any longer). > * It can instead be reused as syntax sugar to cover a very common use case >> > There is no need for such syntactic sugar. It would be harmful > to use assert for something which is not an assertion. > > > > -- Regards, Ivan _______________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From elazarg at gmail.com Tue Nov 28 11:20:58 2017 From: elazarg at gmail.com (Elazar) Date: Tue, 28 Nov 2017 16:20:58 +0000 Subject: [Python-ideas] Repurpose `assert' into a general-purpose check In-Reply-To: <4c619667-9f4f-a360-479d-b3b21e9a6c21@mail.mipt.ru> References: <20171128054135.GN22248@ando.pearwood.info> <4c619667-9f4f-a360-479d-b3b21e9a6c21@mail.mipt.ru> Message-ID: On Tue, Nov 28, 2017 at 6:08 PM Ivan Pozdeev via Python-ideas < python-ideas at python.org> wrote: > On 28.11.2017 16:36, Nick Coghlan wrote: > > > On 28 November 2017 at 15:41, Steven D'Aprano > wrote: > >> On Tue, Nov 28, 2017 at 05:12:36AM +0300, Ivan Pozdeev via Python-ideas > wrote: > >>> Unlike C, Python does the aforementioned checks all the time, i.e. it's > >>> effectively always in "debug mode". > >> Apart from -O which disables assertions. But in any case, the best use > >> of assertions is not checking things which the interpreter is going to > >> do anyway, but checking things which the interpreter can not and does > >> not check automatically: your program logic. There is no way that the > >> Python interpreter is going to do this check automatically, unless I > >> write the assertion: > >> > >> assert 0 <= r < abs(y) > >> > >> That is copied straight out of one of my functions. > > I'll make the same observation I usually do each time one of these > > threads comes up: > > > > * I'm opposed to making assert substantially different from the way it > works now > > * I'm in favour of adding a new "ensure()" builtin that encapsulates > > the check-and-raise logic > > > > The reasons I prefer this approach: > > > > - assert is a statement *solely* so that the compiler can optimise it > > out. If it's not optional, > > it doesn't need to be a statement any more > Another benefit of a statement vs function is only evaluating the > error-related arguments when there's an error > I'm not sure what the use case is, but it could be implemented easily as ensure(not hasattr(e, "exception")) or raise e.exception ... had "raise" been an expression, an idea repeatedly rejected here. It's still implementable with a "throw()" function. Elazar > > - if the existing assert statements are left alone, there are no > > performance or compatibility > > concerns for folks that rely on the current behaviour > > - if it's a function, it doesn't need to be called "assert", it can use > the more > > imperative term "ensure" (meaning "ensure this condition is true > > before continuing") > > - if it's a function, it can easily be emulated on old versions via > > compatibility libraries > > - "ensure() is required, assert is optional" is a better answer to > > complaints about > > assertions being optional than suggesting "if cond: raise > AssertionError(msg)" > > as a reasonable alternative to "assert cond, msg" > > - if it's a function, we get access to all the regular function > > machinery, so we're > > not restricted to positional-only arguments the way the assert > statement is > > > > My initial proposed behaviour for the function: > > > > def ensure(cond, msg=None, exc_type=RuntimeError): > > """Raise an exception if the given condition is not true""" > > if not cond: > > if msg is None: > > frame = sys._getframe(1) > > line = frame.f_lineno > > modname = frame.f_globals.get("__name__", " module>") > > msg = f"Condition not met on line {line:d} in > {modname!r}" > > raise exc_type(msg) > > > > Cheers, > > Nick. > > > > P.S. No, I'm not offering to write that PEP myself, I'd just be in > > favour of the idea if someone else were to write it :) > > > > -- > Regards, > Ivan > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Tue Nov 28 11:34:02 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 29 Nov 2017 03:34:02 +1100 Subject: [Python-ideas] Repurpose `assert' into a general-purpose check In-Reply-To: <5fb6604e-d412-7b26-fb28-564bfaf00b43@mail.mipt.ru> References: <39071bb0-f93a-af09-9ca5-67b75d05234f@nedbatchelder.com> <21a0d033-553c-b7dc-40b3-f957db7be1e0@mail.mipt.ru> <20171128055956.GO22248@ando.pearwood.info> <5fb6604e-d412-7b26-fb28-564bfaf00b43@mail.mipt.ru> Message-ID: <20171128163402.GU22248@ando.pearwood.info> On Tue, Nov 28, 2017 at 10:11:46AM +0300, Ivan Pozdeev via Python-ideas wrote: > I invite you to show me a single use case for those "assertions" because > after ~20 years of experience in coding (that included fairly large > projects), I've yet to see one. I already mentioned not one but multiple use-cases for assertions: - checked comments (assertions are documentation, not just code) - checks of internal algorithm logic - design-by-contract style pre- and post-conditions - checking program invariants - defensive programming against conditions that "cannot happen". John Regehr wrote an excellent post on this from the perspective of a systems programmer: https://blog.regehr.org/archives/1091 (he even links to a post by our own Ned Batchelder) but many of his use-cases for assert applies just as well to Python as C. Assertions also work great with fuzzers: http://www.squarefree.com/2014/02/03/fuzzers-love-assertions/ I've also written on assertions before: https://import-that.dreamwidth.org/676.html Assertions can also be used as a kind of "Continuous Testing" that operates in debug builds (or in Python in __debug__ mode): every time you run the code, it tests its own internal state by running the asserts. As Regehr puts it: "When we write an assertion, we are teaching a program to diagnose bugs in itself." If the code you have worked with needed none of these things, assertions might not be useful to you. But that doesn't mean there are no use-cases for assertions. Not everyone is lucky to work with code that is so self-documenting that checked comments are redundant, or code so obviously correct that there is no point to checking its internal state. In your opening post, you gave a use-case for assertions: "a check to only do in debug mode, when you can't yet trust your code to manage and pass around internal data correctly." Now you say there are none. Would you like to revise one of those statements? > Any, every check that you make at debug time either > * belongs in production as well (all the more because it's harder to > diagnose there), or I'm not going to say that claim is entirely wrong. For example, Microsoft's research project Midori used two kinds of assertions: Debug.Assert only runs under debugging; Release.Assert always runs and it also promised that all contracts will either be checked at runtime, or the compiler can prove that the contract is always satisfied and so can skip the check. Otherwise they cannot be disabled. http://joeduffyblog.com/2016/02/07/the-error-model/ But Midori was written in a custom systems language based on C#, and the lessons from it probably don't apply directly to a rapid application development language / scripting language like Python. In any case, Midori's approach was rather unusual even compared to other systems languages. More conventionally, Eiffel (for example) allows the developer to enable or disable contract checking. By default, pre-conditions are checked in release builds and post-conditions and invariants are skipped, but each one can be enabled or disabled individually. We have the choice to choose faster code or more extensive error checking, depending on which we value more. I think that's an excellent tradeoff to have. Python currently only has a very coarse switch that can only turn assertions on or off, but even that coarse switch is better than nothing. > * belongs in a test -- something coded independently from the program > (if your code as a whole cannot be trusted, how any specific part of it > can?), or I love tests. I write unit tests and doc tests all the time. (Well, except when I'm being lazy, when I only write doc tests.) But tests cannot replace assertions. There are at least two problems: - external tests don't have access to the function locals; - and even if you can write a test for something, its in the wrong place to be useful as a replacement of an assertion. John Regehr discusses this exact issue (see link above) and writes that "there is a strong synergy between assertions and unit tests". If you do them right, they complement each other. And I would argue that assertions are a kind of test integrated in the code. Neither assertions nor unit tests can find all bugs. The wise programmer uses both. In an earlier post, I gave an actual assertion from one of my functions. Here it is again: assert 0 <= r < abs(y) Its not obvious from that line in isolation, but both r and y are local variables of a function that calculates a mathematical result (hence the short and undescriptive names). How can I possibly write a test to check this? They are *local* to the function, so I have no access to them from outside of the test! (Actually, in this *specific case*, r is part of the return value and could be extracted by a test function. But y isn't.) In general, assert is great for tests that occur inside the body of a function, checking assertions about the function internals. You cannot replace that with an external test. Here's an assertion from another function: c = collections.Counter(symbols) assert c The Counter c is an internal implementation detail of this one function. It would be silly to expose it (how?) so I can write a test to check it. Even if I could write such a test, that would be the wrong place for the check. I want the check there inside the function, not in a completely different .py file. The assertion is as much for me, the reader of the code, as for the interpreter. After reading that assertion, I can read the rest of the function knowing that it is safe to assume that c will always have at least one key. It is a checked comment: `assert c` is better than: # c will not be empty because the assertion is checked at runtime unless I disable assert checking, while comments are "lies in code" that rapidly become obsolete or inaccurate. > * isn't needed at all because a fault will inevitably surface somewhere > down the line (as some exception or an incorrect result that a test will > catch). You don't know that an error will be raised. The program may simply do the wrong thing and silently return garbage that you have no way of knowing is garbage. Nor do you know that a test will catch the problem. Most real world code does not have even close to 100% test coverage -- which is why people are always discovering new bugs. But even if there is an obvious failure later on, or a failing test, it is better to catch errors sooner rather than later. The longer it takes to discover the error, the harder it is to debug. Assertions can reduce the distance between where the bug occurs and where you notice it. > Finally, I've got much experience using existing code outside its > original use cases, where the original author's assumptions may no > longer hold but the specific logic can be gauded to produce the desired > result. Coding these assumptions in would undermine that goal. Of course assertions can be misused. The same applies to code that defeats duck-typing with excessive isinstance() checks, or people who make everything private and all classes final (in languages that support that). John Regehr also discusses some poor ways to misuse assertions. This is not a good argument for changing assert. > So, I see "debug assertions" as either intentionally compromizing > correctness for performance (a direct opposite of Python's design > principles), You have that backwards: assertions compromise performance for correctness. I'm very aware that every time I write an assert, that's a runtime check that compromises performance. But I do so when I believe that the gain in correctness, or the advantage in finding bugs closer to their origin, or even their value as documentation, outweighs the performance cost. > or as an inferiour, faulty, half-measure rudiment from > times when CI wasn't a thing (thus not something that should be taught > and promoted as a best practice any longer). If you think that Continuous Integration is a suitable replacement for assertions, then I think you have misunderstood either CI or assertions or both. That's like saying that now that we have CI, we don't need to check the return code on C functions, or catch exceptions. CI and assertions are complementary, not in opposition. CI is a development practice to ensure that the master is always in a working state. That's great. But how do you know the master is working? You need *tests*, and assertions complement tests. Unit tests are not a replacement for assertions. You wouldn't say "unit tests are obsolete now that we have integration tests", or "we don't need fuzzers, we have regression tests". Why would you say that you don't need assertions just because you are using CI? -- Steve From rob.cliffe at btinternet.com Tue Nov 28 11:25:23 2017 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Tue, 28 Nov 2017 16:25:23 +0000 Subject: [Python-ideas] generator vs iterator etc. (was: How assignment should work with generators?) In-Reply-To: <20171128062238.GQ22248@ando.pearwood.info> References: <20171127234449.GJ22248@ando.pearwood.info> <23068.65037.184603.974838@turnbull.sk.tsukuba.ac.jp> <20171128062238.GQ22248@ando.pearwood.info> Message-ID: <41cfb7ca-dbdf-2a5f-62ed-ed396650913d@btinternet.com> Given that we have this kind of arcane discussion fairly regularly (not just in this thread), and it always makes my head spin, and it seems I'm not the only one who gets confused: How about having a module that provides functions such as ??? isgenerator? isiterator? isiterable? etc. or alternatively one function that would return a tuple/list of categories that an object fell into e.g. ('iterator', 'iterable') ??? ??? ??? ??? ??? or a dictionary e.g. { 'iterator' : True, 'iterable' : True, 'generator' : False }. (Bikeshed as appropriate, but providing a dict seems to make it easier to add more things in future without breaking backward compatibility.) Then those of us who are prepared to take care to be precise in our language but could do with some help could use it to clarify our thoughts.? And there should be less noise in the newsgroups from pointless arguments about precisely what is what.? And I suspect it would even have uses in "real" code. Rob Cliffe On 28/11/2017 06:22, Steven D'Aprano wrote: > On Tue, Nov 28, 2017 at 03:11:25PM +0900, Stephen J. Turnbull wrote: >> Steven D'Aprano writes: >> >> > The subset of iterators which are created as generators are *also* >> > called generators, >> >> As long as we're being precise, I don't think that is precisely correct: >> >> >>> (x for x in range(1)) >> at 0x10dee5e08> >> >>> iter(range(1)) >> >> >>> iter((1,)) >> >> >> The two iterators have the same duck-type, the generator is different. > How is the generator different? It quacks like a range_iterator and > tuple_iterator, it swims like them, it flies like them. Is there some > iterator method or protocol that generators don't support? > > >> A generator (object) is, of course, an interable. > And also an iterator: > > py> collections.abc > py> isinstance((x+1 for x in range(5)), collections.abc.Iterator) > True > > >> > Most of the time the distinction doesn't actually matter, since you >> > cannot (easily?) create a generator without first creating a >> > generator function. >> >> At least you can create a generator (object) with the generator >> function created and called implicitly by using a generator >> expression. > Ah yes, I forget about generator expressions, thanks. > > > From ethan at stoneleaf.us Tue Nov 28 12:23:25 2017 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 28 Nov 2017 09:23:25 -0800 Subject: [Python-ideas] Repurpose `assert' into a general-purpose check In-Reply-To: <4c619667-9f4f-a360-479d-b3b21e9a6c21@mail.mipt.ru> References: <20171128054135.GN22248@ando.pearwood.info> <4c619667-9f4f-a360-479d-b3b21e9a6c21@mail.mipt.ru> Message-ID: <5A1D9B8D.2000904@stoneleaf.us> On 11/28/2017 08:03 AM, Ivan Pozdeev via Python-ideas wrote: > On 28.11.2017 16:36, Nick Coghlan wrote: >> it doesn't need to be a statement any more > > Another benefit of a statement vs function is only evaluating the error-related arguments when there's an error The bulk of any processing in assert (or ensure()) should be the actual check -- if that fails, only state information should be included in the exception as anything more complicated runs the risk of also being wrong as the code is now in a failed state. In other words, the "error-related arguments" will often be extremely cheap compared to the test itself. -- ~Ethan~ From vano at mail.mipt.ru Tue Nov 28 13:27:19 2017 From: vano at mail.mipt.ru (Ivan Pozdeev) Date: Tue, 28 Nov 2017 21:27:19 +0300 Subject: [Python-ideas] Repurpose `assert' into a general-purpose check In-Reply-To: <5A1D9B8D.2000904@stoneleaf.us> References: <20171128054135.GN22248@ando.pearwood.info> <4c619667-9f4f-a360-479d-b3b21e9a6c21@mail.mipt.ru> <5A1D9B8D.2000904@stoneleaf.us> Message-ID: On 28.11.2017 20:23, Ethan Furman wrote > On 11/28/2017 08:03 AM, Ivan Pozdeev via Python-ideas wrote: >> On 28.11.2017 16:36, Nick Coghlan wrote: > >>> ?? it doesn't need to be a statement any more > > >> Another benefit of a statement vs function is only evaluating the >> error-related arguments when there's an error > > The bulk of any processing in assert (or ensure()) should be the > actual check -- if that fails, only state information should be > included in the exception as anything more complicated runs the risk > of also being wrong as the code is now in a failed state.? In other > words, the "error-related arguments" will often be extremely cheap > compared to the test itself. > My experience is the contrary. The check is usually trivial -- a type check or comparison. While a useful error message contains problem details, so it incorporates string formatting from a variable or two or expressions thereof, like "expected , got ". > -- > ~Ethan~ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- Regards, Ivan From fakedme+py at gmail.com Tue Nov 28 13:58:42 2017 From: fakedme+py at gmail.com (Soni L.) Date: Tue, 28 Nov 2017 16:58:42 -0200 Subject: [Python-ideas] Repurpose `assert' into a general-purpose check In-Reply-To: References: <20171128054135.GN22248@ando.pearwood.info> <4c619667-9f4f-a360-479d-b3b21e9a6c21@mail.mipt.ru> <5A1D9B8D.2000904@stoneleaf.us> Message-ID: <37525b3c-385c-c0cd-ef42-873370b82ddb@gmail.com> On 2017-11-28 04:27 PM, Ivan Pozdeev via Python-ideas wrote: > On 28.11.2017 20:23, Ethan Furman wrote > >> On 11/28/2017 08:03 AM, Ivan Pozdeev via Python-ideas wrote: >>> On 28.11.2017 16:36, Nick Coghlan wrote: >> >>>> ?? it doesn't need to be a statement any more >> > >>> Another benefit of a statement vs function is only evaluating the >>> error-related arguments when there's an error >> >> The bulk of any processing in assert (or ensure()) should be the >> actual check -- if that fails, only state information should be >> included in the exception as anything more complicated runs the risk >> of also being wrong as the code is now in a failed state. In other >> words, the "error-related arguments" will often be extremely cheap >> compared to the test itself. >> > My experience is the contrary. The check is usually trivial -- a type > check or comparison. While a useful error message contains problem > details, so it incorporates string formatting from a variable or two > or expressions thereof, like "expected , got ". ensure(check, lambda: raise TypeError("expected , got "))? >> -- >> ~Ethan~ >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > From fakedme+py at gmail.com Tue Nov 28 14:03:08 2017 From: fakedme+py at gmail.com (Soni L.) Date: Tue, 28 Nov 2017 17:03:08 -0200 Subject: [Python-ideas] raise in lambda Message-ID: Would be useful to pass exception-raising lambdas around. ensure(cond, lambda: raise TypeError()) I guess one could instead use (and raise in ensure()) ensure(cond, lambda: TypeError()) But I think exception-raising lambdas would be nicer. From rosuav at gmail.com Tue Nov 28 14:09:33 2017 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 29 Nov 2017 06:09:33 +1100 Subject: [Python-ideas] raise in lambda In-Reply-To: References: Message-ID: On Wed, Nov 29, 2017 at 6:03 AM, Soni L. wrote: > Would be useful to pass exception-raising lambdas around. > > ensure(cond, lambda: raise TypeError()) > > I guess one could instead use (and raise in ensure()) > > ensure(cond, lambda: TypeError()) > > But I think exception-raising lambdas would be nicer. You can easily create a throw() function: def throw(exc): raise exc Then you can use that in your lambda function: ensure(cond, lambda: throw(TypeError("..."))) However, before you jump onto the "let's construct this lazily" idea, benchmark the cost of creating a lambda function. It's more expensive than a lot of people realize, and if all you save is a bit of string formatting, you may as well just eagerly format that string. ChrisA From AlonSnir at hotmail.com Tue Nov 28 13:46:18 2017 From: AlonSnir at hotmail.com (Alon Snir) Date: Tue, 28 Nov 2017 18:46:18 +0000 Subject: [Python-ideas] How assignment should work with generators? Message-ID: I would like to mention that the issue of assignment to a target list, is also relevant to the case of elementwise assignment to a mutable sequence (e.g. lists and arrays). Here as well, the rhs of the assignment statement states the number of elements to be assigned. Consequently, the following example will get into infinite loop: >>> from itertools import count >>> A = [] >>> A[:] = count() Writing "A[:2] = count()" will cause the same result. Here as well, it is currently safer to use islice if the rhs is a generator or an iterator. e.g.: >>> it = count() >>> A[:] = islice(it,2) In my opinion, it is be better to devise a solution that could be applied in both cases. Maybe a new kind of assignment operator that will be dedicated to this kind of assignment. i.e. elementwise assignment with restriction on the number of elements to be assigned, based on the length of the lhs object (or the number of targets in the target list). From rosuav at gmail.com Tue Nov 28 14:29:37 2017 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 29 Nov 2017 06:29:37 +1100 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: References: Message-ID: On Wed, Nov 29, 2017 at 5:46 AM, Alon Snir wrote: > I would like to mention that the issue of assignment to a target list, is also relevant to the case of elementwise assignment to a mutable sequence (e.g. lists and arrays). Here as well, the rhs of the assignment statement states the number of elements to be assigned. Consequently, the following example will get into infinite loop: > >>>> from itertools import count >>>> A = [] >>>> A[:] = count() > > Writing "A[:2] = count()" will cause the same result. > Here as well, it is currently safer to use islice if the rhs is a generator or an iterator. e.g.: > >>>> it = count() >>>> A[:] = islice(it,2) > > In my opinion, it is be better to devise a solution that could be applied in both cases. Maybe a new kind of assignment operator that will be dedicated to this kind of assignment. i.e. elementwise assignment with restriction on the number of elements to be assigned, based on the length of the lhs object (or the number of targets in the target list). > Hmm. The trouble is that slice assignment doesn't have a fixed number of targets. If you say "x, y = spam", there's a clear indication that 'spam' needs to provide exactly two values; but "A[:] = spam" could have any number of values, and it'll expand or shrink the list accordingly. ChrisA From python-ideas at mgmiller.net Tue Nov 28 14:25:35 2017 From: python-ideas at mgmiller.net (Mike Miller) Date: Tue, 28 Nov 2017 11:25:35 -0800 Subject: [Python-ideas] Logging Levels Message-ID: Hi, I use the logging module extensively for even the simplest scripts, one of the reasons there's been less difficulty moving to Python 3 at work. One of the "nano features" I've often added to its config is the addition of a custom log level. Never mentioned it before because of its triviality and minor use cases, but just realized that I've been doing it five years now and happy with it, so why not? NOTE (~35) It is called the "note" level and used when one needs to express something important, yet positive, and have it be output by default. The standard levels don't support this currently, you often have to potentially scare the end user with a warning or higher to have a message emitted. Typically I use it to return important information that was asked for specifically and retrieved successfully, e.g.: log.note('Your token is: %s', token) log.note(f'? {item.id} {item.name}') There are other examples. Sphinx, has the concept of note admonitions for docs in addition to warning and danger. Bootstrap has note banners for web/apps. There is something important to express or highlight, but nothing to worry about. FATAL (alias of CRITICAL) Can't find it now, but believe the docs in the past stated that CRITICAL was meant for *unrecoverable* errors. I've never had a project where I didn't shut down immediately after such an occurrence. Therefore I find "FATAL" a more accurate description of what happened. Log4j and other industry loggers use this level name as well. There is also an aesthetic reason to prefer "fatal". It is shorter and so aligns better with other level names for readability, e.g.: console_format = ' %(levelname)-7.7s %(message)s' Tried but never found a good abbreviation for critical, unfortunately. The other option is to add length to align the field. Most messages use the shorter level names (debug, info) so extra length results in wasted space that is very rarely needed. Hopefully someone else finds these useful. Neither depends on the other. -Mike From kirillbalunov at gmail.com Tue Nov 28 14:44:51 2017 From: kirillbalunov at gmail.com (Kirill Balunov) Date: Tue, 28 Nov 2017 22:44:51 +0300 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: References: <5A1C7FDD.6080107@canterbury.ac.nz> <5A1C8E9C.7090706@canterbury.ac.nz> <20171128003155.GL22248@ando.pearwood.info> <5A1CF103.7020804@canterbury.ac.nz> <20171128063151.GR22248@ando.pearwood.info> Message-ID: 2017-11-28 10:06 GMT+03:00 C Anthony Risinger : > > If not already considered, what if the RHS had to be explicitly unpacked? > > Something like: > > a, b, c = *iterator > > Which would essentially be: > > a, b, c = (*iterator,) > > This enables lazy assignment by default but `*` can force complete > expansion (and exact matching) of the RHS. > While I find your suggestions very close to my vision and the initial proposal, which I still like. I saw enough of the discussion to realize that by now it is already impossible. > It's a breaking change, but it does have a straightforward fix (simply > wrap and unpack any relevant RHS > Although I have never used Python 2, the idea to distinguish fixed-sized and something lazy, even for Python 4, reminds me of the transition from str-unicode to the present state of affairs, but with much higher impact.To be honest, I do not like some aspects of how Python 2 issue has been resolved (especially bytes part) but it is another topic. With kind regards, -gdg -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Nov 28 14:45:00 2017 From: guido at python.org (Guido van Rossum) Date: Tue, 28 Nov 2017 11:45:00 -0800 Subject: [Python-ideas] Logging Levels In-Reply-To: References: Message-ID: These look like good improvements. I think you should make an issue on bugs.python.org describing your proposal and if you can submit a PR that implements it. On Tue, Nov 28, 2017 at 11:25 AM, Mike Miller wrote: > Hi, > > I use the logging module extensively for even the simplest scripts, one of > the reasons there's been less difficulty moving to Python 3 at work. One > of the "nano features" I've often added to its config is the addition of a > custom log level. Never mentioned it before because of its triviality and > minor use cases, but just realized that I've been doing it five years now > and happy with it, so why not? > > NOTE (~35) > > It is called the "note" level and used when one needs to express something > important, yet positive, and have it be output by default. The standard > levels don't support this currently, you often have to potentially scare > the end user with a warning or higher to have a message emitted. > > Typically I use it to return important information that was asked for > specifically and retrieved successfully, e.g.: > > log.note('Your token is: %s', token) > log.note(f'? {item.id} {item.name}') > > There are other examples. Sphinx, has the concept of note admonitions for > docs in addition to warning and danger. Bootstrap has note banners for > web/apps. There is something important to express or highlight, but nothing > to worry about. > > > FATAL (alias of CRITICAL) > > Can't find it now, but believe the docs in the past stated that CRITICAL > was meant for *unrecoverable* errors. I've never had a project where I > didn't shut down immediately after such an occurrence. Therefore I find > "FATAL" a more accurate description of what happened. Log4j and other > industry loggers use this level name as well. > > There is also an aesthetic reason to prefer "fatal". It is shorter and so > aligns better with other level names for readability, e.g.: > > console_format = ' %(levelname)-7.7s %(message)s' > > Tried but never found a good abbreviation for critical, unfortunately. > The other option is to add length to align the field. Most messages use > the shorter level names (debug, info) so extra length results in wasted > space that is very rarely needed. > > Hopefully someone else finds these useful. Neither depends on the other. > > -Mike > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rhodri at kynesim.co.uk Tue Nov 28 14:31:30 2017 From: rhodri at kynesim.co.uk (Rhodri James) Date: Tue, 28 Nov 2017 19:31:30 +0000 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: References: Message-ID: On 28/11/17 18:46, Alon Snir wrote: > I would like to mention that the issue of assignment to a target list, is also relevant to the case of elementwise assignment to a mutable sequence (e.g. lists and arrays). Here as well, the rhs of the assignment statement states the number of elements to be assigned. Consequently, the following example will get into infinite loop: > >>>> from itertools import count >>>> A = [] >>>> A[:] = count() Actually the "infinite loop" will terminate when you run out of memory, but that's a mere detail. > Writing "A[:2] = count()" will cause the same result. > Here as well, it is currently safer to use islice if the rhs is a generator or an iterator. e.g.: > >>>> it = count() >>>> A[:] = islice(it,2) Not so much "safer" as "correct", I'm afraid. In the first case you asked for an infinite sequence, in the second case you asked for a finite one. > In my opinion, it is be better to devise a solution that could be applied in both cases. Maybe a new kind of assignment operator that will be dedicated to this kind of assignment. i.e. elementwise assignment with restriction on the number of elements to be assigned, based on the length of the lhs object (or the number of targets in the target list). Flatly, no. It is better not to ask for things you don't want in the first place, in this case the infinite sequence. Still, don't let me discourage you from working on this. If you can define how such an assignment would work, or even the length of A[:] as an assignment target, I'm not going to dismiss it out of hand. -- Rhodri James *-* Kynesim Ltd From guido at python.org Tue Nov 28 14:50:53 2017 From: guido at python.org (Guido van Rossum) Date: Tue, 28 Nov 2017 11:50:53 -0800 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: References: <5A1C7FDD.6080107@canterbury.ac.nz> <5A1C8E9C.7090706@canterbury.ac.nz> <20171128003155.GL22248@ando.pearwood.info> <5A1CF103.7020804@canterbury.ac.nz> <20171128063151.GR22248@ando.pearwood.info> Message-ID: On Tue, Nov 28, 2017 at 11:44 AM, Kirill Balunov wrote: > Although I have never used Python 2, the idea to distinguish fixed-sized > and something lazy, even for Python 4, reminds me of the transition from > str-unicode to the present state of affairs, but with much higher impact.To > be honest, I do not like some aspects of how Python 2 issue has been > resolved (especially bytes part) but it is another topic. > Since Python 4 came up, I'd like to make something clear. Python 4 is *not* going to be a release where we break compatibility with a whole bunch of things at once. Basically if you think you'll need to wait for Python 4 to get your favorite change to the language, you can forget it. You need to come up with a plan to introduce the change without breaking existing code or at least a clear deprecation schedule. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From mariocj89 at gmail.com Tue Nov 28 14:52:49 2017 From: mariocj89 at gmail.com (Mario Corchero) Date: Tue, 28 Nov 2017 19:52:49 +0000 Subject: [Python-ideas] Logging Levels In-Reply-To: References: Message-ID: Not sure if that is what you meant, but here is already "FATAL" which is indeed an alias of CRITICAL: >>> logging.FATAL 50 >>> logging.fatal On 28 November 2017 at 19:45, Guido van Rossum wrote: > These look like good improvements. I think you should make an issue on > bugs.python.org describing your proposal and if you can submit a PR that > implements it. > > On Tue, Nov 28, 2017 at 11:25 AM, Mike Miller > wrote: > >> Hi, >> >> I use the logging module extensively for even the simplest scripts, one >> of the reasons there's been less difficulty moving to Python 3 at work. >> One of the "nano features" I've often added to its config is the addition >> of a custom log level. Never mentioned it before because of its triviality >> and minor use cases, but just realized that I've been doing it five years >> now and happy with it, so why not? >> >> NOTE (~35) >> >> It is called the "note" level and used when one needs to express >> something important, yet positive, and have it be output by default. The >> standard levels don't support this currently, you often have to potentially >> scare the end user with a warning or higher to have a message emitted. >> >> Typically I use it to return important information that was asked for >> specifically and retrieved successfully, e.g.: >> >> log.note('Your token is: %s', token) >> log.note(f'? {item.id} {item.name}') >> >> There are other examples. Sphinx, has the concept of note admonitions >> for docs in addition to warning and danger. Bootstrap has note banners for >> web/apps. There is something important to express or highlight, but nothing >> to worry about. >> >> >> FATAL (alias of CRITICAL) >> >> Can't find it now, but believe the docs in the past stated that CRITICAL >> was meant for *unrecoverable* errors. I've never had a project where I >> didn't shut down immediately after such an occurrence. Therefore I find >> "FATAL" a more accurate description of what happened. Log4j and other >> industry loggers use this level name as well. >> >> There is also an aesthetic reason to prefer "fatal". It is shorter and >> so aligns better with other level names for readability, e.g.: >> >> console_format = ' %(levelname)-7.7s %(message)s' >> >> Tried but never found a good abbreviation for critical, unfortunately. >> The other option is to add length to align the field. Most messages use >> the shorter level names (debug, info) so extra length results in wasted >> space that is very rarely needed. >> >> Hopefully someone else finds these useful. Neither depends on the other. >> >> -Mike >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > > > -- > --Guido van Rossum (python.org/~guido) > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From python-ideas at mgmiller.net Tue Nov 28 14:56:39 2017 From: python-ideas at mgmiller.net (Mike Miller) Date: Tue, 28 Nov 2017 11:56:39 -0800 Subject: [Python-ideas] Logging Levels In-Reply-To: References: Message-ID: Yes, I remember now. However, it still outputs with the level name of critical: >>> logging.fatal('hello') CRITICAL:root:hello On 2017-11-28 11:52, Mario Corchero wrote: > Not sure if that is what you meant, but here is already "FATAL" which is indeed > an alias of CRITICAL: > > >>> logging.FATAL > 50 > >>> logging.fatal > > From guido at python.org Tue Nov 28 14:57:52 2017 From: guido at python.org (Guido van Rossum) Date: Tue, 28 Nov 2017 11:57:52 -0800 Subject: [Python-ideas] Logging Levels In-Reply-To: References: Message-ID: On Tue, Nov 28, 2017 at 11:52 AM, Mario Corchero wrote: > Not sure if that is what you meant, but here is already "FATAL" which is > indeed an alias of CRITICAL: > > >>> logging.FATAL > 50 > >>> logging.fatal > > Maybe it should just go the other way around, making critical an alias for fatal, and switching them around in the docs? No code will be invalidated but it will encourage people to use the simpler, more direct term. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Nov 28 14:59:28 2017 From: guido at python.org (Guido van Rossum) Date: Tue, 28 Nov 2017 11:59:28 -0800 Subject: [Python-ideas] Logging Levels In-Reply-To: References: Message-ID: On Tue, Nov 28, 2017 at 11:56 AM, Mike Miller wrote: > Yes, I remember now. However, it still outputs with the level name of > critical: > > >>> logging.fatal('hello') > > CRITICAL:root:hello > Ah, that makes it slightly more complicated -- changing that to print FATAL is surely going to break some people's log parsers. :-( -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From mariocj89 at gmail.com Tue Nov 28 15:02:30 2017 From: mariocj89 at gmail.com (Mario Corchero) Date: Tue, 28 Nov 2017 20:02:30 +0000 Subject: [Python-ideas] Logging Levels In-Reply-To: References: Message-ID: Yep, I personally prefer fatal so I'd be totally in for it. But as you said I can see an issue on the tools that work around the logs produced (typical alerting/monitoring around an app) and all the logs parsing stacks. On 28 November 2017 at 19:59, Guido van Rossum wrote: > On Tue, Nov 28, 2017 at 11:56 AM, Mike Miller > wrote: > >> Yes, I remember now. However, it still outputs with the level name of >> critical: >> >> >>> logging.fatal('hello') >> >> CRITICAL:root:hello >> > > Ah, that makes it slightly more complicated -- changing that to print > FATAL is surely going to break some people's log parsers. :-( > > -- > --Guido van Rossum (python.org/~guido) > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From python-ideas at mgmiller.net Tue Nov 28 15:08:20 2017 From: python-ideas at mgmiller.net (Mike Miller) Date: Tue, 28 Nov 2017 12:08:20 -0800 Subject: [Python-ideas] Logging Levels In-Reply-To: References: Message-ID: <00a327c3-13c8-c42c-be74-645d8896a5f3@mgmiller.net> Thanks, will do, after some time for folks to comment further. One question was how will the level integer id affect custom levels defined by applications? If a clash could cause an issue, perhaps adding it above the last one at 100(?) makes more sense. -Mike On 2017-11-28 11:45, Guido van Rossum wrote: > These look like good improvements. I think you should make an issue on > bugs.python.org describing your proposal and if you can > submit a PR that implements it. > From kirillbalunov at gmail.com Tue Nov 28 15:08:40 2017 From: kirillbalunov at gmail.com (Kirill Balunov) Date: Tue, 28 Nov 2017 23:08:40 +0300 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: References: <5A1C7FDD.6080107@canterbury.ac.nz> <5A1C8E9C.7090706@canterbury.ac.nz> <20171128003155.GL22248@ando.pearwood.info> <5A1CF103.7020804@canterbury.ac.nz> <20171128063151.GR22248@ando.pearwood.info> Message-ID: 2017-11-28 22:50 GMT+03:00 Guido van Rossum : > On Tue, Nov 28, 2017 at 11:44 AM, Kirill Balunov > wrote: > >> Although I have never used Python 2, the idea to distinguish fixed-sized >> and something lazy, even for Python 4, reminds me of the transition from >> str-unicode to the present state of affairs, but with much higher impact.To >> be honest, I do not like some aspects of how Python 2 issue has been >> resolved (especially bytes part) but it is another topic. >> > > Since Python 4 came up, I'd like to make something clear. Python 4 is > *not* going to be a release where we break compatibility with a whole bunch > of things at once. Basically if you think you'll need to wait for Python 4 > to get your favorite change to the language, you can forget it. You need to > come up with a plan to introduce the change without breaking existing code > or at least a clear deprecation schedule. > > -- > --Guido van Rossum (python.org/~guido) > Oh no, I was misunderstood. I think that we have already come to some consensus which syntax can be discussed. In any case, I did not want to produce unnecessary noise, I apologize. With kind regards, -gdg -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Tue Nov 28 15:17:49 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 28 Nov 2017 22:17:49 +0200 Subject: [Python-ideas] Logging Levels In-Reply-To: References: Message-ID: 28.11.17 21:45, Guido van Rossum ????: > These look like good improvements. I think you should make an issue on > bugs.python.org describing your proposal and if > you can submit a PR that implements it. See https://bugs.python.org/issue31732 It was discussed and rejected. Citing Raymond: "Overall, this seems rehash and second guess the discussions and decisions made 15 years ago when PEP 282 was accepted." The set of logging levels is not closed. The user can extend it to cover more specialized uses by logging.addLevelName(). There are disadvantages of having too much standard names for logging levels (as we can see in Java). From guido at python.org Tue Nov 28 15:31:31 2017 From: guido at python.org (Guido van Rossum) Date: Tue, 28 Nov 2017 12:31:31 -0800 Subject: [Python-ideas] Logging Levels In-Reply-To: References: Message-ID: On Tue, Nov 28, 2017 at 12:17 PM, Serhiy Storchaka wrote: > 28.11.17 21:45, Guido van Rossum ????: > >> These look like good improvements. I think you should make an issue on >> bugs.python.org describing your proposal and if >> you can submit a PR that implements it. >> > > See https://bugs.python.org/issue31732 > > It was discussed and rejected. Citing Raymond: "Overall, this seems rehash > and second guess the discussions and decisions made 15 years ago when PEP > 282 was accepted." > > The set of logging levels is not closed. The user can extend it to cover > more specialized uses by logging.addLevelName(). There are disadvantages of > having too much standard names for logging levels (as we can see in Java). > OK, that's a reasonable discussion, let's not do this. Also I note that in my own usage, "note" comes below "warning", not above it. E.g. mypy has three levels for its end-user facing messages: error, warning, and note. (Note that mypy does not use the logging module -- but it would still feel odd to me to see "note" between warning and error rather than below warning.) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Tue Nov 28 15:42:20 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 28 Nov 2017 22:42:20 +0200 Subject: [Python-ideas] Repurpose `assert' into a general-purpose check In-Reply-To: References: <20171128054135.GN22248@ando.pearwood.info> Message-ID: 28.11.17 15:36, Nick Coghlan ????: > * I'm opposed to making assert substantially different from the way it works now If sometimes we will change assert I would prefer to make it more similar to assert in pytest. Output values of the final and intermediate expressions. The hardest problem -- the repr can be very long and expensive, we need an alternate protocol for producing a shortened representation. > * I'm in favour of adding a new "ensure()" builtin that encapsulates > the check-and-raise logic Then may be add a new "die()" buildin? ;-) 0 <= r < abs(y) or die() From python-ideas at mgmiller.net Tue Nov 28 15:46:50 2017 From: python-ideas at mgmiller.net (Mike Miller) Date: Tue, 28 Nov 2017 12:46:50 -0800 Subject: [Python-ideas] Logging Levels In-Reply-To: References: Message-ID: <35577b73-0943-73f0-931e-a27109646d87@mgmiller.net> Hi, the reason I use note is that I want it to be output by default. So it must be above warning, or perhaps the default level changed. (i.e. There is no need for note if it has the same effect as info. We already have info.) -Mike On 2017-11-28 12:31, Guido van Rossum wrote: > OK, that's a reasonable discussion, let's not do this. Also I note that in my > own usage, "note" comes below "warning", not above it. E.g. mypy has three > levels for its end-user facing messages: error, warning, and note. (Note that > mypy does not use the logging module -- but it would still feel odd to me to see > "note" between warning and error rather than below warning.) > From python-ideas at mgmiller.net Tue Nov 28 15:51:09 2017 From: python-ideas at mgmiller.net (Mike Miller) Date: Tue, 28 Nov 2017 12:51:09 -0800 Subject: [Python-ideas] Logging Levels In-Reply-To: References: Message-ID: <6eb2b097-2597-94e7-ed7e-5fb861f4d455@mgmiller.net> I think the resistance to trace is that it splits an existing level into two, supporting "level inflation." The current level name of critical is similar, describing not just an error, but a really big error! Note (as described here), handles an additional use case. I don't consider it level inflation. -Mike On 2017-11-28 12:17, Serhiy Storchaka wrote: > 28.11.17 21:45, Guido van Rossum ????: >> These look like good improvements. I think you should make an issue on >> bugs.python.org describing your proposal and if you >> can submit a PR that implements it. > > See https://bugs.python.org/issue31732 > > It was discussed and rejected. Citing Raymond: "Overall, this seems rehash and > second guess the discussions and decisions made 15 years ago when PEP 282 was > accepted." > > The set of logging levels is not closed. The user can extend it to cover more > specialized uses by logging.addLevelName(). There are disadvantages of having > too much standard names for logging levels (as we can see in Java). > From steve at pearwood.info Tue Nov 28 19:56:31 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 29 Nov 2017 11:56:31 +1100 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: References: Message-ID: <20171129005631.GW22248@ando.pearwood.info> On Tue, Nov 28, 2017 at 06:46:18PM +0000, Alon Snir wrote: > I would like to mention that the issue of assignment to a target list, > is also relevant to the case of elementwise assignment to a mutable > sequence (e.g. lists and arrays). Here as well, the rhs of the > assignment statement states the number of elements to be assigned. > Consequently, the following example will get into infinite loop: > > >>> from itertools import count > >>> A = [] > >>> A[:] = count() "This damn computer always does what I tell it to do, instead of what I wanted!" What exactly did you expect to happen? To me, this feels like a complaint that writing `while True: pass` enters an infinite loop. > Writing "A[:2] = count()" will cause the same result. > Here as well, it is currently safer to use islice if the rhs is a > generator or an iterator. e.g.: > > >>> it = count() > >>> A[:] = islice(it,2) This isn't merely a matter of safety. (Python is pretty safe -- I can interrupt most infinite loops with Ctrl-C.) It is a matter of actually telling the interpreter what you want it to do. How do you expect the interpreter to predict that you wanted two items, rather than three, or thirty-three? This has little to do with iterators or generators. count() is a rare and special case because it is by design an infinite iterator. Slice assignment is perfectly "safe" with finite iterators: py> a = [] py> a[:] = iter([1, 2, 3, 4, 5]) py> a [1, 2, 3, 4, 5] Slice assignment allows the number of targets and number of values to vary independently. I can do this: a[1:4] = range(10) and Python will happily replace the three slots on the left with the ten items on the right. Or vice versa: a[1:11] = range(4) That is by design, not a bug, and the consequence is that you have to be explicit (on both the left and the right) about how many slots you want to replace and how many items you want to replace them with. We cannot infer one from the other, since they can vary independently. > In my opinion, it is be better to devise a solution that could be > applied in both cases. We already have that solution. py> from itertools import count, islice py> a = [] py> it = count() py> a[:] = islice(it, 5) py> a[:2] = islice(it, 2) py> a[:0] = islice(it, 3) py> a [7, 8, 9, 5, 6, 2, 3, 4] > Maybe a new kind of assignment operator that > will be dedicated to this kind of assignment. i.e. elementwise > assignment with restriction on the number of elements to be assigned, > based on the length of the lhs object (or the number of targets in the > target list). I doubt that would work in your original case: a = [] a[:] = count() Since the list is empty, it would assign zero items from the right, which would make it a no-op. Surely that's not what you want? -- Steve From steve at pearwood.info Tue Nov 28 20:34:03 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 29 Nov 2017 12:34:03 +1100 Subject: [Python-ideas] Proposal: allow length_hint to specify infinite iterators Message-ID: <20171129013403.GX22248@ando.pearwood.info> PEP 424 allows iterators to optionally offer a hint as to how long they will be: https://www.python.org/dev/peps/pep-0424/ Unfortunately, there's no good way for an iterator to report that it is infinitely long. Consequently, even those which are known to be infinite report finite lengths: py> from itertools import count py> from operator import length_hint py> infinite = count() py> length_hint(infinite) 0 This wastes the opportunity to fail fast on operations which cannot possibly succeed, e.g. list(count()) must eventually fail with MemoryError. Or worse: if the OS starts thrashing trying to meet the memory requests, you can lock up the computer. I propose that we: (1) extend the __length_hint__ protocol to allow iterators to report that they are infinite; (2) and recommend that consumers of iterators (such as list) that require finite input should fail fast in the event of an infinite iterator. Four possible ways that __length_hint__ and operator.length_hint might signal an infinite iterator: (a) return a negative value such as -1 (this is currently an error); (b) return some special sentinel value; (c) take the convention that returning sys.maxint means infinity; (d) raise an exception. The advantage of (d) is that consumers of check __length_hint__ don't need to do anything special to fail fast on infinite iterators: py> class Thing: ... def __length_hint__(self): ... raise ValueError('infinite') ... def __iter__(self): ... return count() ... py> x = Thing() py> list(x) Traceback (most recent call last): File "", line 1, in File "", line 3, in __length_hint__ ValueError: infinite but if they can cope with such, they can explicitly catch the exception. Thoughts? -- Steve From steve at pearwood.info Tue Nov 28 20:48:19 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 29 Nov 2017 12:48:19 +1100 Subject: [Python-ideas] generator vs iterator etc. (was: How assignment should work with generators?) In-Reply-To: <41cfb7ca-dbdf-2a5f-62ed-ed396650913d@btinternet.com> References: <20171127234449.GJ22248@ando.pearwood.info> <23068.65037.184603.974838@turnbull.sk.tsukuba.ac.jp> <20171128062238.GQ22248@ando.pearwood.info> <41cfb7ca-dbdf-2a5f-62ed-ed396650913d@btinternet.com> Message-ID: <20171129014819.GY22248@ando.pearwood.info> On Tue, Nov 28, 2017 at 04:25:23PM +0000, Rob Cliffe wrote: > Given that we have this kind of arcane discussion fairly regularly (not > just in this thread), and it always makes my head spin, and it seems I'm > not the only one who gets confused: > > How about having a module that provides functions such as > > ??? isgenerator? isiterator? isiterable? etc. There is no single module that does this, but the inspect module comes close: inspect.isgenerator inspect.isgeneratorfunction will tell you the difference between these two: def gen_function(): yield 1 generator = gen_function() The collections.abc module has ABCs that you can use with isinstance: collections.abc.Iterable collections.abc.Iterator collections.abc.Sequence For example, we know that range is not a generator but is a sequence: py> inspect.isgenerator(range(10)) False py> isinstance(range(10), collections.abc.Sequence) True -- Steve From ncoghlan at gmail.com Tue Nov 28 23:39:53 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 29 Nov 2017 14:39:53 +1000 Subject: [Python-ideas] Repurpose `assert' into a general-purpose check In-Reply-To: <4c619667-9f4f-a360-479d-b3b21e9a6c21@mail.mipt.ru> References: <20171128054135.GN22248@ando.pearwood.info> <4c619667-9f4f-a360-479d-b3b21e9a6c21@mail.mipt.ru> Message-ID: On 29 November 2017 at 02:03, Ivan Pozdeev via Python-ideas wrote: > On 28.11.2017 16:36, Nick Coghlan wrote: >> I'll make the same observation I usually do each time one of these >> threads comes up: >> >> * I'm opposed to making assert substantially different from the way it >> works now >> * I'm in favour of adding a new "ensure()" builtin that encapsulates >> the check-and-raise logic >> >> The reasons I prefer this approach: >> >> - assert is a statement *solely* so that the compiler can optimise it >> out. If it's not optional, >> it doesn't need to be a statement any more > > Another benefit of a statement vs function is only evaluating the > error-related arguments when there's an error If you're worried about that level of micro-optimisation, it's straightforward enough to write your own wrapper function that accepts the components needed to build a suitably formatted message. There's no need to make the builtin more complicated to cope with it (that smooth transition from the builtin behaviour to customised behaviour is one of the other advantages of using a plain function). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Tue Nov 28 23:45:40 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 29 Nov 2017 14:45:40 +1000 Subject: [Python-ideas] Logging Levels In-Reply-To: <35577b73-0943-73f0-931e-a27109646d87@mgmiller.net> References: <35577b73-0943-73f0-931e-a27109646d87@mgmiller.net> Message-ID: On 29 November 2017 at 06:46, Mike Miller wrote: > Hi, the reason I use note is that I want it to be output by default. So it > must be above warning, or perhaps the default level changed. If the message to be displayed is part of the actual UX of a command line tool, our advice is "You don't want the logging module, you want the print() builtin": https://docs.python.org/3/howto/logging.html#when-to-use-logging As a relevant technical detail, it's also worth noting that the default handler emits messages on stderr, while CLI UX messages should generally be displayed on stdout. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From steve at pearwood.info Wed Nov 29 01:08:45 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 29 Nov 2017 17:08:45 +1100 Subject: [Python-ideas] [Python-Dev] What's the status of PEP 505: None-aware operators? In-Reply-To: <50C74ECC-D462-4FAC-8A9C-A12BC939ADEB@gmail.com> References: <28D91255-56A9-4CC2-B45D-F83ECD715544@langa.pl> <50C74ECC-D462-4FAC-8A9C-A12BC939ADEB@gmail.com> Message-ID: <20171129060843.GZ22248@ando.pearwood.info> On Tue, Nov 28, 2017 at 12:31:06PM -0800, Raymond Hettinger wrote: > > > I also cc python-dev to see if anybody here is strongly in favor or against this inclusion. > > Put me down for a strong -1. The proposal would occasionally save a > few keystokes but comes at the expense of giving Python a more Perlish > look and a more arcane feel. I think that's an unfair characterisation of the benefits of the PEP. It's not just "a few keystrokes". Ironically, the equivalent in Perl is // which Python has used for truncating division since version 2.4 or so. So if we're in danger of looking "Perlish", that ship has sailed a long time ago. Perl is hardly the only language with null-coalescing operators -- we might better describe ?? as being familiar to C#, PHP, Swift and Dart. That's two mature, well-known languages and two up-and-coming languages. [...] > timeout ?? local_timeout ?? global_timeout As opposed to the status quo: timeout if timeout is not None else (local_timeout if local_timeout is not None else global_timeout) Or shorter, but even harder to understand: (global_timeout if local_timeout is None else local_timeout) if timeout is None else timeout I'd much prefer to teach the version with ?? -- it has a simple explanation: "the first of the three given values which isn't None". The ?? itself needs to be memorized, but that's no different from any other operator. The first time I saw ** I was perplexed and couldn't imagine what it meaned. Here ?? doesn't merely save a few keystrokes, it significantly reduces the length and complexity of the expression and entirely cuts out the duplication of names. If you can teach timeout or local_timeout or global_timeout then you ought to be able to teach ??, as it is simpler: it only compares to None, and avoids needing to explain or justify Python's truthiness model. > 'foo' in (None ?? ['foo', 'bar']) If you can understand 'foo' in (False or ['foo', 'bar']) then surely you can understand the version with ??. > requested_quantity ?? default_quantity * price Again: (default_quantity if requested_quantity is None else requested_quantity) * price I'd much prefer to read, write and teach the version with ?? over the status quo. -- Steve From mertz at gnosis.cx Wed Nov 29 01:13:36 2017 From: mertz at gnosis.cx (David Mertz) Date: Tue, 28 Nov 2017 22:13:36 -0800 Subject: [Python-ideas] [Python-Dev] What's the status of PEP 505: None-aware operators? In-Reply-To: <20171129060843.GZ22248@ando.pearwood.info> References: <28D91255-56A9-4CC2-B45D-F83ECD715544@langa.pl> <50C74ECC-D462-4FAC-8A9C-A12BC939ADEB@gmail.com> <20171129060843.GZ22248@ando.pearwood.info> Message-ID: Strong -1 still from me. Too special case for syntax. Just write a function 'first_non_none()' that can perfectly will handle the need. On Nov 28, 2017 10:09 PM, "Steven D'Aprano" wrote: > On Tue, Nov 28, 2017 at 12:31:06PM -0800, Raymond Hettinger wrote: > > > > > I also cc python-dev to see if anybody here is strongly in favor or > against this inclusion. > > > > Put me down for a strong -1. The proposal would occasionally save a > > few keystokes but comes at the expense of giving Python a more Perlish > > look and a more arcane feel. > > I think that's an unfair characterisation of the benefits of the PEP. > It's not just "a few keystrokes". > > Ironically, the equivalent in Perl is // which Python has used for > truncating division since version 2.4 or so. So if we're in danger of > looking "Perlish", that ship has sailed a long time ago. > > Perl is hardly the only language with null-coalescing operators -- we > might better describe ?? as being familiar to C#, PHP, Swift and Dart. > That's two mature, well-known languages and two up-and-coming languages. > > > [...] > > timeout ?? local_timeout ?? global_timeout > > As opposed to the status quo: > > timeout if timeout is not None else (local_timeout if local_timeout is > not None else global_timeout) > > Or shorter, but even harder to understand: > > (global_timeout if local_timeout is None else local_timeout) if > timeout is None else timeout > > I'd much prefer to teach the version with ?? -- it has a simple > explanation: "the first of the three given values which isn't None". The > ?? itself needs to be memorized, but that's no different from any other > operator. The first time I saw ** I was perplexed and couldn't imagine > what it meaned. > > Here ?? doesn't merely save a few keystrokes, it significantly reduces > the length and complexity of the expression and entirely cuts out the > duplication of names. > > If you can teach > > timeout or local_timeout or global_timeout > > then you ought to be able to teach ??, as it is simpler: it only > compares to None, and avoids needing to explain or justify Python's > truthiness model. > > > > 'foo' in (None ?? ['foo', 'bar']) > > If you can understand > > 'foo' in (False or ['foo', 'bar']) > > then surely you can understand the version with ??. > > > > requested_quantity ?? default_quantity * price > > Again: > > (default_quantity if requested_quantity is None else > requested_quantity) * price > > > I'd much prefer to read, write and teach the version with ?? over the > status quo. > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Wed Nov 29 02:14:12 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 29 Nov 2017 09:14:12 +0200 Subject: [Python-ideas] [Python-Dev] What's the status of PEP 505: None-aware operators? In-Reply-To: <20171129060843.GZ22248@ando.pearwood.info> References: <28D91255-56A9-4CC2-B45D-F83ECD715544@langa.pl> <50C74ECC-D462-4FAC-8A9C-A12BC939ADEB@gmail.com> <20171129060843.GZ22248@ando.pearwood.info> Message-ID: 29.11.17 08:08, Steven D'Aprano ????: > Perl is hardly the only language with null-coalescing operators -- we > might better describe ?? as being familiar to C#, PHP, Swift and Dart. > That's two mature, well-known languages and two up-and-coming languages. What is the syntax of the ternary operator in these languages? From storchaka at gmail.com Wed Nov 29 02:39:57 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 29 Nov 2017 09:39:57 +0200 Subject: [Python-ideas] Proposal: allow length_hint to specify infinite iterators In-Reply-To: <20171129013403.GX22248@ando.pearwood.info> References: <20171129013403.GX22248@ando.pearwood.info> Message-ID: 29.11.17 03:34, Steven D'Aprano ????: > This wastes the opportunity to fail fast on operations which cannot > possibly succeed, e.g. list(count()) must eventually fail with > MemoryError. Or worse: if the OS starts thrashing trying to meet the > memory requests, you can lock up the computer. > > I propose that we: > > (1) extend the __length_hint__ protocol to allow iterators to report > that they are infinite; > > (2) and recommend that consumers of iterators (such as list) that > require finite input should fail fast in the event of an infinite > iterator. Infinite iterators are rare. And count() is even more special in the sense that iterating it doesn't have side effects and is not interruptible. > (c) take the convention that returning sys.maxint means infinity; Returning sys.maxsize will likely lead to failing fast with MemoryError. From uranusjr at gmail.com Wed Nov 29 02:17:04 2017 From: uranusjr at gmail.com (Tzu-ping Chung) Date: Tue, 28 Nov 2017 23:17:04 -0800 (PST) Subject: [Python-ideas] Looking for input to help with the pip situation In-Reply-To: References: <5A087DE7.4050306@brenbarn.net> <8d5574dc-b614-ccfe-3ce8-f1ba86508179@python.org> Message-ID: <1881f5f0-1911-4c12-b9ea-e7bcdcbfcac8@googlegroups.com> Hi everyone, I just discovered this thread and thought I this is a good chance I do some advertisement. I was helping a friend doing Python tutorial and face a somewhat similar situation with beginners using Windows. It is not particularly difficult to teach them to use py, but with the vast majority of resources not mentioning it, it?s a bit of a stretch to expect a beginner to know when to substitute what command to what after they finish the workshop. And a workshop is next to useless (in my opinion) if attendees can?t somehow go on by themselves after the event. Anyway, I eventually decide it is best if Windows users can mimic how people run Python things on POSIX systems. The end product is this: https://github.com/uranusjr/snafu This works somehow like Homebrew and APT, but just for CPython. It allows you to use almost all the familiar commands, like *python3*, *pip3*, anything else you install with pip, on Windows. I made a conscious decision to not include python, pip, etc. Although I agree ?python? is better in the long run, we are always stuck with the Python 2-3 situation for now, and need a solution to *somehow* make the distinction. But from what I?ve tested, everything else works as expected without much hassle. The project is still in a very early stage (I only told a handful of friends about it), and I?ve put up working on it recently because my day job doesn?t allow me to think too much about it at the moment. But reading this thread I hope this can provide some ideas to others, and maybe find some interested people working with me on this issue. Thanks! TP -- Tzu-ping Chung (@uranusjr) uranusjr at gmail.com https://uranusjr.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Nov 29 03:14:36 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 29 Nov 2017 18:14:36 +1000 Subject: [Python-ideas] [Python-Dev] What's the status of PEP 505: None-aware operators? In-Reply-To: References: <28D91255-56A9-4CC2-B45D-F83ECD715544@langa.pl> <50C74ECC-D462-4FAC-8A9C-A12BC939ADEB@gmail.com> <20171129060843.GZ22248@ando.pearwood.info> Message-ID: On 29 November 2017 at 16:13, David Mertz wrote: > Strong -1 still from me. Too special case for syntax. Just write a function > 'first_non_none()' that can perfectly will handle the need. That's the equivalent of SQL's COALESCE, and it's insufficient for the same reason "and" and "or" are syntax rather than builtins: the function form doesn't provide short-circuiting behaviour. As far as utility goes, I put it in a similar category to matrix multiplication: if you don't need it, you don't need it, but when you do need it, you need it a *lot*. The use case where these operations come up is when you're working with partially structured hierarchical data (*cough*JSON*cough*). In those kinds of structures, None is frequently used as a marker to say "this entire subtree is missing", and you either want to propagate that None, or else replace it with something else (and the "something else" may be a network call to a different service, so you definitely don't want to do it if you don't need to). So I'd remind folks to try to avoid the "I don't need this, so nobody needs this" mistake. It's OK to say "the use case exists, but I still don't want that particular syntax for it in Python" (I'm personally inclined to agree with you on that front). It's not OK to try to claim there are no use cases where the status quo is awkward enough to become irritating (since it's an empirically false statement that you don't need to be making). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From kirillbalunov at gmail.com Wed Nov 29 03:49:22 2017 From: kirillbalunov at gmail.com (Kirill Balunov) Date: Wed, 29 Nov 2017 11:49:22 +0300 Subject: [Python-ideas] [Python-Dev] What's the status of PEP 505: None-aware operators? In-Reply-To: References: <28D91255-56A9-4CC2-B45D-F83ECD715544@langa.pl> <50C74ECC-D462-4FAC-8A9C-A12BC939ADEB@gmail.com> <20171129060843.GZ22248@ando.pearwood.info> Message-ID: 2017-11-29 11:14 GMT+03:00 Nick Coghlan : > It's OK to say "the use case exists, but I still > don't want that particular syntax for it in Python" (I'm personally > inclined to agree with you on that front). It's not OK to try to claim > there are no use cases where the status quo is awkward enough to > become irritating (since it's an empirically false statement that you > don't need to be making). > If the problem with the proposed syntax, but there are cases for use, it may be worth to bikeshed one more time? 2017-11-29 9:08 GMT+03:00 Steven D'Aprano : I'd much prefer to read, write and teach the version with ?? over the > status quo. Since the proposed semantics is more similar to the idea of "or", may be it is better to consider something like: timeout then local_timeout then global_timeout I do not know how much this is a frequent case to be worthy of a keyword. With kind regards, -gdg 2017-11-29 11:14 GMT+03:00 Nick Coghlan : > On 29 November 2017 at 16:13, David Mertz wrote: > > Strong -1 still from me. Too special case for syntax. Just write a > function > > 'first_non_none()' that can perfectly will handle the need. > > That's the equivalent of SQL's COALESCE, and it's insufficient for the > same reason "and" and "or" are syntax rather than builtins: the > function form doesn't provide short-circuiting behaviour. > > As far as utility goes, I put it in a similar category to matrix > multiplication: if you don't need it, you don't need it, but when you > do need it, you need it a *lot*. > > The use case where these operations come up is when you're working > with partially structured hierarchical data (*cough*JSON*cough*). In > those kinds of structures, None is frequently used as a marker to say > "this entire subtree is missing", and you either want to propagate > that None, or else replace it with something else (and the "something > else" may be a network call to a different service, so you definitely > don't want to do it if you don't need to). > > So I'd remind folks to try to avoid the "I don't need this, so nobody > needs this" mistake. It's OK to say "the use case exists, but I still > don't want that particular syntax for it in Python" (I'm personally > inclined to agree with you on that front). It's not OK to try to claim > there are no use cases where the status quo is awkward enough to > become irritating (since it's an empirically false statement that you > don't need to be making). > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Wed Nov 29 03:52:51 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 29 Nov 2017 10:52:51 +0200 Subject: [Python-ideas] Add a dict with the attribute access capability Message-ID: In 3.7 I have removed an old-deprecated plistlib.Dict. [1] Actually it already was deprecated when the plistlib module was added to the regular stdlib in Python 2.6. This is a dict subclass which allows to access items as attributes. d = plistlib.Dict() d['a'] = 1 assert d.a == 1 d.b = 2 assert d['b'] == 2 Raymond noticed that that capability seemed nice to have. What do you think about reviving this type as general purpose type in collections or types? Perhaps it can be convenient for working with JSON, plists, configuration files, databases and in other cases that need a dict with string keys. If reintroduce it, there are open questions. 1. The name of the type. 2. The location of the type. collections or types? Or other variants? 3. How it will collaborate with OrderedDict, defaultdict, etc? 4. Should it be a dict subclass, or a mixin, or a proxy? Or add several types? [1] https://bugs.python.org/issue29196 From ncoghlan at gmail.com Wed Nov 29 04:00:43 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 29 Nov 2017 19:00:43 +1000 Subject: [Python-ideas] [Python-Dev] What's the status of PEP 505: None-aware operators? In-Reply-To: References: <28D91255-56A9-4CC2-B45D-F83ECD715544@langa.pl> <50C74ECC-D462-4FAC-8A9C-A12BC939ADEB@gmail.com> <20171129060843.GZ22248@ando.pearwood.info> Message-ID: On 29 November 2017 at 18:49, Kirill Balunov wrote: > 2017-11-29 11:14 GMT+03:00 Nick Coghlan : >> >> It's OK to say "the use case exists, but I still >> don't want that particular syntax for it in Python" (I'm personally >> inclined to agree with you on that front). It's not OK to try to claim >> there are no use cases where the status quo is awkward enough to >> become irritating (since it's an empirically false statement that you >> don't need to be making). > > If the problem with the proposed syntax, but there are cases for use, it may > be worth to bikeshed one more time? I don't think enough time has passed since we first discussed it. Since Guido has already said "not for 3.7", we can give folks another 12+ months to ponder the problem, and then maybe revisit it for 3.8. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From steve at pearwood.info Wed Nov 29 04:45:27 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 29 Nov 2017 20:45:27 +1100 Subject: [Python-ideas] [Python-Dev] What's the status of PEP 505: None-aware operators? In-Reply-To: References: <28D91255-56A9-4CC2-B45D-F83ECD715544@langa.pl> <50C74ECC-D462-4FAC-8A9C-A12BC939ADEB@gmail.com> <20171129060843.GZ22248@ando.pearwood.info> Message-ID: <20171129094527.GA22248@ando.pearwood.info> On Wed, Nov 29, 2017 at 09:14:12AM +0200, Serhiy Storchaka wrote: > 29.11.17 08:08, Steven D'Aprano ????: > >Perl is hardly the only language with null-coalescing operators -- we > >might better describe ?? as being familiar to C#, PHP, Swift and Dart. > >That's two mature, well-known languages and two up-and-coming languages. > > What is the syntax of the ternary operator in these languages? All four use: condition ? first : second for the ternary if operator. -- Steve From elazarg at gmail.com Wed Nov 29 04:54:46 2017 From: elazarg at gmail.com (Elazar) Date: Wed, 29 Nov 2017 09:54:46 +0000 Subject: [Python-ideas] [Python-Dev] What's the status of PEP 505: None-aware operators? In-Reply-To: <20171129094527.GA22248@ando.pearwood.info> References: <28D91255-56A9-4CC2-B45D-F83ECD715544@langa.pl> <50C74ECC-D462-4FAC-8A9C-A12BC939ADEB@gmail.com> <20171129060843.GZ22248@ando.pearwood.info> <20171129094527.GA22248@ando.pearwood.info> Message-ID: ?????? ??? ??, 29 ????? 2017, 11:46, ??? Steven D'Aprano ?< steve at pearwood.info>: > On Wed, Nov 29, 2017 at 09:14:12AM +0200, Serhiy Storchaka wrote: > > 29.11.17 08:08, Steven D'Aprano ????: > > >Perl is hardly the only language with null-coalescing operators -- we > > >might better describe ?? as being familiar to C#, PHP, Swift and Dart. > > >That's two mature, well-known languages and two up-and-coming languages. > > > > What is the syntax of the ternary operator in these languages? > > All four use: > > condition ? first : second > > for the ternary if operator. > This suggests something like "ifNone" keyword : x = a ifNone b ifNone c Elazar -------------- next part -------------- An HTML attachment was scrubbed... URL: From lele at metapensiero.it Wed Nov 29 05:28:14 2017 From: lele at metapensiero.it (Lele Gaifax) Date: Wed, 29 Nov 2017 11:28:14 +0100 Subject: [Python-ideas] [Python-Dev] What's the status of PEP 505: None-aware operators? References: <28D91255-56A9-4CC2-B45D-F83ECD715544@langa.pl> <50C74ECC-D462-4FAC-8A9C-A12BC939ADEB@gmail.com> <20171129060843.GZ22248@ando.pearwood.info> Message-ID: <878tep8jlt.fsf@metapensiero.it> Kirill Balunov writes: > Since the proposed semantics is more similar to the idea of "or", may be it > is better to consider something like: > > timeout then local_timeout then global_timeout > > I do not know how much this is a frequent case to be worthy of a keyword. That sounds awkward... what about timeout else local_timeout else global_timeout instead? ciao, lele. -- nickname: Lele Gaifax | Quando vivr? di quello che ho pensato ieri real: Emanuele Gaifas | comincer? ad aver paura di chi mi copia. lele at metapensiero.it | -- Fortunato Depero, 1929. From elazarg at gmail.com Wed Nov 29 05:39:42 2017 From: elazarg at gmail.com (Elazar) Date: Wed, 29 Nov 2017 10:39:42 +0000 Subject: [Python-ideas] [Python-Dev] What's the status of PEP 505: None-aware operators? In-Reply-To: <878tep8jlt.fsf@metapensiero.it> References: <28D91255-56A9-4CC2-B45D-F83ECD715544@langa.pl> <50C74ECC-D462-4FAC-8A9C-A12BC939ADEB@gmail.com> <20171129060843.GZ22248@ando.pearwood.info> <878tep8jlt.fsf@metapensiero.it> Message-ID: ?????? ??? ??, 29 ????? 2017, 12:29, ??? Lele Gaifax ?: > Kirill Balunov writes: > > > Since the proposed semantics is more similar to the idea of "or", may be > it > > is better to consider something like: > > > > timeout then local_timeout then global_timeout > > > > I do not know how much this is a frequent case to be worthy of a keyword. > > That sounds awkward... what about > > timeout else local_timeout else global_timeout > > instead? > I think then and else can be ruled out since they can be very confusing when used in a conditinal statement or a comprehension. x = a if x else y else b # wait. what? Elazar -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Wed Nov 29 06:07:24 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 29 Nov 2017 12:07:24 +0100 Subject: [Python-ideas] Proposal: allow length_hint to specify infinite iterators References: <20171129013403.GX22248@ando.pearwood.info> Message-ID: <20171129120724.4c900ee7@fsol> On Wed, 29 Nov 2017 09:39:57 +0200 Serhiy Storchaka wrote: > 29.11.17 03:34, Steven D'Aprano ????: > > This wastes the opportunity to fail fast on operations which cannot > > possibly succeed, e.g. list(count()) must eventually fail with > > MemoryError. Or worse: if the OS starts thrashing trying to meet the > > memory requests, you can lock up the computer. > > > > I propose that we: > > > > (1) extend the __length_hint__ protocol to allow iterators to report > > that they are infinite; > > > > (2) and recommend that consumers of iterators (such as list) that > > require finite input should fail fast in the event of an infinite > > iterator. > > Infinite iterators are rare. And count() is even more special in the > sense that iterating it doesn't have side effects and is not interruptible. Not to mention that many infinite iterators cannot be predicted in advance to be infinite. Only the more trivial cases such as count() would apply (but not a takewhile() involving count(), for example). Regards Antoine. From AlonSnir at hotmail.com Tue Nov 28 18:15:02 2017 From: AlonSnir at hotmail.com (Alon Snir) Date: Tue, 28 Nov 2017 23:15:02 +0000 Subject: [Python-ideas] How assignment should work with generators? Message-ID: On Wed, Nov 29, 2017 at 5:46 AM, Alon Snir wrote: > I would like to mention that the issue of assignment to a target list, is also relevant to the case of elementwise assignment to a mutable sequence (e.g. lists and arrays). Here as well, the rhs of the assignment statement states the number of elements to be assigned. Consequently, the following example will get into infinite loop: > >>>> from itertools import count >>>> A = [] >>>> A[:] = count() > > Writing "A[:2] = count()" will cause the same result. > Here as well, it is currently safer to use islice if the rhs is a generator or an iterator. e.g.: > >>>> it = count() >>>> A[:] = islice(it,2) > > In my opinion, it is be better to devise a solution that could be applied in both cases. Maybe a new kind of assignment operator that will be dedicated to this kind of assignment. i.e. elementwise assignment with restriction on the number of elements to be assigned, based on the length of the lhs object (or the number of targets in the target list). > ChrisA wrote: Hmm. The trouble is that slice assignment doesn't have a fixed number of targets. If you say "x, y = spam", there's a clear indication that 'spam' needs to provide exactly two values; but "A[:] = spam" could have any number of values, and it'll expand or shrink the list accordingly. Rhodri James wrote: Flatly, no. It is better not to ask for things you don't want in the first place, in this case the infinite sequence. Still, don't let me discourage you from working on this. If you can define how such an assignment would work, or even the length of A[:] as an assignment target, I'm not going to dismiss it out of hand. My answer: The idea is to define an alternative assignment rule, that is to assign exactly as many elements as the current length of the lhs object (without expanding or shrinking it). Suppose "?=" is the operator for the alternative assignment rule; A=[None]*2; and "iterator" is any iterator (finite or infinite). In this case, the following code: >>> A ?= iterator would behave like this: >>> A[:] = islice(iterator, 2) # where 2 is the length of A And as suggested earlier for the case of assignment to a target list, the following code: >>> x, y ?= iterator would behave like this: >>> x, y = islice(iterator, 2) # where 2 is the number of targets Regarding the length issue: Is there any difficulty in finding the length of a sliced sequence? After all, the range object has a len method. Therefore, the length of A[s] (where s is a slice object) could be evaluated as follows: >>> len(range(*s.indices(len(A)))) From storchaka at gmail.com Wed Nov 29 06:53:42 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 29 Nov 2017 13:53:42 +0200 Subject: [Python-ideas] [Python-Dev] What's the status of PEP 505: None-aware operators? In-Reply-To: <20171129094527.GA22248@ando.pearwood.info> References: <28D91255-56A9-4CC2-B45D-F83ECD715544@langa.pl> <50C74ECC-D462-4FAC-8A9C-A12BC939ADEB@gmail.com> <20171129060843.GZ22248@ando.pearwood.info> <20171129094527.GA22248@ando.pearwood.info> Message-ID: 29.11.17 11:45, Steven D'Aprano ????: > On Wed, Nov 29, 2017 at 09:14:12AM +0200, Serhiy Storchaka wrote: >> 29.11.17 08:08, Steven D'Aprano ????: >>> Perl is hardly the only language with null-coalescing operators -- we >>> might better describe ?? as being familiar to C#, PHP, Swift and Dart. >>> That's two mature, well-known languages and two up-and-coming languages. >> >> What is the syntax of the ternary operator in these languages? > > All four use: > > condition ? first : second > > for the ternary if operator. If all four use ?, it is natural that in operators which are shortcuts of the ternary operator they use ?. But in Python the bar of introducing ? is higher. From rhodri at kynesim.co.uk Wed Nov 29 07:21:35 2017 From: rhodri at kynesim.co.uk (Rhodri James) Date: Wed, 29 Nov 2017 12:21:35 +0000 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: References: Message-ID: <0b0dbf31-d41a-b81f-ae58-0d19898918f3@kynesim.co.uk> On 28/11/17 23:15, Alon Snir wrote: > On Wed, Nov 29, 2017 at 5:46 AM, Alon Snir wrote: >> I would like to mention that the issue of assignment to a target list, is also relevant to the case of elementwise assignment to a mutable sequence (e.g. lists and arrays). Here as well, the rhs of the assignment statement states the number of elements to be assigned. Consequently, the following example will get into infinite loop: >> >>>>> from itertools import count >>>>> A = [] >>>>> A[:] = count() >> >> Writing "A[:2] = count()" will cause the same result. >> Here as well, it is currently safer to use islice if the rhs is a generator or an iterator. e.g.: >> >>>>> it = count() >>>>> A[:] = islice(it,2) >> >> In my opinion, it is be better to devise a solution that could be applied in both cases. Maybe a new kind of assignment operator that will be dedicated to this kind of assignment. i.e. elementwise assignment with restriction on the number of elements to be assigned, based on the length of the lhs object (or the number of targets in the target list). >> > > ChrisA wrote: > Hmm. The trouble is that slice assignment doesn't have a fixed number of targets. If you say "x, y = spam", there's a clear indication that 'spam' needs to provide exactly two values; but "A[:] = spam" could have any number of values, and it'll expand or shrink the list accordingly. > > Rhodri James wrote: > Flatly, no. It is better not to ask for things you don't want in the first place, in this case the infinite sequence. > Still, don't let me discourage you from working on this. If you can define how such an assignment would work, or even the length of A[:] as an assignment target, I'm not going to dismiss it out of hand. > > > My answer: > > The idea is to define an alternative assignment rule, that is to assign exactly as many elements as the current length of the lhs object (without expanding or shrinking it). Suppose "?=" is the operator for the alternative assignment rule; A=[None]*2; and "iterator" is any iterator (finite or infinite). In this case, the following code: > >>>> A ?= iterator > > would behave like this: > >>>> A[:] = islice(iterator, 2) # where 2 is the length of A > > And as suggested earlier for the case of assignment to a target list, the following code: > >>>> x, y ?= iterator > > would behave like this: > >>>> x, y = islice(iterator, 2) # where 2 is the number of targets You're still getting -1000 from me on this. You want to introduce magic syntax to make writing sloppy code easier. That's two big barriers you aren't getting over. -- Rhodri James *-* Kynesim Ltd From stephanh42 at gmail.com Wed Nov 29 07:38:26 2017 From: stephanh42 at gmail.com (Stephan Houben) Date: Wed, 29 Nov 2017 13:38:26 +0100 Subject: [Python-ideas] [Python-Dev] What's the status of PEP 505: None-aware operators? In-Reply-To: References: <28D91255-56A9-4CC2-B45D-F83ECD715544@langa.pl> <50C74ECC-D462-4FAC-8A9C-A12BC939ADEB@gmail.com> <20171129060843.GZ22248@ando.pearwood.info> <20171129094527.GA22248@ando.pearwood.info> Message-ID: What about more English-like syntax: X or else Y E.g. cache.get(foo) or else expensive_call(foo) Stephan Op 29 nov. 2017 12:54 schreef "Serhiy Storchaka" : 29.11.17 11:45, Steven D'Aprano ????: On Wed, Nov 29, 2017 at 09:14:12AM +0200, Serhiy Storchaka wrote: > >> 29.11.17 08:08, Steven D'Aprano ????: >> >>> Perl is hardly the only language with null-coalescing operators -- we >>> might better describe ?? as being familiar to C#, PHP, Swift and Dart. >>> That's two mature, well-known languages and two up-and-coming languages. >>> >> >> What is the syntax of the ternary operator in these languages? >> > > All four use: > > condition ? first : second > > for the ternary if operator. > If all four use ?, it is natural that in operators which are shortcuts of the ternary operator they use ?. But in Python the bar of introducing ? is higher. _______________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Nov 29 07:39:52 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 29 Nov 2017 22:39:52 +1000 Subject: [Python-ideas] [Python-Dev] What's the status of PEP 505: None-aware operators? In-Reply-To: References: <28D91255-56A9-4CC2-B45D-F83ECD715544@langa.pl> <50C74ECC-D462-4FAC-8A9C-A12BC939ADEB@gmail.com> <20171129060843.GZ22248@ando.pearwood.info> <20171129094527.GA22248@ando.pearwood.info> Message-ID: On 29 November 2017 at 21:53, Serhiy Storchaka wrote: > 29.11.17 11:45, Steven D'Aprano ????: >> On Wed, Nov 29, 2017 at 09:14:12AM +0200, Serhiy Storchaka wrote: >>> What is the syntax of the ternary operator in these languages? >> >> All four use: >> >> condition ? first : second >> >> for the ternary if operator. > > If all four use ?, it is natural that in operators which are shortcuts of > the ternary operator they use ?. But in Python the bar of introducing ? is > higher. It's also noteworthy that they all offer "&&" and "||" as short circuiting "and" and "or" operators. When you have that pattern established, then "??" fits right in. With Python's only spelling for the logical operators being "and" and "or", the cryptic unpronounceable nature of "??" becomes much harder to ignore. "a and b" -> pronounced "a and b" "a or b" -> pronounced "a or b" "a ?? b" -> pronounced "a b" The only potentially pronounceable versions I've been able to come up with are a fair bit longer: "a if def else b" -> pronounced "a if defined, else b" "a if ?? is not None else b" -> pronounced "a if the leftmost operand is not None, else b" "a if ?? is None else ??.b" -> pronounced "a if the leftmost operand is None, else the leftmost operand dot b" (The last two examples there are a version where "??" appearing in either the condition expression or the RHS of a conditional expression would cause the leftmost operand to be eagerly evaluated, placed in a hidden temporary variable and then referenced multiple times as a subexpression. A similar construct could then also be used in filter expressions in comprehensions to refer back to the loop's result clause: "[f(x) for x in iterable if ?? is not None]". It's still magical and hard to look up syntax, but "??" as a magic placeholder to say "Reference the leftmost operand in the current expression here, but still only evaluate it once" seems nicer to me than using it as a cryptic binary operator) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Wed Nov 29 07:41:23 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 29 Nov 2017 22:41:23 +1000 Subject: [Python-ideas] [Python-Dev] What's the status of PEP 505: None-aware operators? In-Reply-To: References: <28D91255-56A9-4CC2-B45D-F83ECD715544@langa.pl> <50C74ECC-D462-4FAC-8A9C-A12BC939ADEB@gmail.com> <20171129060843.GZ22248@ando.pearwood.info> <20171129094527.GA22248@ando.pearwood.info> Message-ID: On 29 November 2017 at 22:38, Stephan Houben wrote: > What about more English-like syntax: > > X or else Y The problem with constructs like this is that they look like they should mean the same thing as "X or Y". Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From stephanh42 at gmail.com Wed Nov 29 08:01:12 2017 From: stephanh42 at gmail.com (Stephan Houben) Date: Wed, 29 Nov 2017 14:01:12 +0100 Subject: [Python-ideas] [Python-Dev] What's the status of PEP 505: None-aware operators? In-Reply-To: References: <28D91255-56A9-4CC2-B45D-F83ECD715544@langa.pl> <50C74ECC-D462-4FAC-8A9C-A12BC939ADEB@gmail.com> <20171129060843.GZ22248@ando.pearwood.info> <20171129094527.GA22248@ando.pearwood.info> Message-ID: What about a more general: A if B else C which would allow A if is not None else C but also e.g. A if >= 1 else 0 Stephan Op 29 nov. 2017 13:41 schreef "Nick Coghlan" : > On 29 November 2017 at 22:38, Stephan Houben wrote: > > What about more English-like syntax: > > > > X or else Y > > The problem with constructs like this is that they look like they > should mean the same thing as "X or Y". > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Wed Nov 29 08:01:04 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 29 Nov 2017 15:01:04 +0200 Subject: [Python-ideas] [Python-Dev] What's the status of PEP 505: None-aware operators? In-Reply-To: References: <28D91255-56A9-4CC2-B45D-F83ECD715544@langa.pl> <50C74ECC-D462-4FAC-8A9C-A12BC939ADEB@gmail.com> <20171129060843.GZ22248@ando.pearwood.info> <20171129094527.GA22248@ando.pearwood.info> Message-ID: 29.11.17 14:39, Nick Coghlan ????: > "a if def else b" -> pronounced "a if defined, else b" I understand "a if defined, else b" as try: result = a except (NameError, AttributeError, LookupError): result = b The problem is that None is not undefined. This is a regular value. Seems it is not so special in Python as NULL or undef in other languages. Python even don't have a special purposed NullPointerException (or NoneError). From p.f.moore at gmail.com Wed Nov 29 08:06:14 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 29 Nov 2017 13:06:14 +0000 Subject: [Python-ideas] [Python-Dev] What's the status of PEP 505: None-aware operators? In-Reply-To: References: <28D91255-56A9-4CC2-B45D-F83ECD715544@langa.pl> <50C74ECC-D462-4FAC-8A9C-A12BC939ADEB@gmail.com> <20171129060843.GZ22248@ando.pearwood.info> <20171129094527.GA22248@ando.pearwood.info> Message-ID: On 29 November 2017 at 12:41, Nick Coghlan wrote: > On 29 November 2017 at 22:38, Stephan Houben wrote: >> What about more English-like syntax: >> >> X or else Y > > The problem with constructs like this is that they look like they > should mean the same thing as "X or Y". Keyword based and multi-word approaches also break down much faster when you get more terms. X or else Y looks OK (ignoring Nick's comment - I could pick another keyword-based proposal, but I'm too lazy to look for one I like), but when you have 4 options, X or else Y or else Z or else W the benefit isn't as obvious. Use lower-case and longer names item_one or else item_two or else list_one[the_index] or dict_one['key_one'] and it becomes just a muddle of words. Conversely, punctuation-based examples look worse with shorter variables and with expressions rather than identifiers: item_one ?? item_two ?? another_item ?? one_more_possibility vs x ?? y[2] ?? kw['id'] ?? 3 + 7 IMO, this is a case where artificial examples are unusually bad at conveying the actual feel of a proposal. It's pretty easy to turn someone's acceptable-looking example into an incomprehensible mess, just by changing variable names and example terms. So I think it's critically important for any proposal along these lines (even just posts to the mailing list, and definitely for a PEP), that it's argued in terms of actual code examples in current projects that would reasonably be modified to use the proposed syntax. And people wanting to be particularly honest in their proposals should probably include both best-case and worst-case examples of readability. Paul PS Also, I want a pony. (I really do understand that the above is not realistic, but maybe I can hope that at least anyone writing a PEP take it into consideration :-)) From storchaka at gmail.com Wed Nov 29 08:11:12 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 29 Nov 2017 15:11:12 +0200 Subject: [Python-ideas] [Python-Dev] What's the status of PEP 505: None-aware operators? In-Reply-To: References: <28D91255-56A9-4CC2-B45D-F83ECD715544@langa.pl> <50C74ECC-D462-4FAC-8A9C-A12BC939ADEB@gmail.com> <20171129060843.GZ22248@ando.pearwood.info> <20171129094527.GA22248@ando.pearwood.info> Message-ID: 29.11.17 15:01, Stephan Houben ????: > What about a more general: > > A if B else C > > which would allow > > A if is not None else C > > but also e.g. > > A if >= 1 else 0 This look the most "natural" to me. I.e. the least "unnatural". If we even will introduce a new special syntax I will prefer this syntax. The only disadvantage is that this will prevent introducing "angular parenthesis" in future: A if == B else C. From dmoisset at machinalis.com Wed Nov 29 08:21:28 2017 From: dmoisset at machinalis.com (Daniel Moisset) Date: Wed, 29 Nov 2017 10:21:28 -0300 Subject: [Python-ideas] Add a dict with the attribute access capability In-Reply-To: References: Message-ID: As reference of prior art, there is https://pypi.python.org/pypi/munch in PyPI On 29 November 2017 at 05:52, Serhiy Storchaka wrote: > In 3.7 I have removed an old-deprecated plistlib.Dict. [1] Actually it > already was deprecated when the plistlib module was added to the regular > stdlib in Python 2.6. > > This is a dict subclass which allows to access items as attributes. > > d = plistlib.Dict() > d['a'] = 1 > assert d.a == 1 > d.b = 2 > assert d['b'] == 2 > > Raymond noticed that that capability seemed nice to have. > > What do you think about reviving this type as general purpose type in > collections or types? Perhaps it can be convenient for working with JSON, > plists, configuration files, databases and in other cases that need a dict > with string keys. > > If reintroduce it, there are open questions. > > 1. The name of the type. > > 2. The location of the type. collections or types? Or other variants? > > 3. How it will collaborate with OrderedDict, defaultdict, etc? > > 4. Should it be a dict subclass, or a mixin, or a proxy? Or add several > types? > > > [1] https://bugs.python.org/issue29196 > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Daniel F. Moisset - UK Country Manager - Machinalis Limited www.machinalis.co.uk Skype: @dmoisset T: + 44 7398 827139 1 Fore St, London, EC2Y 9DT Machinalis Limited is a company registered in England and Wales. Registered number: 10574987. -------------- next part -------------- An HTML attachment was scrubbed... URL: From elazarg at gmail.com Wed Nov 29 08:28:59 2017 From: elazarg at gmail.com (Elazar) Date: Wed, 29 Nov 2017 13:28:59 +0000 Subject: [Python-ideas] [Python-Dev] What's the status of PEP 505: None-aware operators? In-Reply-To: References: <28D91255-56A9-4CC2-B45D-F83ECD715544@langa.pl> <50C74ECC-D462-4FAC-8A9C-A12BC939ADEB@gmail.com> <20171129060843.GZ22248@ando.pearwood.info> <20171129094527.GA22248@ando.pearwood.info> Message-ID: On Wed, Nov 29, 2017 at 3:12 PM Serhiy Storchaka wrote: > 29.11.17 15:01, Stephan Houben ????: > > What about a more general: > > > > A if B else C > > > > which would allow > > > > A if is not None else C > > > > but also e.g. > > > > A if >= 1 else 0 > > This look the most "natural" to me. I.e. the least "unnatural". If we > even will introduce a new special syntax I will prefer this syntax. > > The only disadvantage is that this will prevent introducing "angular > parenthesis" in future: A if == B else C. > > Also, how do you treat more complex conditions? do you prohibit them? A if >= 0 and < 2 else 0 B if >= 0 or < -1 else -1 C if and B else A # i.e. C if C and B else E D if -y else E # is this an unary minus or a binary operation? If any of these is allowed, it might confuse people. It also break the ability to extract subexpression to variables. (I kinda like this idea, but it doesn't look like a small, simple change to the language, whereas ?? does). Elazar -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Wed Nov 29 09:49:14 2017 From: mertz at gnosis.cx (David Mertz) Date: Wed, 29 Nov 2017 06:49:14 -0800 Subject: [Python-ideas] [Python-Dev] What's the status of PEP 505: None-aware operators? In-Reply-To: References: <28D91255-56A9-4CC2-B45D-F83ECD715544@langa.pl> <50C74ECC-D462-4FAC-8A9C-A12BC939ADEB@gmail.com> <20171129060843.GZ22248@ando.pearwood.info> Message-ID: I didn't say no use case exists, but rather "too special case." And/or shortcutting is great, but truthiness feels much more general than noneness to me. But yes, 'first_non_none()" might have to address laziness in some manner, depending on the use case. Zero argument lambdas and expression quoting are the usual hacks (yes, both ugly in their own ways). On Nov 29, 2017 12:14 AM, "Nick Coghlan" wrote: On 29 November 2017 at 16:13, David Mertz wrote: > Strong -1 still from me. Too special case for syntax. Just write a function > 'first_non_none()' that can perfectly will handle the need. That's the equivalent of SQL's COALESCE, and it's insufficient for the same reason "and" and "or" are syntax rather than builtins: the function form doesn't provide short-circuiting behaviour. As far as utility goes, I put it in a similar category to matrix multiplication: if you don't need it, you don't need it, but when you do need it, you need it a *lot*. The use case where these operations come up is when you're working with partially structured hierarchical data (*cough*JSON*cough*). In those kinds of structures, None is frequently used as a marker to say "this entire subtree is missing", and you either want to propagate that None, or else replace it with something else (and the "something else" may be a network call to a different service, so you definitely don't want to do it if you don't need to). So I'd remind folks to try to avoid the "I don't need this, so nobody needs this" mistake. It's OK to say "the use case exists, but I still don't want that particular syntax for it in Python" (I'm personally inclined to agree with you on that front). It's not OK to try to claim there are no use cases where the status quo is awkward enough to become irritating (since it's an empirically false statement that you don't need to be making). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From c at anthonyrisinger.com Wed Nov 29 11:39:02 2017 From: c at anthonyrisinger.com (C Anthony Risinger) Date: Wed, 29 Nov 2017 10:39:02 -0600 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: <5A1C8879.2050105@canterbury.ac.nz> References: <20171127135502.GF22248@ando.pearwood.info> <20171127153550.GI22248@ando.pearwood.info> <5A1C8879.2050105@canterbury.ac.nz> Message-ID: On Mon, Nov 27, 2017 at 3:49 PM, Greg Ewing wrote: > C Anthony Risinger wrote: > >> * Perhaps existence of `__len__` should influence unpacking? There is a >> semantic difference (and typically a visual one too) between 1-to-1 >> matching a fixed-width sequence/container on the RHS to identifiers on the >> LHS, even if they look similar (ie. "if RHS has a length make it fit, >> otherwise don't"). >> > > -1. There's a convention that an iterator can implement __len__ > to provide a hint about the number of items it will return > (useful for preallocating space, etc.) It would be very > annoying if such an iterator behaved differently from other > iterators when unpacking. > Is __len__ a viable option now that __length_hint__ has been identified for hints? IOW, if it's possible to know the full length of RHS ahead of time, and the LHS is also fixed width, then unpack like today else unpack lazily. This does make unpacking behave slightly different if the RHS is `Sized` (per ABC, has __len__), but that doesn't seem too unreasonable. It's different after all. Using Ellipsis, eg. `a, b, ... = it()`, seems like a OK approach too but it's unfortunate we are effectively working around the current force-matching behavior of unpacking... is this the appropriate default? Is there precedence or guidance elsewhere? `a, b, ...` to me says "pull out a and b and throw away the rest", sort of like the spread operator in JS/ECMA. The mere presence of more characters (...) implies something else will *happen* to the remaining items, not that they will be skipped. What about the explicit RHS unpacking idea? Kirill commented on this approach but I'm not sure others did: >>> a, b = iter([1, 2, 3]); a, b (1, 2) >>> a, b = *iter([1, 2, 3]); a, b Traceback (most recent call last): File "", line 1, in ValueError: too many values to unpack (expected 2) This reads very clearly to me that the RHS is expected to be 100% unraveled, and would work with any iterable in the same way. Thanks, -------------- next part -------------- An HTML attachment was scrubbed... URL: From python-ideas at mgmiller.net Wed Nov 29 12:09:11 2017 From: python-ideas at mgmiller.net (Mike Miller) Date: Wed, 29 Nov 2017 09:09:11 -0800 Subject: [Python-ideas] Logging Levels In-Reply-To: References: <35577b73-0943-73f0-931e-a27109646d87@mgmiller.net> Message-ID: <42f2fbdd-2d34-500b-ac58-b9b2f0eb354f@mgmiller.net> Hi, As mentioned, using logging everywhere has some advantages. It doesn't have to be added later on, and it avoided a decent fraction of the work porting to Python 3. Have found users tend to like the labeling of messages, from INFO to ERROR. We skip the time stamp on the console and send to stdout. And with a new-file template, it isn't a burden to set it and argparse up in new scripts. (--verbose etc) -Mike On 2017-11-28 20:45, Nick Coghlan wrote: > On 29 November 2017 at 06:46, Mike Miller wrote: >> Hi, the reason I use note is that I want it to be output by default. So it >> must be above warning, or perhaps the default level changed. > > If the message to be displayed is part of the actual UX of a command > line tool, our advice is "You don't want the logging module, you want > the print() builtin": > https://docs.python.org/3/howto/logging.html#when-to-use-logging > > As a relevant technical detail, it's also worth noting that the > default handler emits messages on stderr, while CLI UX messages should > generally be displayed on stdout. > > Cheers, > Nick. > From asen.bozhilov at gmail.com Wed Nov 29 12:30:54 2017 From: asen.bozhilov at gmail.com (Asen Bozhilov) Date: Wed, 29 Nov 2017 19:30:54 +0200 Subject: [Python-ideas] Immutable dictionaries Message-ID: This is my first post here. I have strong experience with JavaScript and I'm lucky that I could move forward to Python. What I miss in Python are immutable dictionaries. They are especially useful for configurations and call functions which expect dictionary as argument. In my opinion they would let a place for underlying optimizations. I'd like to propose also literaling syntax for immutable dictionaries. immutable_dict = ( 'key1' : 'value1', 'key2' : 'value2' ) This syntax is not ambiguous with expression statements and tuple literals, but it requires a bit lookahed during the parsing. I would appreciate your opinions on the topic. Most interesting for me is why they are not already part of the language? Kind regards, Asen Bozhilov -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Wed Nov 29 12:40:21 2017 From: mertz at gnosis.cx (David Mertz) Date: Wed, 29 Nov 2017 09:40:21 -0800 Subject: [Python-ideas] [Python-Dev] What's the status of PEP 505: None-aware operators? In-Reply-To: <1511974989.903268.1188352080.36BF6ADC@webmail.messagingengine.com> References: <28D91255-56A9-4CC2-B45D-F83ECD715544@langa.pl> <50C74ECC-D462-4FAC-8A9C-A12BC939ADEB@gmail.com> <1511974989.903268.1188352080.36BF6ADC@webmail.messagingengine.com> Message-ID: I like much of the thinking in Random's approach. But I still think None isn't quite special enough to warrant it's own syntax. However, his '(or None: name.strip()[4:].upper())' makes me realize that what is being asked in all the '?(', '?.', '?[' syntax ideas is a kind of ternary expression. Except the ternary isn't based on whether a predicate holds, but rather on whether an exception occurs (AttributeError, KeyError, TypeError). And the fallback in the ternary is always None rather than being general. I think we could generalize this to get something both more Pythonic and more flexible. E.g.: val = name.strip()[4:].upper() except None This would just be catching all errors, which is perhaps too broad. But it *would* allow a fallback other than None: val = name.strip()[4:].upper() except -1 I think some syntax could be possible to only "catch" some exceptions and let others propagate. Maybe: val = name.strip()[4:].upper() except (AttributeError, KeyError): -1 I don't really like throwing a colon in an expression though. Perhaps some other word or symbol could work instead. How does this read: val = name.strip()[4:].upper() except -1 in (AttributeError, KeyError) Where the 'in' clause at the end would be optional, and default to 'Exception'. I'll note that what this idea DOES NOT get us is: val = timeout ?? local_timeout ?? global_timeout Those values that are "possibly None" don't raise exceptions, so they wouldn't apply to this syntax. Yours, David... On Wed, Nov 29, 2017 at 9:03 AM, Random832 wrote: > On Tue, Nov 28, 2017, at 15:31, Raymond Hettinger wrote: > > > > > I also cc python-dev to see if anybody here is strongly in favor or > against this inclusion. > > > > Put me down for a strong -1. The proposal would occasionally save a few > > keystokes but comes at the expense of giving Python a more Perlish look > > and a more arcane feel. > > > > One of the things I like about Python is that I can walk non-programmers > > through the code and explain what it does. The examples in PEP 505 look > > like a step in the wrong direction. They don't "look like Python" and > > make me feel like I have to decrypt the code to figure-out what it does. > > > > timeout ?? local_timeout ?? global_timeout > > 'foo' in (None ?? ['foo', 'bar']) > > requested_quantity ?? default_quantity * price > > name?.strip()[4:].upper() > > user?.first_name.upper() > > Since we're looking at different syntax for the ?? operator, I have a > suggestion for the ?. operator - and related ?[] and ?() that appeared > in some of the proposals. How about this approach? > > Something like (or None: ...) as a syntax block in which any operation > [lexically within the expression, not within e.g. called functions, so > it's different from simply catching AttributeError etc, even if that > could be limited to only catching when the operand is None] on None that > is not valid for None will yield None instead. > > This isn't *entirely* equivalent, but offers finer control. > > v = name?.strip()[4:].upper() under the old proposal would be more or > less equivalent to: > > v = name.strip()[4:].upper() if name is not None else None > > Whereas, you could get the same result with: > (or None: name.strip()[4:].upper()) > > Though that would technically be equivalent to these steps: > v = name.strip if name is not None else None > v = v() if v """"" > v = v[4:] """"""" > v = v.upper """"""" > v = v() """"""" > > The compiler could optimize this case since it knows none of the > operations are valid on None. This has the advantage of being explicit > about what scope the modified rules apply to, rather than simply > implicitly being "to the end of the chain of dot/bracket/call operators" > > It could also be extended to apply, without any additional syntax, to > binary operators (result is None if either operand is None) (or None: a > + b), for example, could return None if either a or b is none. > > [I think I proposed this before with the syntax ?(...), the (or None: > ...) is just an idea to make it look more like Python.] > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > mertz%40gnosis.cx > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Wed Nov 29 12:42:09 2017 From: mertz at gnosis.cx (David Mertz) Date: Wed, 29 Nov 2017 09:42:09 -0800 Subject: [Python-ideas] Immutable dictionaries In-Reply-To: References: Message-ID: https://www.python.org/dev/peps/pep-0416/ Also: https://pypi.python.org/pypi/frozendict On Wed, Nov 29, 2017 at 9:30 AM, Asen Bozhilov wrote: > This is my first post here. I have strong experience with JavaScript and > I'm lucky that I could move forward to Python. > What I miss in Python are immutable dictionaries. They are especially > useful for configurations and call functions which expect dictionary as > argument. In my opinion they would let a place for underlying > optimizations. > > I'd like to propose also literaling syntax for immutable dictionaries. > > immutable_dict = ( > 'key1' : 'value1', > 'key2' : 'value2' > ) > > This syntax is not ambiguous with expression statements and tuple > literals, but it requires a bit lookahed during the parsing. > > I would appreciate your opinions on the topic. Most interesting for me is > why they are not already part of the language? > > Kind regards, > Asen Bozhilov > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From random832 at fastmail.com Wed Nov 29 12:43:53 2017 From: random832 at fastmail.com (Random832) Date: Wed, 29 Nov 2017 12:43:53 -0500 Subject: [Python-ideas] Immutable dictionaries In-Reply-To: References: Message-ID: <1511977433.914920.1188405824.6E139835@webmail.messagingengine.com> On Wed, Nov 29, 2017, at 12:30, Asen Bozhilov wrote: > I would appreciate your opinions on the topic. Most interesting for me is > why they are not already part of the language? See the rejection notes at https://www.python.org/dev/peps/pep-0416/ From prometheus235 at gmail.com Wed Nov 29 12:43:34 2017 From: prometheus235 at gmail.com (Nick Timkovich) Date: Wed, 29 Nov 2017 11:43:34 -0600 Subject: [Python-ideas] Add a dict with the attribute access capability In-Reply-To: References: Message-ID: On Wed, Nov 29, 2017 at 7:21 AM, Daniel Moisset wrote: > As reference of prior art, there is https://pypi.python.org/pypi/munch in > PyPI > Box is also popular as it permits deeper nesting: https://pypi.python.org/pypi/python-box/ https://github.com/cdgriffith/Box Addict is similar: https://pypi.python.org/pypi/addict/2.1.1 https://github.com/mewwts/addict and I've seen AttrDict a few times in code I maintain: https://pypi.python.org/pypi/attrdict https://github.com/bcj/AttrDict With a cursory search, also found: * DotMap: https://pypi.python.org/pypi/dotmap https://github.com/drgrib/dotmap * EasyDict https://pypi.python.org/pypi/easydict https://github.com/makinacorpus/easydict * Treedict (older) https://pypi.python.org/pypi/treedict * Bunch (older) https://pypi.python.org/pypi/bunch ...I spy a theme :P Haven't dug into them that much, but I'd try to answer most questions with how they do it. I'm not sure if the recursive 'items as attributes within items as attributes...' feature is a can of worms or if it's something like defaultdict(dict), but with more levels. Getting deeply nested items with dots seems to be the usual use (i.e. complicated JSONs), and I think the degree of laziness in creating/setting nested items is variable. Nick -------------- next part -------------- An HTML attachment was scrubbed... URL: From python-ideas at mgmiller.net Wed Nov 29 12:51:13 2017 From: python-ideas at mgmiller.net (Mike Miller) Date: Wed, 29 Nov 2017 09:51:13 -0800 Subject: [Python-ideas] Logging Levels In-Reply-To: References: Message-ID: Hi, IMHO the prior decision(s) are too conservative. Reading the bugs, we can see lots of folks reinventing the wheel with common use cases for no good reason. I also gave examples in the log4j, docs, and web apps world that these levels are recognized needs. An addition would represent a very tiny fractional increase in the complexity of the logging module, from ~6 to ~8 in one small corner. It's not like we're adding human expressions to cats and piles of poo here are we? ;-) The builtin java logging suffers from backwards compatibility with an awkward initial design. log4j doesn't (nor we) have that problem, but it includes some cases suggested. In short, have been using note and fatal for years productively. Not a strong proponent of trace and have survived without it so far, but if it were there I'd have used it a few times over the years. On 2017-11-28 12:17, Serhiy Storchaka wrote: > 28.11.17 21:45, Guido van Rossum ????: >> These look like good improvements. I think you should make an issue on >> bugs.python.org describing your proposal and if you >> can submit a PR that implements it. > > See https://bugs.python.org/issue31732 > > It was discussed and rejected. Citing Raymond: "Overall, this seems rehash and > second guess the discussions and decisions made 15 years ago when PEP 282 was > accepted." > > The set of logging levels is not closed. The user can extend it to cover more > specialized uses by logging.addLevelName(). There are disadvantages of having > too much standard names for logging levels (as we can see in Java). From eric at trueblade.com Wed Nov 29 12:55:07 2017 From: eric at trueblade.com (Eric V. Smith) Date: Wed, 29 Nov 2017 12:55:07 -0500 Subject: [Python-ideas] [Python-Dev] What's the status of PEP 505: None-aware operators? In-Reply-To: References: <28D91255-56A9-4CC2-B45D-F83ECD715544@langa.pl> <50C74ECC-D462-4FAC-8A9C-A12BC939ADEB@gmail.com> <1511974989.903268.1188352080.36BF6ADC@webmail.messagingengine.com> Message-ID: > On Nov 29, 2017, at 12:40 PM, David Mertz wrote: > > I like much of the thinking in Random's approach. But I still think None isn't quite special enough to warrant it's own syntax. > > However, his '(or None: name.strip()[4:].upper())' makes me realize that what is being asked in all the '?(', '?.', '?[' syntax ideas is a kind of ternary expression. Except the ternary isn't based on whether a predicate holds, but rather on whether an exception occurs (AttributeError, KeyError, TypeError). And the fallback in the ternary is always None rather than being general. > > I think we could generalize this to get something both more Pythonic and more flexible. E.g.: > > val = name.strip()[4:].upper() except None > > This would just be catching all errors, which is perhaps too broad. But it *would* allow a fallback other than None: > > val = name.strip()[4:].upper() except -1 > > I think some syntax could be possible to only "catch" some exceptions and let others propagate. Maybe: > > val = name.strip()[4:].upper() except (AttributeError, KeyError): -1 > > I don't really like throwing a colon in an expression though. Perhaps some other word or symbol could work instead. How does this read: > > val = name.strip()[4:].upper() except -1 in (AttributeError, KeyError) > > Where the 'in' clause at the end would be optional, and default to 'Exception'. > > I'll note that what this idea DOES NOT get us is: > > val = timeout ?? local_timeout ?? global_timeout > > Those values that are "possibly None" don't raise exceptions, so they wouldn't apply to this syntax. See the rejected PEP 463 for Exception catching expressions. Eric. > > Yours, David... > > >> On Wed, Nov 29, 2017 at 9:03 AM, Random832 wrote: >> On Tue, Nov 28, 2017, at 15:31, Raymond Hettinger wrote: >> > >> > > I also cc python-dev to see if anybody here is strongly in favor or against this inclusion. >> > >> > Put me down for a strong -1. The proposal would occasionally save a few >> > keystokes but comes at the expense of giving Python a more Perlish look >> > and a more arcane feel. >> > >> > One of the things I like about Python is that I can walk non-programmers >> > through the code and explain what it does. The examples in PEP 505 look >> > like a step in the wrong direction. They don't "look like Python" and >> > make me feel like I have to decrypt the code to figure-out what it does. >> > >> > timeout ?? local_timeout ?? global_timeout >> > 'foo' in (None ?? ['foo', 'bar']) >> > requested_quantity ?? default_quantity * price >> > name?.strip()[4:].upper() >> > user?.first_name.upper() >> >> Since we're looking at different syntax for the ?? operator, I have a >> suggestion for the ?. operator - and related ?[] and ?() that appeared >> in some of the proposals. How about this approach? >> >> Something like (or None: ...) as a syntax block in which any operation >> [lexically within the expression, not within e.g. called functions, so >> it's different from simply catching AttributeError etc, even if that >> could be limited to only catching when the operand is None] on None that >> is not valid for None will yield None instead. >> >> This isn't *entirely* equivalent, but offers finer control. >> >> v = name?.strip()[4:].upper() under the old proposal would be more or >> less equivalent to: >> >> v = name.strip()[4:].upper() if name is not None else None >> >> Whereas, you could get the same result with: >> (or None: name.strip()[4:].upper()) >> >> Though that would technically be equivalent to these steps: >> v = name.strip if name is not None else None >> v = v() if v """"" >> v = v[4:] """"""" >> v = v.upper """"""" >> v = v() """"""" >> >> The compiler could optimize this case since it knows none of the >> operations are valid on None. This has the advantage of being explicit >> about what scope the modified rules apply to, rather than simply >> implicitly being "to the end of the chain of dot/bracket/call operators" >> >> It could also be extended to apply, without any additional syntax, to >> binary operators (result is None if either operand is None) (or None: a >> + b), for example, could return None if either a or b is none. >> >> [I think I proposed this before with the syntax ?(...), the (or None: >> ...) is just an idea to make it look more like Python.] >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: https://mail.python.org/mailman/options/python-dev/mertz%40gnosis.cx > > > > -- > Keeping medicines from the bloodstreams of the sick; food > from the bellies of the hungry; books from the hands of the > uneducated; technology from the underdeveloped; and putting > advocates of freedom in prisons. Intellectual property is > to the 21st century what the slave trade was to the 16th. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/eric%2Ba-python-dev%40trueblade.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Wed Nov 29 13:02:58 2017 From: barry at python.org (Barry Warsaw) Date: Wed, 29 Nov 2017 13:02:58 -0500 Subject: [Python-ideas] [Python-Dev] What's the status of PEP 505: None-aware operators? In-Reply-To: References: <28D91255-56A9-4CC2-B45D-F83ECD715544@langa.pl> <50C74ECC-D462-4FAC-8A9C-A12BC939ADEB@gmail.com> <1511974989.903268.1188352080.36BF6ADC@webmail.messagingengine.com> Message-ID: <718169B1-C1AE-4FCA-92E7-D4E246096612@python.org> On Nov 29, 2017, at 12:40, David Mertz wrote: > I think some syntax could be possible to only "catch" some exceptions and let others propagate. Maybe: > > val = name.strip()[4:].upper() except (AttributeError, KeyError): -1 > > I don't really like throwing a colon in an expression though. Perhaps some other word or symbol could work instead. How does this read: > > val = name.strip()[4:].upper() except -1 in (AttributeError, KeyError) I don?t know whether I like any of this but I think a more natural spelling would be: val = name.strip()[4:].upper() except (AttributeError, KeyError) as -1 which could devolve into: val = name.strip()[4:].upper() except KeyError as -1 or: val = name.strip()[4:].upper() except KeyError # Implicit `as None` I would *not* add any spelling for an explicit bare-except equivalent. You would have to write: val = name.strip()[4:].upper() except Exception as -1 Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From victor.stinner at gmail.com Wed Nov 29 13:03:41 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 29 Nov 2017 19:03:41 +0100 Subject: [Python-ideas] Immutable dictionaries In-Reply-To: References: Message-ID: 2017-11-29 18:30 GMT+01:00 Asen Bozhilov : > I'd like to propose also literaling syntax for immutable dictionaries. > > immutable_dict = ( > 'key1' : 'value1', > 'key2' : 'value2' > ) Since Python 3.3, you can write: vstinner at apu$ python3 Python 3.6.3 (default, Oct 9 2017, 12:07:10) >>> import types >>> immutable_dict = types.MappingProxyType({"key": "value"}) >>> immutable_dict.pop('key') AttributeError: 'mappingproxy' object has no attribute 'pop' >>> immutable_dict['key'] = 'value2' TypeError: 'mappingproxy' object does not support item assignment >>> immutable_dict['key2'] = 'value3' TypeError: 'mappingproxy' object does not support item assignment Maybe not the ideal syntax, but it already works without having to modify the Python syntax, and it works on Python 3.3 and newer ;-) Victor From turnbull.stephen.fw at u.tsukuba.ac.jp Wed Nov 29 13:24:28 2017 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Thu, 30 Nov 2017 03:24:28 +0900 Subject: [Python-ideas] generator vs iterator etc. (was: How assignment should work with generators?) In-Reply-To: <20171128062238.GQ22248@ando.pearwood.info> References: <20171127234449.GJ22248@ando.pearwood.info> <23068.65037.184603.974838@turnbull.sk.tsukuba.ac.jp> <20171128062238.GQ22248@ando.pearwood.info> Message-ID: <23070.64348.176331.477003@turnbull.sk.tsukuba.ac.jp> Steven D'Aprano writes: > > The two iterators have the same duck-type, the generator is different. > > How is the generator different? My bad, I got the comparison backward. The generator *is* different, but it's because the generator has *extra* public methods, not fewer. Sorry for the noise. From barry at python.org Wed Nov 29 14:07:11 2017 From: barry at python.org (Barry Warsaw) Date: Wed, 29 Nov 2017 14:07:11 -0500 Subject: [Python-ideas] Immutable dictionaries In-Reply-To: References: Message-ID: David Mertz wrote: > https://www.python.org/dev/peps/pep-0416/ PEP 351 (also rejected) is related to this. -Barry From barry at python.org Wed Nov 29 14:11:17 2017 From: barry at python.org (Barry Warsaw) Date: Wed, 29 Nov 2017 14:11:17 -0500 Subject: [Python-ideas] Add a dict with the attribute access capability In-Reply-To: References: Message-ID: Serhiy Storchaka wrote: > In 3.7 I have removed an old-deprecated plistlib.Dict. [1] Actually it > already was deprecated when the plistlib module was added to the regular > stdlib in Python 2.6. > > Raymond noticed that that capability seemed nice to have. So nice in fact that I'm sure I've reimplemented something similar several times. :) > What do you think about reviving this type as general purpose type in > collections or types? Perhaps it can be convenient for working with > JSON, plists, configuration files, databases and in other cases that > need a dict with string keys. > > If reintroduce it, there are open questions. > > 1. The name of the type. > > 2. The location of the type. collections or types? Or other variants? > > 3. How it will collaborate with OrderedDict, defaultdict, etc? > > 4. Should it be a dict subclass, or a mixin, or a proxy? Or add several > types? I also wonder whether PEP 557 dataclasses could provide this except in the opposite direction, e.g. by optionally adding __getitem__ support. -Barry From barry at python.org Wed Nov 29 14:31:10 2017 From: barry at python.org (Barry Warsaw) Date: Wed, 29 Nov 2017 14:31:10 -0500 Subject: [Python-ideas] Repurpose `assert' into a general-purpose check In-Reply-To: References: Message-ID: Nathan Schneider wrote: > I think it would be interesting to investigate how assert statements are > used in the wild. I can think of three kinds of uses: assert statements are also a form of executable documentation. It says something like "I think the internal state of my code must be this way, otherwise I don't really understand what I wrote". Now, maybe you could argue that such use should be enabled unconditionally, but I disagree since usually your understanding of the state of your code is correct, so the additional checks are unnecessary, and people *do* use -O and -OO in practice. And these days it's even more practical to do so, given the `opt-*` level of pyc tagging: % python3 -c "import foo" % python3 -O -c "import foo" % python3 -OO -c "import foo" % ls foo/__pycache__/ __init__.cpython-36.opt-1.pyc __init__.cpython-36.pyc __init__.cpython-36.opt-2.pyc I also wonder how this would interact with pytest's um, 'hijacking' of the assert statement. Cheers, -Barry From mertz at gnosis.cx Wed Nov 29 14:36:36 2017 From: mertz at gnosis.cx (David Mertz) Date: Wed, 29 Nov 2017 11:36:36 -0800 Subject: [Python-ideas] [Python-Dev] What's the status of PEP 505: None-aware operators? In-Reply-To: References: <28D91255-56A9-4CC2-B45D-F83ECD715544@langa.pl> <50C74ECC-D462-4FAC-8A9C-A12BC939ADEB@gmail.com> <1511974989.903268.1188352080.36BF6ADC@webmail.messagingengine.com> Message-ID: True enough. I remember the prior discussion, but didn't look up the PEP number. I know it's hard to revisit these ideas, but occasionally it works, especially if a nicer spelling is found (Barry proposed some a bit different than my ideas) On Nov 29, 2017 9:55 AM, "Eric V. Smith" wrote: On Nov 29, 2017, at 12:40 PM, David Mertz wrote: I like much of the thinking in Random's approach. But I still think None isn't quite special enough to warrant it's own syntax. However, his '(or None: name.strip()[4:].upper())' makes me realize that what is being asked in all the '?(', '?.', '?[' syntax ideas is a kind of ternary expression. Except the ternary isn't based on whether a predicate holds, but rather on whether an exception occurs (AttributeError, KeyError, TypeError). And the fallback in the ternary is always None rather than being general. I think we could generalize this to get something both more Pythonic and more flexible. E.g.: val = name.strip()[4:].upper() except None This would just be catching all errors, which is perhaps too broad. But it *would* allow a fallback other than None: val = name.strip()[4:].upper() except -1 I think some syntax could be possible to only "catch" some exceptions and let others propagate. Maybe: val = name.strip()[4:].upper() except (AttributeError, KeyError): -1 I don't really like throwing a colon in an expression though. Perhaps some other word or symbol could work instead. How does this read: val = name.strip()[4:].upper() except -1 in (AttributeError, KeyError) Where the 'in' clause at the end would be optional, and default to 'Exception'. I'll note that what this idea DOES NOT get us is: val = timeout ?? local_timeout ?? global_timeout Those values that are "possibly None" don't raise exceptions, so they wouldn't apply to this syntax. See the rejected PEP 463 for Exception catching expressions. Eric. Yours, David... On Wed, Nov 29, 2017 at 9:03 AM, Random832 wrote: > On Tue, Nov 28, 2017, at 15:31, Raymond Hettinger wrote: > > > > > I also cc python-dev to see if anybody here is strongly in favor or > against this inclusion. > > > > Put me down for a strong -1. The proposal would occasionally save a few > > keystokes but comes at the expense of giving Python a more Perlish look > > and a more arcane feel. > > > > One of the things I like about Python is that I can walk non-programmers > > through the code and explain what it does. The examples in PEP 505 look > > like a step in the wrong direction. They don't "look like Python" and > > make me feel like I have to decrypt the code to figure-out what it does. > > > > timeout ?? local_timeout ?? global_timeout > > 'foo' in (None ?? ['foo', 'bar']) > > requested_quantity ?? default_quantity * price > > name?.strip()[4:].upper() > > user?.first_name.upper() > > Since we're looking at different syntax for the ?? operator, I have a > suggestion for the ?. operator - and related ?[] and ?() that appeared > in some of the proposals. How about this approach? > > Something like (or None: ...) as a syntax block in which any operation > [lexically within the expression, not within e.g. called functions, so > it's different from simply catching AttributeError etc, even if that > could be limited to only catching when the operand is None] on None that > is not valid for None will yield None instead. > > This isn't *entirely* equivalent, but offers finer control. > > v = name?.strip()[4:].upper() under the old proposal would be more or > less equivalent to: > > v = name.strip()[4:].upper() if name is not None else None > > Whereas, you could get the same result with: > (or None: name.strip()[4:].upper()) > > Though that would technically be equivalent to these steps: > v = name.strip if name is not None else None > v = v() if v """"" > v = v[4:] """"""" > v = v.upper """"""" > v = v() """"""" > > The compiler could optimize this case since it knows none of the > operations are valid on None. This has the advantage of being explicit > about what scope the modified rules apply to, rather than simply > implicitly being "to the end of the chain of dot/bracket/call operators" > > It could also be extended to apply, without any additional syntax, to > binary operators (result is None if either operand is None) (or None: a > + b), for example, could return None if either a or b is none. > > [I think I proposed this before with the syntax ?(...), the (or None: > ...) is just an idea to make it look more like Python.] > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/mertz% > 40gnosis.cx > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. _______________________________________________ Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ eric%2Ba-python-dev%40trueblade.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From levkivskyi at gmail.com Wed Nov 29 14:57:21 2017 From: levkivskyi at gmail.com (Ivan Levkivskyi) Date: Wed, 29 Nov 2017 20:57:21 +0100 Subject: [Python-ideas] Add a dict with the attribute access capability In-Reply-To: References: Message-ID: On 29 November 2017 at 20:11, Barry Warsaw wrote: > Serhiy Storchaka wrote: > > In 3.7 I have removed an old-deprecated plistlib.Dict. [1] Actually it > > already was deprecated when the plistlib module was added to the regular > > stdlib in Python 2.6. > > > > Raymond noticed that that capability seemed nice to have. > > So nice in fact that I'm sure I've reimplemented something similar > several times. :) > > > What do you think about reviving this type as general purpose type in > > collections or types? Perhaps it can be convenient for working with > > JSON, plists, configuration files, databases and in other cases that > > need a dict with string keys. > > > > If reintroduce it, there are open questions. > > > > 1. The name of the type. > > > > 2. The location of the type. collections or types? Or other variants? > > > > 3. How it will collaborate with OrderedDict, defaultdict, etc? > > > > 4. Should it be a dict subclass, or a mixin, or a proxy? Or add several > > types? > > I also wonder whether PEP 557 dataclasses could provide this except in > the opposite direction, e.g. by optionally adding __getitem__ support. > This was discussed in https://github.com/ericvsmith/dataclasses/issues/21 and it was decided to postpone this. -- Ivan -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Wed Nov 29 16:27:34 2017 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 30 Nov 2017 10:27:34 +1300 Subject: [Python-ideas] [Python-Dev] What's the status of PEP 505: None-aware operators? In-Reply-To: References: <28D91255-56A9-4CC2-B45D-F83ECD715544@langa.pl> <50C74ECC-D462-4FAC-8A9C-A12BC939ADEB@gmail.com> <20171129060843.GZ22248@ando.pearwood.info> <20171129094527.GA22248@ando.pearwood.info> Message-ID: <5A1F2646.1080701@canterbury.ac.nz> Nick Coghlan wrote: >>What about more English-like syntax: >> >>X or else Y > > The problem with constructs like this is that they look like they > should mean the same thing as "X or Y". How about: x otherwise y It looks different enough from "or" that you're not going to accidentally read it that way when skimming. The meaning is something you'll have to learn if you don't know, but at least it's a word that can be googled for. -- Greg From greg.ewing at canterbury.ac.nz Wed Nov 29 16:29:18 2017 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 30 Nov 2017 10:29:18 +1300 Subject: [Python-ideas] [Python-Dev] What's the status of PEP 505: None-aware operators? In-Reply-To: References: <28D91255-56A9-4CC2-B45D-F83ECD715544@langa.pl> <50C74ECC-D462-4FAC-8A9C-A12BC939ADEB@gmail.com> <20171129060843.GZ22248@ando.pearwood.info> <20171129094527.GA22248@ando.pearwood.info> Message-ID: <5A1F26AE.3040307@canterbury.ac.nz> Stephan Houben wrote: > X or else Y Sounds like a Python dialect for Mafia use. customer.repay(loan) or else apply(baseball_bat, customer.kneecaps) -- Greg From greg.ewing at canterbury.ac.nz Wed Nov 29 16:34:35 2017 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 30 Nov 2017 10:34:35 +1300 Subject: [Python-ideas] [Python-Dev] What's the status of PEP 505: None-aware operators? In-Reply-To: References: <28D91255-56A9-4CC2-B45D-F83ECD715544@langa.pl> <50C74ECC-D462-4FAC-8A9C-A12BC939ADEB@gmail.com> <20171129060843.GZ22248@ando.pearwood.info> <20171129094527.GA22248@ando.pearwood.info> Message-ID: <5A1F27EB.3000604@canterbury.ac.nz> Stephan Houben wrote: > A if is not None else C Reading that gives me the feeling that something in my brain has slipped a tooth. It would read better with some kind of pronoun in there: A if it is not None else C Hypercard's Hypertalk had a special variable "it" that worked sort of like that. -- Greg From greg.ewing at canterbury.ac.nz Wed Nov 29 16:49:19 2017 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 30 Nov 2017 10:49:19 +1300 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: References: <20171127135502.GF22248@ando.pearwood.info> <20171127153550.GI22248@ando.pearwood.info> <5A1C8879.2050105@canterbury.ac.nz> Message-ID: <5A1F2B5F.9010603@canterbury.ac.nz> C Anthony Risinger wrote: > Is __len__ a viable option now that __length_hint__ has been identified > for hints? No, I misremembered that feature, sorry. But I still don't like the idea of changing behaviour depending on whether the RHS "looks like" an iterator or not. I'm not sure how to explain why I feel that way, but I think it's because the proposal would make the behaviour depend on the presence or otherwise of a method that's not actually used by the operation being performed. Unpacking doesn't need to know the length in advance, so it shouldn't care whether there is a __len__ method. -- Greg From stephanh42 at gmail.com Wed Nov 29 16:56:08 2017 From: stephanh42 at gmail.com (Stephan Houben) Date: Wed, 29 Nov 2017 22:56:08 +0100 Subject: [Python-ideas] [Python-Dev] What's the status of PEP 505: None-aware operators? In-Reply-To: <5A1F27EB.3000604@canterbury.ac.nz> References: <28D91255-56A9-4CC2-B45D-F83ECD715544@langa.pl> <50C74ECC-D462-4FAC-8A9C-A12BC939ADEB@gmail.com> <20171129060843.GZ22248@ando.pearwood.info> <20171129094527.GA22248@ando.pearwood.info> <5A1F27EB.3000604@canterbury.ac.nz> Message-ID: Op 29 nov. 2017 22:35 schreef "Greg Ewing" : It would read better with some kind of pronoun in there: A if it is not None else C Hypercard's Hypertalk had a special variable "it" that worked sort of like that. I considered that, but there are two issues. 1. Backward-incompatible change 2. The semantics of A if B else C now depends on if B contains 'it', in which case A gets evaluated unconditionally and prior to B. What if evaluation of 'it' is itself conditional, e.g. print("hello") if a or it else print("goodbye") Note that in the A if B else C proposal the evaluation of the implicit 'it' is never conditional in that way. Stephan -- Greg _______________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Wed Nov 29 17:21:48 2017 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 30 Nov 2017 11:21:48 +1300 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: References: <20171127135502.GF22248@ando.pearwood.info> <20171127153550.GI22248@ando.pearwood.info> <5A1C8879.2050105@canterbury.ac.nz> Message-ID: <5A1F32FC.2010502@canterbury.ac.nz> C Anthony Risinger wrote: > `a, b, ...` to me says "pull out a and b and throw away the rest"... > The mere presence of more > characters (...) implies something else will *happen* to the remaining > items, not that they will be skipped. It seems that many people think about unpacking rather differently from the way I do. I think the difference is procedural vs. declarative. To my way of thinking, something like a, b, c = x is a pattern-matching operation. It's declaring that x is a sequence of three things, and giving names to those things. It's not saying to *do* anything to x. With that interpretation, a, b, ... = x is declaring that x is a sequence of at least two items, and giving names to the first two. The ellipsis just means that there could be more items, but we don't want to give them names. On the other hand, some people seem to be interpreting the word "unpack" as in "unpack a suitcase", i.e. the suitcase is empty afterwards. But unpacking has never meant that in Python! If it did, then x = [1, 2, 3] a, b, c = x would leave x == [] afterwards. The only case where unpacking behaves like that is when the rhs is an iterator rather than a sequence, in which case a side effect is unavoidable. The question then is what the side effect should be. I would argue that, since the side effect is something that's not really wanted, it should be as *small* as possible. By that argument, a, b, ... = some_iterator should do as *little* as possible to fulfill what's being asked, i.e. give names to the first two items produced by the rhs. Consuming those two items is unavoidable, but there's no need to consume any more. As for the "extra syntax", we only need it because we've defined the existing unpacking syntax to imply verifying that the rhs has exactly the same length as the pattern. We now want to express patterns which don't impose a length constraint, so we need to write them some other way. -- Greg From rosuav at gmail.com Wed Nov 29 17:40:14 2017 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 30 Nov 2017 09:40:14 +1100 Subject: [Python-ideas] [Python-Dev] What's the status of PEP 505: None-aware operators? In-Reply-To: References: <28D91255-56A9-4CC2-B45D-F83ECD715544@langa.pl> <50C74ECC-D462-4FAC-8A9C-A12BC939ADEB@gmail.com> <20171129060843.GZ22248@ando.pearwood.info> <20171129094527.GA22248@ando.pearwood.info> <5A1F27EB.3000604@canterbury.ac.nz> Message-ID: On Thu, Nov 30, 2017 at 8:56 AM, Stephan Houben wrote: > > > Op 29 nov. 2017 22:35 schreef "Greg Ewing" : > > > It would read better with some kind of pronoun in there: > > A if it is not None else C > > Hypercard's Hypertalk had a special variable "it" that > worked sort of like that. > > > I considered that, but there are two issues. > > 1. Backward-incompatible change > > 2. The semantics of > A if B else C > now depends on if B contains 'it', > in which case A gets evaluated unconditionally and prior to B. > What if evaluation of 'it' is itself conditional, e.g. > > print("hello") if a or it else print("goodbye") > > Note that in the > > A if B else C > > proposal the evaluation of the implicit 'it' is never conditional in that > way. The only way would be for "it" to be a keyword. You can't misinterpret "A if pass is not None else C", or "A if def is not None else C", as they're currently syntax errors. Obviously we don't want the word "it" to become a keyword (massive breakage right there), and I'm pretty sure there's no existing keyword that actually makes sense here, so this proposal would depend on coming up with a useful word that wouldn't collide too much. I'm -0.9 unless someone can find the perfect word. ChrisA From encukou at gmail.com Wed Nov 29 17:43:33 2017 From: encukou at gmail.com (Petr Viktorin) Date: Wed, 29 Nov 2017 23:43:33 +0100 Subject: [Python-ideas] Repurpose `assert' into a general-purpose check In-Reply-To: References: Message-ID: <315b39f2-e6a6-d331-e0a9-de5fb9c9289e@gmail.com> On 11/29/2017 08:31 PM, Barry Warsaw wrote: > > Nathan Schneider wrote: > >> I think it would be interesting to investigate how assert statements are >> used in the wild. I can think of three kinds of uses: > > assert statements are also a form of executable documentation. It says > something like "I think the internal state of my code must be this way, > otherwise I don't really understand what I wrote". > > Now, maybe you could argue that such use should be enabled > unconditionally, but I disagree since usually your understanding of the > state of your code is correct, so the additional checks are unnecessary, > and people *do* use -O and -OO in practice. And these days it's even > more practical to do so, given the `opt-*` level of pyc tagging: > > % python3 -c "import foo" > % python3 -O -c "import foo" > % python3 -OO -c "import foo" > % ls foo/__pycache__/ > __init__.cpython-36.opt-1.pyc __init__.cpython-36.pyc > __init__.cpython-36.opt-2.pyc > > I also wonder how this would interact with pytest's um, 'hijacking' of > the assert statement. Pytest compiles from source, and uses its own cache tags which look like ".cpython-36-PYTEST.pyc". https://github.com/pytest-dev/pytest/blob/master/_pytest/assertion/rewrite.py From steve at pearwood.info Wed Nov 29 17:52:02 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 30 Nov 2017 09:52:02 +1100 Subject: [Python-ideas] [Python-Dev] What's the status of PEP 505: None-aware operators? In-Reply-To: References: <28D91255-56A9-4CC2-B45D-F83ECD715544@langa.pl> <50C74ECC-D462-4FAC-8A9C-A12BC939ADEB@gmail.com> <20171129060843.GZ22248@ando.pearwood.info> Message-ID: <20171129225201.GB22248@ando.pearwood.info> On Wed, Nov 29, 2017 at 06:49:14AM -0800, David Mertz wrote: > I didn't say no use case exists, but rather "too special case." And/or > shortcutting is great, but truthiness feels much more general than noneness > to me. Of course truthiness is much more general than Noneness -- and that's precisely the problem with using `or`. It's too general. If you haven't read the PEP, you should, because this is extensively discussed there. One of the problems the author discusses is that `or` is recommended on Stackoverflow as the best way to perform short-cutting None tests, often with no caveats given. We've been here before. Python old-timers may remember the former idiomatic way to get a ternary if expression: condition and first or second which was also buggy (if first happens to be falsey, second is returned even when condition is true). -- Steve From python at mrabarnett.plus.com Wed Nov 29 18:05:47 2017 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 29 Nov 2017 23:05:47 +0000 Subject: [Python-ideas] [Python-Dev] What's the status of PEP 505: None-aware operators? In-Reply-To: <5A1F2646.1080701@canterbury.ac.nz> References: <28D91255-56A9-4CC2-B45D-F83ECD715544@langa.pl> <50C74ECC-D462-4FAC-8A9C-A12BC939ADEB@gmail.com> <20171129060843.GZ22248@ando.pearwood.info> <20171129094527.GA22248@ando.pearwood.info> <5A1F2646.1080701@canterbury.ac.nz> Message-ID: On 2017-11-29 21:27, Greg Ewing wrote: > Nick Coghlan wrote: > >>>What about more English-like syntax: >>> >>>X or else Y >> >> The problem with constructs like this is that they look like they >> should mean the same thing as "X or Y". > > How about: > > x otherwise y > > It looks different enough from "or" that you're not going > to accidentally read it that way when skimming. > > The meaning is something you'll have to learn if you don't > know, but at least it's a word that can be googled for. > It has the disadvantage that it's quite long. How about "alt" (abbreviation for "alternatively"): x alt y From brenbarn at brenbarn.net Wed Nov 29 18:08:43 2017 From: brenbarn at brenbarn.net (Brendan Barnwell) Date: Wed, 29 Nov 2017 15:08:43 -0800 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: <5A1F32FC.2010502@canterbury.ac.nz> References: <20171127135502.GF22248@ando.pearwood.info> <20171127153550.GI22248@ando.pearwood.info> <5A1C8879.2050105@canterbury.ac.nz> <5A1F32FC.2010502@canterbury.ac.nz> Message-ID: <5A1F3DFB.3070207@brenbarn.net> On 2017-11-29 14:21, Greg Ewing wrote: > C Anthony Risinger wrote: > >> `a, b, ...` to me says "pull out a and b and throw away the rest"... > > The mere presence of more >> characters (...) implies something else will *happen* to the remaining >> items, not that they will be skipped. > > It seems that many people think about unpacking rather > differently from the way I do. I think the difference > is procedural vs. declarative. > > To my way of thinking, something like > > a, b, c = x > > is a pattern-matching operation. It's declaring that > x is a sequence of three things, and giving names to > those things. It's not saying to *do* anything to x. > > With that interpretation, > > a, b, ... = x > > is declaring that x is a sequence of at least two > items, and giving names to the first two. The ellipsis > just means that there could be more items, but we > don't want to give them names. > > On the other hand, some people seem to be interpreting > the word "unpack" as in "unpack a suitcase", i.e. the > suitcase is empty afterwards. But unpacking has never > meant that in Python! If it did, then > > x = [1, 2, 3] > a, b, c = x > > would leave x == [] afterwards. > > The only case where unpacking behaves like that is when > the rhs is an iterator rather than a sequence, in which > case a side effect is unavoidable. The question then is > what the side effect should be. That's an interesting analysis, but I don't think your view is really the right one. It *is* unpacking a suitcase, it's just that *if necessary* the suitcase is constructed just in time for you to unpack it. In other words, the suitcase is not the list [1, 2, 3], but an iterator over this list. This is the same as the behavior for "for" loops: if you do "for item in [1, 2, 3]", the actual thing you're unrolling is an iterator over the list. In some sense the point of the iterable/iterator distinction is to distinguish suitcases (iterators) from things-that-produce-suitcases-on-demand (iterables). It's just that Python syntax (very nicely) allows us to omit the explicit iter() call. The fact that iteration is taking place is specified by the context; that could be a for loop, or it could be multiple assignment targets, but it's iteration all the same. > I would argue that, since the side effect is something > that's not really wanted, it should be as *small* as > possible. By that argument, > > a, b, ... = some_iterator > > should do as *little* as possible to fulfill what's > being asked, i.e. give names to the first two items > produced by the rhs. Consuming those two items is > unavoidable, but there's no need to consume any more. I see your point, but I think that middle ground doesn't really give the benefits of either. If you expect your suitcase to remain unopened, it's pretty cold comfort to find that someone has opened it and taken only your pants and shoes but left the rest. If the side effect isn't wanted, you really need the RHS to be something that isn't affected (i.e., a re-iterable). It does seem that in some cases you may want the iterator to be exhausted, and in others not, but I don't think it's a good idea to try to "hide" the unpacking by limiting the number of iterations. The important difference is between any irreversible unpacking at all, and none at all. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown From chris.barker at noaa.gov Wed Nov 29 18:34:11 2017 From: chris.barker at noaa.gov (Chris Barker) Date: Wed, 29 Nov 2017 15:34:11 -0800 Subject: [Python-ideas] Add a dict with the attribute access capability In-Reply-To: References: Message-ID: On Wed, Nov 29, 2017 at 12:52 AM, Serhiy Storchaka wrote: > This is a dict subclass which allows to access items as attributes. > > d = plistlib.Dict() > d['a'] = 1 > assert d.a == 1 > d.b = 2 > assert d['b'] == 2 > > What do you think about reviving this type as general purpose type in > collections or types Am I I the only one that thinks this is a "Bad Idea"? For me, it simply confuses even more the question of "is this code or is this data?" -- which is a difficult enough design question in a dynamic language. And the couple of libraries I"ve worked with that do this I liked at first, but grew to find problematic. The key problem is that keys in dicts can be anything immutable, while names have a lot more restrictions on them. And in a case like this, you can't use a key that happens to be the same as an existing dict attribute -- at least not in the same way. In particular, I've struggled with the netCDF4 lib and Colander. I've used netCDF4 more, so I'll talk about that: netCDF4 is a Python wrapper around the netcdf scientific data file format. netcdf has "variables" whiuch hold arrays of data, and attributes (for meta data). The netCDF4 python lib wraps a native netcdf4 variable with a python object that python attributes (names) for each of the netcdf variable attributes. Pretty nifty, really, you can do things like: vel = dataset.variables['velocity'] print("velocity is in units of:", vel.units) kinda cool, but in fact it gets harder to deal with in "real code" as opposed to quicky scripts than if variables has a dict-like attribute: print("velocity is in units of:", vel.atts['units']) Why? because netcdf variables can have arbitrary attributes, so your code tends to be hard-coded for specific data sets, and breaks fast if the data files are not compliant. That's what I mean by the difference between code and data -- the attributes of a netcdf variable is data -- not code. Granted, with Python, you can use try: except AttributeError, and setattr, and all that to work with these attributes as though they are data, but then you lose all the advantages of having them as attributes anyway. setattr and the like shulud be used for when metaprogramign is called for - not for everyday data manipulation. Oh, and it's also uglier to get/set values in bulk (though the proposed dict_with_attributes would present both APIs, so not as restrictive.) You also have problems with keys that mirror actual attributes. So you use the dict interface to deal with those -- but if you have to choose your interface according to what the values of the data are -- you've got the whole, it's not code, it's data! problem again. Anyway -- those are my $0.02 -- from experience working with such things in the real world, with real code, and real data.... -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Wed Nov 29 18:47:09 2017 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 29 Nov 2017 15:47:09 -0800 Subject: [Python-ideas] Add a dict with the attribute access capability In-Reply-To: References: Message-ID: <5A1F46FD.7060509@stoneleaf.us> On 11/29/2017 03:34 PM, Chris Barker wrote: > Am I I the only one that thinks this is a "Bad Idea"? No, you're not. > For me, it simply confuses even more the question of "is this code or is this data?" -- which is a difficult enough > design question in a dynamic language. As one of those who have created their own version of AttrDict, I can say it's both neat and tricky to use. I'm happy to have it* as a third-party package, but I don't think it belongs in the stdlib. -- ~Ethan~ * attribute-access dicts in general, not mine in particular From rob.cliffe at btinternet.com Wed Nov 29 20:03:41 2017 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Thu, 30 Nov 2017 01:03:41 +0000 Subject: [Python-ideas] Add a dict with the attribute access capability In-Reply-To: References: Message-ID: On 29/11/2017 23:34, Chris Barker wrote: > > > On Wed, Nov 29, 2017 at 12:52 AM, Serhiy Storchaka > > wrote: > > This is a dict subclass which allows to access items as attributes. > > d = plistlib.Dict() > d['a'] = 1 > assert d.a == 1 > d.b = 2 > assert d['b'] == 2 > > What do you think about reviving this type as general purpose type > in collections or types > > > Am I I the only one that thinks this is a "Bad Idea"? > > For me, it simply confuses even more the question of "is this code or > is this data?" -- which is a difficult enough design question in a > dynamic language. > > And the couple of libraries I"ve worked with that do this I liked at > first, but grew to find problematic. > > I have also implemented something like this (as a convenience, to hold a row from an SQL table).? But it never occurred to me that there should be something like it in the stdlib. Rob Cliffe -------------- next part -------------- An HTML attachment was scrubbed... URL: From rob.cliffe at btinternet.com Wed Nov 29 20:16:48 2017 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Thu, 30 Nov 2017 01:16:48 +0000 Subject: [Python-ideas] Immutable dictionaries In-Reply-To: References: Message-ID: <4b39e82d-1b97-522e-87c9-0678d5001a59@btinternet.com> On 29/11/2017 17:30, Asen Bozhilov wrote: > This is my first post here. I have strong experience with JavaScript > and I'm lucky that I could move forward to Python. > What I miss in Python are immutable dictionaries. They are especially > useful for configurations and call functions which expect dictionary > as argument.? In my opinion they would let a place for underlying > optimizations. > > I'd like to propose also literaling syntax for immutable dictionaries. > > immutable_dict = ( > ? ? 'key1' : 'value1', > ? ? 'key2' : 'value2' > ) > > This syntax is not ambiguous with expression statements and tuple > literals, but it requires a bit lookahed during the parsing. > > This is sort of a subset of an idea I once posted on Python-Ideas: Dictionaries, sets and lists (etc. ?) could have a mutable flag, and once it was set to False you could neither change it back nor change the object.? (OK there might be some backdoor hacks, this is Python.)? This would avoid having to explain to beginners "Why does Python have lists and tuples?" because instead of a tuple, all you need is an immutable list.? (Or if you prefer, instead of a list, all you need is a mutable tuple.) Rob Cliffe From electronnn at gmail.com Wed Nov 29 21:48:04 2017 From: electronnn at gmail.com (electronnn at gmail.com) Date: Thu, 30 Nov 2017 06:18:04 +0330 Subject: [Python-ideas] [Python-Dev] What's the status of PEP 505: None-aware operators? In-Reply-To: References: <28D91255-56A9-4CC2-B45D-F83ECD715544@langa.pl> <50C74ECC-D462-4FAC-8A9C-A12BC939ADEB@gmail.com> <20171129060843.GZ22248@ando.pearwood.info> <20171129094527.GA22248@ando.pearwood.info> <5A1F2646.1080701@canterbury.ac.nz> Message-ID: I, too, think that this is too special a case for a dedicated syntax but so far I prefer: x None-or y The meaning is more clear to me. None-or is like regular or except that y is evaluated only if x is None. On Thu, Nov 30, 2017 at 2:35 AM, MRAB wrote: > On 2017-11-29 21:27, Greg Ewing wrote: > >> Nick Coghlan wrote: >> >> What about more English-like syntax: >>>> >>>> X or else Y >>>> >>> >>> The problem with constructs like this is that they look like they >>> should mean the same thing as "X or Y". >>> >> >> How about: >> >> x otherwise y >> >> It looks different enough from "or" that you're not going >> to accidentally read it that way when skimming. >> >> The meaning is something you'll have to learn if you don't >> know, but at least it's a word that can be googled for. >> >> It has the disadvantage that it's quite long. How about "alt" > (abbreviation for "alternatively"): > > x alt y > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Wed Nov 29 22:40:13 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 30 Nov 2017 14:40:13 +1100 Subject: [Python-ideas] Add a dict with the attribute access capability In-Reply-To: References: Message-ID: <20171130034012.GC22248@ando.pearwood.info> On Wed, Nov 29, 2017 at 10:52:51AM +0200, Serhiy Storchaka wrote: > In 3.7 I have removed an old-deprecated plistlib.Dict. [1] Actually it > already was deprecated when the plistlib module was added to the regular > stdlib in Python 2.6. > > This is a dict subclass which allows to access items as attributes. Isn't that a trivial subclass? class AttrDict(dict): def __init__(self, *args, **kw): super().__init__(*args, **kw) self.__dict__ = self Possibly apart from a nicer repr, have I missed anything? > Raymond noticed that that capability seemed nice to have. > > What do you think about reviving this type as general purpose type in > collections or types? Perhaps it can be convenient for working with > JSON, plists, configuration files, databases and in other cases that > need a dict with string keys. I doubt that it is as convenient as you think. Most of the time I work with external data, I don't know the keys in advance, so I couldn't use obj.attr syntax. Configuration files (and equivalent) are possibly an exception. But in general I'm suspicious of using attribute access for key lookups. I cannot help feeling that, while laziness and hubris are virtues in programmers, this is *too* lazy. This conflates two distinct concepts, attributes and key/value pairs. Attributes are conceptually part of the object, as in dog.tail or car.engine, or data about the object, like page.top_margin or engine.capacity. Key/value data is not. Another way to look at this would be to say that deleting attributes is rare and unusual, while deleting keys is generally frequent and commonplace. Its not a completely hard distinction, I acknowledge that its already a bit fuzzy, but I'm not sure it is a good idea to make it even fuzzier. Consequently I'm strongly opposed to making this standard behaviour for dict and other mappings, but I'm neutral about making it opt-in for a distinct mapping type. > If reintroduce it, there are open questions. > > 1. The name of the type. AttrDict seems like the obvious name. > 2. The location of the type. collections or types? Or other variants? It is a mapping, which is a collection, so collections. > 3. How it will collaborate with OrderedDict, defaultdict, etc? Does it need to? > 4. Should it be a dict subclass, or a mixin, or a proxy? Or add several > types? Aren't these implementation details? -- Steve From steve at pearwood.info Wed Nov 29 22:55:01 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 30 Nov 2017 14:55:01 +1100 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: <5A1F2B5F.9010603@canterbury.ac.nz> References: <20171127135502.GF22248@ando.pearwood.info> <20171127153550.GI22248@ando.pearwood.info> <5A1C8879.2050105@canterbury.ac.nz> <5A1F2B5F.9010603@canterbury.ac.nz> Message-ID: <20171130035501.GD22248@ando.pearwood.info> On Thu, Nov 30, 2017 at 10:49:19AM +1300, Greg Ewing wrote: > C Anthony Risinger wrote: > >Is __len__ a viable option now that __length_hint__ has been identified > >for hints? > > No, I misremembered that feature, sorry. > > But I still don't like the idea of changing behaviour > depending on whether the RHS "looks like" an iterator > or not. The reason I oppose that is that in all other ways related to iteration, iterators are a perfect substitute for any other iterable: for x in iterable: pass list(iterable) function(*iterable) a, *b, c = iterable a, *b = iterable all behave identically whether iterable is a list or an iterator. This would make the last example, and only that, behave differently depending on whether you pass an iterator or some other iterable. -- Steve From steve at pearwood.info Wed Nov 29 23:04:41 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 30 Nov 2017 15:04:41 +1100 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: <5A1F32FC.2010502@canterbury.ac.nz> References: <20171127135502.GF22248@ando.pearwood.info> <20171127153550.GI22248@ando.pearwood.info> <5A1C8879.2050105@canterbury.ac.nz> <5A1F32FC.2010502@canterbury.ac.nz> Message-ID: <20171130040439.GE22248@ando.pearwood.info> On Thu, Nov 30, 2017 at 11:21:48AM +1300, Greg Ewing wrote: > It seems that many people think about unpacking rather > differently from the way I do. I think the difference > is procedural vs. declarative. > > To my way of thinking, something like > > a, b, c = x > > is a pattern-matching operation. It's declaring that > x is a sequence of three things, and giving names to > those things. It's not saying to *do* anything to x. I hadn't thought of that interpretation before, but now that Greg mentions it, its so obvious-in-hindsight that I completely agree with it. I think that we should promote this as the "one obvious" interpretation. Obviously as a practical matter, there are some x (namely iterators) where you cannot extract items without modifying x, but in all other cases I think that the pattern-matching interpretation is superiour. > With that interpretation, > > a, b, ... = x > > is declaring that x is a sequence of at least two > items, and giving names to the first two. The ellipsis > just means that there could be more items, but we > don't want to give them names. Indeed. > On the other hand, some people seem to be interpreting > the word "unpack" as in "unpack a suitcase", i.e. the > suitcase is empty afterwards. But unpacking has never > meant that in Python! If it did, then > > x = [1, 2, 3] > a, b, c = x > > would leave x == [] afterwards. > > The only case where unpacking behaves like that is when > the rhs is an iterator rather than a sequence, in which > case a side effect is unavoidable. The question then is > what the side effect should be. > > I would argue that, since the side effect is something > that's not really wanted, it should be as *small* as > possible. By that argument, > > a, b, ... = some_iterator > > should do as *little* as possible to fulfill what's > being asked, i.e. give names to the first two items > produced by the rhs. Consuming those two items is > unavoidable, but there's no need to consume any more. I'm still in 100% agreement with all of this. > As for the "extra syntax", we only need it because we've > defined the existing unpacking syntax to imply verifying > that the rhs has exactly the same length as the pattern. > > We now want to express patterns which don't impose > a length constraint, so we need to write them some > other way. To be clear, I'm still luke-warm on the need for this, I think islice is adequate, but if others believe this is a use-case important enough for a syntactic solution, I've come around to making ... my prefered syntax. -- Steve From steve at pearwood.info Wed Nov 29 23:43:49 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 30 Nov 2017 15:43:49 +1100 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: <5A1F3DFB.3070207@brenbarn.net> References: <20171127135502.GF22248@ando.pearwood.info> <20171127153550.GI22248@ando.pearwood.info> <5A1C8879.2050105@canterbury.ac.nz> <5A1F32FC.2010502@canterbury.ac.nz> <5A1F3DFB.3070207@brenbarn.net> Message-ID: <20171130044348.GF22248@ando.pearwood.info> On Wed, Nov 29, 2017 at 03:08:43PM -0800, Brendan Barnwell wrote: > On 2017-11-29 14:21, Greg Ewing wrote: > >On the other hand, some people seem to be interpreting > >the word "unpack" as in "unpack a suitcase", i.e. the > >suitcase is empty afterwards. But unpacking has never > >meant that in Python! If it did, then > > > > x = [1, 2, 3] > > a, b, c = x > > > >would leave x == [] afterwards. > > > >The only case where unpacking behaves like that is when > >the rhs is an iterator rather than a sequence, in which > >case a side effect is unavoidable. The question then is > >what the side effect should be. > > That's an interesting analysis, but I don't think your view is > really the right one. It *is* unpacking a suitcase, it's just that *if > necessary* the suitcase is constructed just in time for you to unpack > it. In other words, the suitcase is not the list [1, 2, 3], but an > iterator over this list. At the point that you are conjuring from thin air an invisible suitcase that is an exact clone of the original suitcase, in order to unpack the clone without disturbing the original, I think the metaphor is so far from the real-world unpacking of suitcases that it no longer applies. Besides, it's not even correct to say that an invisible suitcase (iterator) is constructed. # Python 3.5 py> dis.dis("a, b, c = [97, 98, x]") 1 0 LOAD_CONST 0 (97) 3 LOAD_CONST 1 (98) 6 LOAD_NAME 0 (x) 9 ROT_THREE 10 ROT_TWO 11 STORE_NAME 1 (a) 14 STORE_NAME 2 (b) 17 STORE_NAME 3 (c) 20 LOAD_CONST 2 (None) 23 RETURN_VALUE Before iterators existed, Python had sequence unpacking going back to at least Python 1.5 if not older, so even if Python used a temporary and invisible iterator *now* that has not always been the case and it might not be the case in the future or in alternate interpreters. Even if Python *sometimes* builds a temporary and invisible iterator, it doesn't do it all the time, and when it does, it is pure implementation, not interface. The interpreter is free to do whatever it likes behind the scenes, but there's no iterator involved in the high-level Python statement: a, b, c = [1, 2, 3] That code involves a list and three assignment targets, that is all. > This is the same as the behavior for "for" > loops: if you do "for item in [1, 2, 3]", the actual thing you're > unrolling is an iterator over the list. No, the actual thing *I* am unrolling is exactly what I write in my code, which is the list [1, 2, 3]. I don't care what the Python interpreter iterates over internally, so long as the results are the same. It can use an iterator, or unroll the loop at compile-time, or turn it into recursion over a linked list for all I care. As much as possible, we should avoid thinking about implementation details when trying to understand high-level semantics of Python code. Otherwise, our mental models become obsolete when the implementation changes. Or worse, we become incapable of thinking about better implementations (or better high-level semantics!) because the current implementation is so entwined with our understanding of the high-level semantics of the code. If we had allowed the old sequence protocol implementation to take priority over the abstract, high-level concept of iteration, we wouldn't have iterators today. -- Steve From brenbarn at brenbarn.net Thu Nov 30 00:25:25 2017 From: brenbarn at brenbarn.net (Brendan Barnwell) Date: Wed, 29 Nov 2017 21:25:25 -0800 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: <20171130044348.GF22248@ando.pearwood.info> References: <20171127135502.GF22248@ando.pearwood.info> <20171127153550.GI22248@ando.pearwood.info> <5A1C8879.2050105@canterbury.ac.nz> <5A1F32FC.2010502@canterbury.ac.nz> <5A1F3DFB.3070207@brenbarn.net> <20171130044348.GF22248@ando.pearwood.info> Message-ID: <5A1F9645.5040809@brenbarn.net> On 2017-11-29 20:43, Steven D'Aprano wrote: > At the point that you are conjuring from thin air an invisible suitcase > that is an exact clone of the original suitcase, in order to unpack the > clone without disturbing the original, I think the metaphor is so far > from the real-world unpacking of suitcases that it no longer applies. It is not an exact clone of the original suitcase, because the original suitcase is a collection with stable contents (i.e., cannot be exhausted), but the "clone" (the iterator) CAN be exhausted. It iterates over the same *values*, but that doesn't mean it's the same thing. > Besides, it's not even correct to say that an invisible suitcase > (iterator) is constructed. > > > # Python 3.5 > py> dis.dis("a, b, c = [97, 98, x]") > 1 0 LOAD_CONST 0 (97) > 3 LOAD_CONST 1 (98) > 6 LOAD_NAME 0 (x) > 9 ROT_THREE > 10 ROT_TWO > 11 STORE_NAME 1 (a) > 14 STORE_NAME 2 (b) > 17 STORE_NAME 3 (c) > 20 LOAD_CONST 2 (None) > 23 RETURN_VALUE > > > > Before iterators existed, Python had sequence unpacking going back to at > least Python 1.5 if not older, so even if Python used a temporary and > invisible iterator *now* that has not always been the case and it might > not be the case in the future or in alternate interpreters. > > Even if Python *sometimes* builds a temporary and invisible iterator, it > doesn't do it all the time, and when it does, it is pure implementation, > not interface. The interpreter is free to do whatever it likes behind > the scenes, but there's no iterator involved in the high-level Python > statement: > > a, b, c = [1, 2, 3] > > That code involves a list and three assignment targets, that is all. The code only has a list and three assignment targets, but that doesn't mean that that's all it "involves". The expression "a + b" only has two variables and a plus sign, but it involves a call to __add__ which is not explicitly represented. Things like this aren't implementation details. Indeed, they're precisely the opposite: they are a high level specification of an API for how syntax is converted into semantics. > >> This is the same as the behavior for "for" >> loops: if you do "for item in [1, 2, 3]", the actual thing you're >> unrolling is an iterator over the list. > > No, the actual thing *I* am unrolling is exactly what I write in my > code, which is the list [1, 2, 3]. > > I don't care what the Python interpreter iterates over internally, so > long as the results are the same. It can use an iterator, or unroll the > loop at compile-time, or turn it into recursion over a linked list for > all I care. > > As much as possible, we should avoid thinking about implementation > details when trying to understand high-level semantics of Python code. > Otherwise, our mental models become obsolete when the implementation > changes. Don't you see a bit of irony in arguing based on the compiled bytecode, and then saying you don't care about implementation details? :-) Here is a simpler example: class Foo(object): def __iter__(self): print("You tried to call iter on me!") >>> a, b, c = Foo() You tried to call iter on me! Traceback (most recent call last): File "", line 1, in a, b, c = Foo() TypeError: iter() returned non-iterator of type 'NoneType' You can see that iter() is called, even though "exactly what I wrote in the code" is not iter(Foo()) but just Foo(). The "implementation detail" is that this function call is concealed within a bytecode called "UNPACK_SEQUENCE". Another implementation detail is that in your example that bytecode not used, but that's only because you decompiled an expression with a literal list. If you do "x = [1, 2, 3]" and then decompile "a, b, c = x", you will see the UNPACK_SEQUENCE bytecode. These details of the bytecode are implementation details. What is not an implementation detail is the iteration protocol, which is exactly the kind of high-level semantic thing you're talking about. The iteration protocol says that when you go to iterate over something, iter() is called on it, and then next() is called on the result of that call. Because of this, I am loath to pretend that whether "a, b, c = x" is "like unpacking a suitcase" depends on whether x happens to be a list, some other iterable, or some other iterator. The end result in all cases is that the thing that actually doles out the items is an iterator. Sometimes that iterator is connected to a stable base (some kind of collection) that can be re-iterated; sometimes it isn't. But that doesn't change the iteration protocol. The interpreter is not free to do what it likes behind the scenes; an implementation that did not call __iter__ in the above case would be errroneous. __iter__ is part of the interface of any type that defines it. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown From greg.ewing at canterbury.ac.nz Thu Nov 30 00:32:02 2017 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 30 Nov 2017 18:32:02 +1300 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: <5A1F3DFB.3070207@brenbarn.net> References: <20171127135502.GF22248@ando.pearwood.info> <20171127153550.GI22248@ando.pearwood.info> <5A1C8879.2050105@canterbury.ac.nz> <5A1F32FC.2010502@canterbury.ac.nz> <5A1F3DFB.3070207@brenbarn.net> Message-ID: <5A1F97D2.5090200@canterbury.ac.nz> Brendan Barnwell wrote: > That's an interesting analysis, but I don't think your view is > really the right one. It *is* unpacking a suitcase, it's just that *if > necessary* the suitcase is constructed just in time for you to unpack > it. I don't think that's right. An iterator created from a sequence is not a container in its own right, it's something that provides access to a container. It's not a suitcase, it's a view of the contents of a suitcase. > If you expect your suitcase to remain unopened, > it's pretty cold comfort to find that someone has opened it and taken > only your pants and shoes but left the rest. So you think it's somehow better if he takes *all* of your clothes instead? Here's another way to think about it. If it's acceptable to exhaust the iterator when only the first few items are requested, then you're planning to throw the iterator away afterwards. In that case, what purpose is served by extracting the rest of the items? Remember, the ellipsis explicitly says you don't care how many more items there are. The only reason I can think of is if you're relying on side effects performed by the iterator, which is a pretty obscure way to design code. If the iterator really must be exhausted for some reason, it would be better to be explicit about it, e.g. a, b, ... = spam_iterator for _ in spam_iterator: pass # Mustn't leave any spam in the fridge or it will go bad -- Greg From gadgetsteve at live.co.uk Wed Nov 29 14:33:54 2017 From: gadgetsteve at live.co.uk (Steve Barnes) Date: Wed, 29 Nov 2017 19:33:54 +0000 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: References: Message-ID: On 28/11/2017 23:15, Alon Snir wrote: > On Wed, Nov 29, 2017 at 5:46 AM, Alon Snir wrote: >> I would like to mention that the issue of assignment to a target list, is also relevant to the case of elementwise assignment to a mutable sequence (e.g. lists and arrays). Here as well, the rhs of the assignment statement states the number of elements to be assigned. Consequently, the following example will get into infinite loop: >> >>>>> from itertools import count >>>>> A = [] >>>>> A[:] = count() >> >> Writing "A[:2] = count()" will cause the same result. >> Here as well, it is currently safer to use islice if the rhs is a generator or an iterator. e.g.: >> >>>>> it = count() >>>>> A[:] = islice(it,2) >> >> In my opinion, it is be better to devise a solution that could be applied in both cases. Maybe a new kind of assignment operator that will be dedicated to this kind of assignment. i.e. elementwise assignment with restriction on the number of elements to be assigned, based on the length of the lhs object (or the number of targets in the target list). >> > > ChrisA wrote: > Hmm. The trouble is that slice assignment doesn't have a fixed number of targets. If you say "x, y = spam", there's a clear indication that 'spam' needs to provide exactly two values; but "A[:] = spam" could have any number of values, and it'll expand or shrink the list accordingly. > > Rhodri James wrote: > Flatly, no. It is better not to ask for things you don't want in the first place, in this case the infinite sequence. > Still, don't let me discourage you from working on this. If you can define how such an assignment would work, or even the length of A[:] as an assignment target, I'm not going to dismiss it out of hand. > > > My answer: > > The idea is to define an alternative assignment rule, that is to assign exactly as many elements as the current length of the lhs object (without expanding or shrinking it). Suppose "?=" is the operator for the alternative assignment rule; A=[None]*2; and "iterator" is any iterator (finite or infinite). In this case, the following code: > >>>> A ?= iterator > > would behave like this: > >>>> A[:] = islice(iterator, 2) # where 2 is the length of A > > And as suggested earlier for the case of assignment to a target list, the following code: > >>>> x, y ?= iterator > > would behave like this: > >>>> x, y = islice(iterator, 2) # where 2 is the number of targets > > Regarding the length issue: > Is there any difficulty in finding the length of a sliced sequence? After all, the range object has a len method. Therefore, the length of A[s] (where s is a slice object) could be evaluated as follows: > >>>> len(range(*s.indices(len(A)))) Just a thought but what about a syntax something along the lines of: a, b, *remainder = iterable Where remainder becomes the iterable with the first two values consumed by assigning to a & b. If the iterator has less than 2 values, (in the above case), remaining it should error, if it has exactly 2 then remainder would become an exhausted iterable. Of course the user could also use: a, b, *iterable = iterable Others may differ but this syntax has a lot of similarity to the f(a, b, *args) syntax, possibly enough that most users could understand it. -- Steve (Gadget) Barnes Any opinions in this message are my personal opinions and do not reflect those of my employer. --- This email has been checked for viruses by AVG. http://www.avg.com From stephanh42 at gmail.com Thu Nov 30 06:35:26 2017 From: stephanh42 at gmail.com (Stephan Houben) Date: Thu, 30 Nov 2017 12:35:26 +0100 Subject: [Python-ideas] Immutable dictionaries In-Reply-To: <4b39e82d-1b97-522e-87c9-0678d5001a59@btinternet.com> References: <4b39e82d-1b97-522e-87c9-0678d5001a59@btinternet.com> Message-ID: 2017-11-30 2:16 GMT+01:00 Rob Cliffe : > This is sort of a subset of an idea I once posted on Python-Ideas: > Dictionaries, sets and lists (etc. ?) could have a mutable flag, and once > it was set to False you could neither change it back nor change the > object. (OK there might be some backdoor hacks, this is Python.) This > would avoid having to explain to beginners "Why does Python have lists and > tuples?" because instead of a tuple, all you need is an immutable list. > (Or if you prefer, instead of a list, all you need is a mutable tuple.) > Here is a concept implementation of a freeze() function which does something similar. Rather than mutating the mutable object to become immutable, it makes an immutable copy. >>> freeze({"x": [1,2,{3,4}]}) mappingproxy({'x': (1, 2, frozenset({3, 4}))}) https://gist.github.com/stephanh42/d277170dd8a3a2f026c272a4fda15396 Stephan > Rob Cliffe > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Thu Nov 30 10:50:24 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 1 Dec 2017 02:50:24 +1100 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: References: Message-ID: <20171130155024.GG22248@ando.pearwood.info> On Wed, Nov 29, 2017 at 07:33:54PM +0000, Steve Barnes wrote: > Just a thought but what about a syntax something along the lines of: > > a, b, *remainder = iterable > > Where remainder becomes the iterable with the first two values consumed > by assigning to a & b. Guido's time machine strikes again. That has worked since 3.3 if not older. (Probably 3.0 or so, but I don't have that on this computer to test it.) py> a, b, *remainder = range(10) py> a 0 py> b 1 py> remainder [2, 3, 4, 5, 6, 7, 8, 9] -- Steve From vano at mail.mipt.ru Thu Nov 30 10:56:56 2017 From: vano at mail.mipt.ru (Ivan Pozdeev) Date: Thu, 30 Nov 2017 18:56:56 +0300 Subject: [Python-ideas] Add a dict with the attribute access capability In-Reply-To: References: Message-ID: <3d3c8f2c-0722-1012-6d15-a5706f79ee55@mail.mipt.ru> I use argparse.Namespace whenever I need this. In reply to Chris Barker's concern of "is this code or is this data", the last time I used it is when I needed to hold an external function reference in an object instance (if I assigned it to an attribute, it was converted into an instance method). It just didn't feel right to invoke it via a container lookup. A telling name for the object also clears doubts about what it's supposed to hold. On 29.11.2017 11:52, Serhiy Storchaka wrote: > In 3.7 I have removed an old-deprecated plistlib.Dict. [1] Actually it > already was deprecated when the plistlib module was added to the > regular stdlib in Python 2.6. > > This is a dict subclass which allows to access items as attributes. > > d = plistlib.Dict() > d['a'] = 1 > assert d.a == 1 > d.b = 2 > assert d['b'] == 2 > > Raymond noticed that that capability seemed nice to have. > > What do you think about reviving this type as general purpose type in > collections or types? Perhaps it can be convenient for working with > JSON, plists, configuration files, databases and in other cases that > need a dict with string keys. > > If reintroduce it, there are open questions. > > 1. The name of the type. > > 2. The location of the type. collections or types? Or other variants? > > 3. How it will collaborate with OrderedDict, defaultdict, etc? > > 4. Should it be a dict subclass, or a mixin, or a proxy? Or add > several types? > > > [1] https://bugs.python.org/issue29196 > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -- Regards, Ivan From ronaldoussoren at mac.com Thu Nov 30 10:23:49 2017 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Thu, 30 Nov 2017 16:23:49 +0100 Subject: [Python-ideas] PEP 447: Adding type.__getdescriptor__ Message-ID: <69FE6318-7D46-4EA2-B37D-7488B6CDFCE4@mac.com> Hi, I?m trying to get up to speed again w.r.t. PEP 447 (and to be honest, with CPython?s development workflow as its been too long since I?ve actually done work on the CPython codebase?). Let me start by recapping the PEP (>) is about: The problem I?m trying to solve is that super.__getattribute__ basicy assumes looking at the __dict__ of classes on the MRO is all that?s needed to check if those classes provide an attribute. The core of both object.__getattribute__ and super.__getattribute__ for finding a descriptor on the class is basically (from the PEP): def _PyType_Lookup(tp, name): mro = tp.mro() assert isinstance(mro, tuple) for base in mro: assert isinstance(base, type) try: return base.__dict__[name] except KeyError: pass return None Note that the real implementation in in C, and accesses base.__dict__ directly instead of through Python?s attribute lookup (which would lead to infinite recursion). This is problematic when the class is dynamically populated, as the descriptor may not yet be present in the class __dict__. This is not a problem for normal attribute lookup, as you can replace the __getattribute__ method of the class to do the additional work (at the cost of having to reimplement all of __getattribute__). This is a problem for super() calls though, as super will unconditionally use the lookup code above. The PEP proposed to replace ?base.__dict__[name]? by ?base.__class__.__getdescriptor__(name)? (once again, in C with direct access to the two slots instead of accessing them through normal attribute lookup). I have two open questions about this PEP: 1) Last time around Mark Shannon worried that this introduces infinite recursion in the language itself (in my crummy summary, please read this message to get the real concern >). Is this truly a problem? I don?t think there is a problem, but I?m worried that I don?t fully understand Mark?s concerns. 2) PEP 487 introduced __init_subclass__ as a class method to avoid having to write a metaclass for a number of use cases. My PEP currently does require a metaclass, but it might be nicer to switch to a regular class method instead (like __init_subclass__). My primary usecase for this PEP is PyObjC, which does populate the class dictionary on demand for performance and correctness reasons and PyObC therefore currently includes a replacement for super. For PyObjC?s implementation making __getdescriptor__ a classmethod would be a win as this would avoid creating yet another level of metaclasses in C code (PyObjC already has a Python class and metaclass for every Objective-C class, PEP 447 would require the introduction of a metaclass for those metaclasses). BTW. Two other open issues w.r.t. this PEP are forward porting the patch (in issue 18181) and performing benchmarks. The patch doesn?t apply cleanly to the current tip of the tree, I?ll try to update it tomorrow. And with some luck will manage to update PyObjC?s implementation as well to validate the design. Ronald -------------- next part -------------- An HTML attachment was scrubbed... URL: From vano at mail.mipt.ru Thu Nov 30 12:02:08 2017 From: vano at mail.mipt.ru (Ivan Pozdeev) Date: Thu, 30 Nov 2017 20:02:08 +0300 Subject: [Python-ideas] [Python-Dev] What's the status of PEP 505: None-aware operators? In-Reply-To: <20171129060843.GZ22248@ando.pearwood.info> References: <28D91255-56A9-4CC2-B45D-F83ECD715544@langa.pl> <50C74ECC-D462-4FAC-8A9C-A12BC939ADEB@gmail.com> <20171129060843.GZ22248@ando.pearwood.info> Message-ID: <44f6f218-de45-6b33-d7eb-99b98dea4e35@mail.mipt.ru> On 29.11.2017 9:08, Steven D'Aprano wrote: > On Tue, Nov 28, 2017 at 12:31:06PM -0800, Raymond Hettinger wrote: >>> I also cc python-dev to see if anybody here is strongly in favor or against this inclusion. >> Put me down for a strong -1. The proposal would occasionally save a >> few keystokes but comes at the expense of giving Python a more Perlish >> look and a more arcane feel. > I think that's an unfair characterisation of the benefits of the PEP. > It's not just "a few keystrokes". > > Ironically, the equivalent in Perl is // which Python has used for > truncating division since version 2.4 or so. So if we're in danger of > looking "Perlish", that ship has sailed a long time ago. > > Perl is hardly the only language with null-coalescing operators -- we > might better describe ?? as being familiar to C#, PHP, Swift and Dart. > That's two mature, well-known languages and two up-and-coming languages. My experience with these operators in C# says: * They do save "more than a few keystrokes". Even more importantly, they allow to avoid double evaluation or the need for a temporary variable workaround that are inherent in " if else " ??? * (An alternative solution for the latter problem would be an assignment expression, another regularly rejected proposal.) * They make it temptingly easy and implicit to ignore errors. * They are alien to Python's standard semantics on search failure which is to raise an exception rather than return None > > [...] >> timeout ?? local_timeout ?? global_timeout > As opposed to the status quo: > > timeout if timeout is not None else (local_timeout if local_timeout is not None else global_timeout) > > Or shorter, but even harder to understand: > > (global_timeout if local_timeout is None else local_timeout) if timeout is None else timeout > > I'd much prefer to teach the version with ?? -- it has a simple > explanation: "the first of the three given values which isn't None". The > ?? itself needs to be memorized, but that's no different from any other > operator. The first time I saw ** I was perplexed and couldn't imagine > what it meaned. > > Here ?? doesn't merely save a few keystrokes, it significantly reduces > the length and complexity of the expression and entirely cuts out the > duplication of names. > > If you can teach > > timeout or local_timeout or global_timeout > > then you ought to be able to teach ??, as it is simpler: it only > compares to None, and avoids needing to explain or justify Python's > truthiness model. > > >> 'foo' in (None ?? ['foo', 'bar']) > If you can understand > > 'foo' in (False or ['foo', 'bar']) > > then surely you can understand the version with ??. > > >> requested_quantity ?? default_quantity * price > Again: > > (default_quantity if requested_quantity is None else requested_quantity) * price > > > I'd much prefer to read, write and teach the version with ?? over the > status quo. > > -- Regards, Ivan From random832 at fastmail.com Thu Nov 30 12:06:13 2017 From: random832 at fastmail.com (Random832) Date: Thu, 30 Nov 2017 12:06:13 -0500 Subject: [Python-ideas] Conditional exception catching, lexical exceptions, None-specific exceptions In-Reply-To: References: <28D91255-56A9-4CC2-B45D-F83ECD715544@langa.pl> <50C74ECC-D462-4FAC-8A9C-A12BC939ADEB@gmail.com> <1511974989.903268.1188352080.36BF6ADC@webmail.messagingengine.com> Message-ID: <1512061573.1035805.1189598384.0698EC95@webmail.messagingengine.com> On Wed, Nov 29, 2017, at 12:40, David Mertz wrote: > I like much of the thinking in Random's approach. But I still think > None isn't quite special enough to warrant it's own syntax. > > However, his '(or None: name.strip()[4:].upper())' makes me realize > that what is being asked in all the '?(', '?.', '?[' syntax ideas is a > kind of ternary expression. Except the ternary isn't based on whether > a predicate holds, but rather on whether an exception occurs > (AttributeError, KeyError, TypeError). And the fallback in the > ternary is always None rather than being general. I did say, though, that it should only "catch" errors arising from operations that are lexically within the block. Seeing this, I got an idea in my head of a feature set that (along with expression syntax that has already been suggested), could together fulfill this need and provide more generality. --- Conditional exception catching --- "except [ExceptionTypes] [as name] [when condition]" as a syntax to only catch if [condition] is true. the byte code appears to already evaluate an arbitrary condition, it just currently only ever uses it to check types with the special "exception match" compare_op. We could use "if" as the keyword for conditions instead of "when", but I'm not sure if that creates ambiguity with proposed except-expression syntax when a conditional operator might also be used. I don't know if this has been proposed before. C# and VB have it, as do some C and Javascript implementations. I don't know if it would be equivalent or subtly different from re-raising if the condition is false, but even if so it would make it much easier to implement this sort of logic in an expression. --- Lexical exceptions --- Have at least some exceptions (say, AttributeError) carry enough information to allow for an exception catching statement to only catch when the exception was caused by the current code, not a function being called. I don't have much idea of how exactly this could work - something to do with the stack trace, maybe? A complicating factor might be that we may want to e.g. catch an AttributeError that comes from a raise statement within a __getattribute__ function, but not from an attribute lookup within such a function. This could be implemented in a function to be used with the above conditional feature, rather than made another keyword. --- None-specific exceptions --- Create a NoneError class to act as a second base class for subclasses of TypeError and AttributeError used when the value causing the error was None. I don't know if the exception catching machinery allows for multiple inheritance, if not, the conditional exception catching feature could be used to use a regular type check instead of the special exception match check [i.e. except as e when isinstance(e, NoneError)] None might not be special enough to warrant its own syntax, but I think it is special enough for this. From brett at python.org Thu Nov 30 13:24:53 2017 From: brett at python.org (Brett Cannon) Date: Thu, 30 Nov 2017 18:24:53 +0000 Subject: [Python-ideas] [Python-Dev] What's the status of PEP 505: None-aware operators? In-Reply-To: <5A1F2646.1080701@canterbury.ac.nz> References: <28D91255-56A9-4CC2-B45D-F83ECD715544@langa.pl> <50C74ECC-D462-4FAC-8A9C-A12BC939ADEB@gmail.com> <20171129060843.GZ22248@ando.pearwood.info> <20171129094527.GA22248@ando.pearwood.info> <5A1F2646.1080701@canterbury.ac.nz> Message-ID: On Wed, 29 Nov 2017 at 13:28 Greg Ewing wrote: > Nick Coghlan wrote: > > >>What about more English-like syntax: > >> > >>X or else Y > > > > The problem with constructs like this is that they look like they > > should mean the same thing as "X or Y". > > How about: > > x otherwise y > > It looks different enough from "or" that you're not going > to accidentally read it that way when skimming. > > The meaning is something you'll have to learn if you don't > know, but at least it's a word that can be googled for. > In terms of syntax, this is my favourite one so far. -------------- next part -------------- An HTML attachment was scrubbed... URL: From gadgetsteve at live.co.uk Thu Nov 30 11:16:21 2017 From: gadgetsteve at live.co.uk (Steve Barnes) Date: Thu, 30 Nov 2017 16:16:21 +0000 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: <20171130155024.GG22248@ando.pearwood.info> References: <20171130155024.GG22248@ando.pearwood.info> Message-ID: On 30/11/2017 15:50, Steven D'Aprano wrote: > On Wed, Nov 29, 2017 at 07:33:54PM +0000, Steve Barnes wrote: > >> Just a thought but what about a syntax something along the lines of: >> >> a, b, *remainder = iterable >> >> Where remainder becomes the iterable with the first two values consumed >> by assigning to a & b. > > Guido's time machine strikes again. That has worked since 3.3 if not > older. (Probably 3.0 or so, but I don't have that on this computer to > test it.) > > py> a, b, *remainder = range(10) > py> a > 0 > py> b > 1 > py> remainder > [2, 3, 4, 5, 6, 7, 8, 9] > > > > I had a sneaky feeling that it did, which raises the question of what the bleep this enormous thread is about, since the fundamental syntax currently exists.... -- Steve (Gadget) Barnes Any opinions in this message are my personal opinions and do not reflect those of my employer. --- This email has been checked for viruses by AVG. http://www.avg.com From solipsis at pitrou.net Thu Nov 30 13:49:56 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 30 Nov 2017 19:49:56 +0100 Subject: [Python-ideas] PEP 505 vs matrix multiplication References: <28D91255-56A9-4CC2-B45D-F83ECD715544@langa.pl> <50C74ECC-D462-4FAC-8A9C-A12BC939ADEB@gmail.com> <20171129060843.GZ22248@ando.pearwood.info> Message-ID: <20171130194956.73732307@fsol> On Wed, 29 Nov 2017 18:14:36 +1000 Nick Coghlan wrote: > > As far as utility goes, I put it in a similar category to matrix > multiplication: if you don't need it, you don't need it, but when you > do need it, you need it a *lot*. As someone who appreciates both the matrix multiplication operator and "async/await", I really don't think PEP 505-style operators (regardless of their spellings) fall into the same conceptual bucket. There's no risk of matrix multiplication operators bleeding into non-domain specific code, and the readers of domain specific code already know about the matrix multiplication operator and what it does (or they should anyway, since it's so damn useful). It's like "async/await": you won't find them in regular non-async code, so the mental burden only falls on specialists who write and read event-driven networking code (mostly, even though Guido would like to see parsers based on the idiom too :-)). Conversely, PEP 505-style operators may appear in everyday code regardless of their application domain or target. This in turn increases the mental burden for *everyone*. Regards Antoine. From p.f.moore at gmail.com Thu Nov 30 14:19:30 2017 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 30 Nov 2017 19:19:30 +0000 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: References: <20171130155024.GG22248@ando.pearwood.info> Message-ID: On 30 November 2017 at 16:16, Steve Barnes wrote: > I had a sneaky feeling that it did, which raises the question of what > the bleep this enormous thread is about, since the fundamental syntax > currently exists.... Essentially, it's about the fact that to build remainder you need to copy all of the remaining items out of the RHS. And for an infinite iterator like count() this is an infinite loop. There's also the point that if the RHS is an iterator, reading *any* values out of it changes its state - and 1. a, b, *remainder = rhs therefore exhausts rhs 2. a.b = rhs reads "one too many" values from rhs to check if there are extra values (which the programmer has asserted shouldn't be there by not including *remainder). Mostly corner cases, and I don't believe there have been any non-artificial examples posted in this thread. Certainly no-one has offered a real-life code example that is made significantly worse by the current semantics, and/or which couldn't be easily worked around without needing a language change. Paul From tseaver at palladion.com Thu Nov 30 13:45:07 2017 From: tseaver at palladion.com (Tres Seaver) Date: Thu, 30 Nov 2017 13:45:07 -0500 Subject: [Python-ideas] What's the status of PEP 505: None-aware operators? In-Reply-To: <718169B1-C1AE-4FCA-92E7-D4E246096612@python.org> References: <28D91255-56A9-4CC2-B45D-F83ECD715544@langa.pl> <50C74ECC-D462-4FAC-8A9C-A12BC939ADEB@gmail.com> <1511974989.903268.1188352080.36BF6ADC@webmail.messagingengine.com> <718169B1-C1AE-4FCA-92E7-D4E246096612@python.org> Message-ID: On 11/29/2017 01:02 PM, Barry Warsaw wrote: > I don?t know whether I like any of this but I think a more > natural spelling would be: > > val = name.strip()[4:].upper() except (AttributeError, KeyError) as -1 > > which could devolve into: > > val = name.strip()[4:].upper() except KeyError as -1 > > or: > > val = name.strip()[4:].upper() except KeyError # Implicit `as None` Of all the proposed spellings for the idea, this one feels most "normal" to me, too (I'm -0 on the idea as a whole). > I would *not* add any spelling for an explicit bare-except equivalent. > You would have to write: > > val = name.strip()[4:].upper() except Exception as -1 Wouldn't that really need to be this instead, for a true 'except:' equivalence: val = name.strip()[4:].upper() except BaseException as -1 Tres. -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com From rosuav at gmail.com Thu Nov 30 14:45:33 2017 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 1 Dec 2017 06:45:33 +1100 Subject: [Python-ideas] [Python-Dev] What's the status of PEP 505: None-aware operators? In-Reply-To: References: <28D91255-56A9-4CC2-B45D-F83ECD715544@langa.pl> <50C74ECC-D462-4FAC-8A9C-A12BC939ADEB@gmail.com> <1511974989.903268.1188352080.36BF6ADC@webmail.messagingengine.com> <718169B1-C1AE-4FCA-92E7-D4E246096612@python.org> Message-ID: On Fri, Dec 1, 2017 at 5:45 AM, Tres Seaver wrote: >> I would *not* add any spelling for an explicit bare-except equivalent. >> You would have to write: >> >> val = name.strip()[4:].upper() except Exception as -1 > > > Wouldn't that really need to be this instead, for a true 'except:' equivalence: > > val = name.strip()[4:].upper() except BaseException as -1 > Read the rejected PEP 463 for all the details and arguments. All this has been gone into. ChrisA From prometheus235 at gmail.com Thu Nov 30 15:34:51 2017 From: prometheus235 at gmail.com (Nick Timkovich) Date: Thu, 30 Nov 2017 14:34:51 -0600 Subject: [Python-ideas] [Python-Dev] What's the status of PEP 505: None-aware operators? In-Reply-To: References: <28D91255-56A9-4CC2-B45D-F83ECD715544@langa.pl> <50C74ECC-D462-4FAC-8A9C-A12BC939ADEB@gmail.com> <20171129060843.GZ22248@ando.pearwood.info> <20171129094527.GA22248@ando.pearwood.info> <5A1F2646.1080701@canterbury.ac.nz> Message-ID: On Thu, Nov 30, 2017 at 12:24 PM, Brett Cannon wrote: > On Wed, 29 Nov 2017 at 13:28 Greg Ewing > wrote: > >> Nick Coghlan wrote: >> >> >>What about more English-like syntax: >> >> >> >>X or else Y >> > >> > The problem with constructs like this is that they look like they >> > should mean the same thing as "X or Y". >> >> How about: >> >> x otherwise y >> >> It looks different enough from "or" that you're not going >> to accidentally read it that way when skimming. >> >> The meaning is something you'll have to learn if you don't >> know, but at least it's a word that can be googled for. >> > > In terms of syntax, this is my favourite one so far. > Was the issue that "or else" looks too similar or looks too similar but acts very different? I'm interpreting the behavior like a(n ab)use of "or", just where it's looking for None. Am I overlooking some conditions? x = False y = 'something' x or y # 'something' x or else y # False x = None x or else y # 'something' Overlooking 'else' when reading would be surprising, though might be a source of confusion for newcomers when "x or else y" and "x or y" usually act identically, kinda like "x == y" and "x is y" (for small integers). Nick -------------- next part -------------- An HTML attachment was scrubbed... URL: From kulakov.ilya at gmail.com Thu Nov 30 16:08:04 2017 From: kulakov.ilya at gmail.com (Ilya Kulakov) Date: Thu, 30 Nov 2017 13:08:04 -0800 Subject: [Python-ideas] Stub class for Generic to further improve PEP 560 Message-ID: <2FCB1F0E-24AF-4ACC-922F-D5E44B618C37@gmail.com> As explained in PEP 560, creation of Generic subclasses is much slower (~7x on my machine). My guess is that a limited number of applications actually need Generic classes at "production" runtime (i.e. when not under analysis of mypy or IDE). I propose a special class that will alias typing.Generic whenever typing.TYPE_CHECKING == True and a no-op stub otherwise. Best Regards, Ilya Kulakov -------------- next part -------------- An HTML attachment was scrubbed... URL: From levkivskyi at gmail.com Thu Nov 30 16:22:27 2017 From: levkivskyi at gmail.com (Ivan Levkivskyi) Date: Thu, 30 Nov 2017 22:22:27 +0100 Subject: [Python-ideas] Stub class for Generic to further improve PEP 560 In-Reply-To: <2FCB1F0E-24AF-4ACC-922F-D5E44B618C37@gmail.com> References: <2FCB1F0E-24AF-4ACC-922F-D5E44B618C37@gmail.com> Message-ID: On 30 November 2017 at 22:08, Ilya Kulakov wrote: > As explained in PEP 560, creation of Generic subclasses is much slower > (~7x on my machine). > My guess is that a limited number of applications actually need Generic > classes at "production" runtime (i.e. when not under analysis of mypy or > IDE). > > I propose a special class that will alias typing.Generic whenever > typing.TYPE_CHECKING == True and a no-op stub otherwise. > > FWIW, with PEP 560 generic classes will stay generic at runtime. I am not sure what you mean by this, but practically all existing runtime APIs (with few exceptions, see PEP 560) for generics will stay. -- Ivan -------------- next part -------------- An HTML attachment was scrubbed... URL: From kulakov.ilya at gmail.com Thu Nov 30 16:25:22 2017 From: kulakov.ilya at gmail.com (Ilya Kulakov) Date: Thu, 30 Nov 2017 13:25:22 -0800 Subject: [Python-ideas] Stub class for Generic to further improve PEP 560 In-Reply-To: References: <2FCB1F0E-24AF-4ACC-922F-D5E44B618C37@gmail.com> Message-ID: My point is to have a class with public interface identical to typing.Generic, but without all runtime overhead. It's based on the assumption that few developers benefit with typing.Generic addition during production application execution. Those who do, can subclass typing.Generic directly. > On Nov 30, 2017, at 1:22 PM, Ivan Levkivskyi wrote: > > FWIW, with PEP 560 generic classes will stay generic at runtime. I am not sure what you mean by this, but > practically all existing runtime APIs (with few exceptions, see PEP 560) for generics will stay. From levkivskyi at gmail.com Thu Nov 30 16:35:00 2017 From: levkivskyi at gmail.com (Ivan Levkivskyi) Date: Thu, 30 Nov 2017 22:35:00 +0100 Subject: [Python-ideas] Stub class for Generic to further improve PEP 560 In-Reply-To: References: <2FCB1F0E-24AF-4ACC-922F-D5E44B618C37@gmail.com> Message-ID: On 30 November 2017 at 22:25, Ilya Kulakov wrote: > My point is to have a class with public interface identical to > typing.Generic, but without all runtime overhead. > It's based on the assumption that few developers benefit with > typing.Generic addition during production application execution. > > Those who do, can subclass typing.Generic directly Could you please give some examples for these statements? They still look to abstract for me. -- Ivan -------------- next part -------------- An HTML attachment was scrubbed... URL: From kulakov.ilya at gmail.com Thu Nov 30 16:38:28 2017 From: kulakov.ilya at gmail.com (Ilya Kulakov) Date: Thu, 30 Nov 2017 13:38:28 -0800 Subject: [Python-ideas] Stub class for Generic to further improve PEP 560 In-Reply-To: References: <2FCB1F0E-24AF-4ACC-922F-D5E44B618C37@gmail.com> Message-ID: <3ECD02CE-F378-4A75-BEA6-4F9AE1FF55AF@gmail.com> A very rough implementation: typing.py: class _GenericMetaNoop(type): def __getitem__(self, params): return self class _GenericNoop(metaclass=_GenericMetaNoop) pass GenericStub = Generic if TYPE_CHECKING else _GenericNoop elsewhere.py: import typing T = typing.TypeVar('T') class MyClass(typing.GenericStub[T]): pass Now when run under type checkers MyClass will inherit typing.Generic with all runtime complexity. However when run outside of them code will just work. > On Nov 30, 2017, at 1:35 PM, Ivan Levkivskyi wrote: > > Could you please give some examples for these statements? They still look to abstract for me. From greg.ewing at canterbury.ac.nz Thu Nov 30 17:19:49 2017 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 01 Dec 2017 11:19:49 +1300 Subject: [Python-ideas] Add a dict with the attribute access capability In-Reply-To: <3d3c8f2c-0722-1012-6d15-a5706f79ee55@mail.mipt.ru> References: <3d3c8f2c-0722-1012-6d15-a5706f79ee55@mail.mipt.ru> Message-ID: <5A208405.6030407@canterbury.ac.nz> Ivan Pozdeev via Python-ideas wrote: > I needed to hold an external function > reference in an object instance (if I assigned it to an attribute, it > was converted into an instance method). No, that only happens to functions stored in *class* attributes, not instance attributes. >>> class A: ... pass ... >>> a = A() >>> >>> def f(): ... print("I'm just a function") ... >>> a.x = f >>> a.x() I'm just a function -- Greg From levkivskyi at gmail.com Thu Nov 30 17:24:08 2017 From: levkivskyi at gmail.com (Ivan Levkivskyi) Date: Thu, 30 Nov 2017 23:24:08 +0100 Subject: [Python-ideas] Stub class for Generic to further improve PEP 560 In-Reply-To: <3ECD02CE-F378-4A75-BEA6-4F9AE1FF55AF@gmail.com> References: <2FCB1F0E-24AF-4ACC-922F-D5E44B618C37@gmail.com> <3ECD02CE-F378-4A75-BEA6-4F9AE1FF55AF@gmail.com> Message-ID: On 30 November 2017 at 22:38, Ilya Kulakov wrote: > A very rough implementation: > > typing.py: > > class _GenericMetaNoop(type): > def __getitem__(self, params): > return self > > class _GenericNoop(metaclass=_GenericMetaNoop) > pass > > GenericStub = Generic if TYPE_CHECKING else _GenericNoop > > elsewhere.py: > > import typing > > T = typing.TypeVar('T') > > class MyClass(typing.GenericStub[T]): > pass > > OK, I see. Note that all type checkers known to me never actually read content of typing.py (it is always heavily special cased). Therefore, one can safely write `GenericStub = _GenericNoop`. Anyway, my expectation is that going along this way (i.e. removing all runtime API apart from a necessary minimum) will give a minor speed-up as compared to PEP 560 at the cost of a breaking change (even for small number of developers). PEP 560 already gives overhead of 80% as compared to normal classes in worst case scenario (empty body with a single generic base). This is actually less than for ABCs (they can give up to 120% in worst case scenario). Moreover, performance is not a single motivation for PEP 560, there are other arguments such as metaclass conflicts which will not be solved without the machinery proposed by the PEP. -- Ivan -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Thu Nov 30 17:26:02 2017 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 01 Dec 2017 11:26:02 +1300 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: <20171130155024.GG22248@ando.pearwood.info> References: <20171130155024.GG22248@ando.pearwood.info> Message-ID: <5A20857A.9000901@canterbury.ac.nz> Steven D'Aprano wrote: > On Wed, Nov 29, 2017 at 07:33:54PM +0000, Steve Barnes wrote: > >>Just a thought but what about a syntax something along the lines of: >> >>a, b, *remainder = iterable >> >>Where remainder becomes the iterable with the first two values consumed >>by assigning to a & b. > > Guido's time machine strikes again. That has worked since 3.3 if not > older. This is not quite the same thing -- the rest of the items are extracted and put in a new list. I think Steve Barnes is suggesting that the iterator should be left alone and bound to the * target instead. That would be an incompatible change. -- Greg From gadgetsteve at live.co.uk Thu Nov 30 17:59:05 2017 From: gadgetsteve at live.co.uk (Steve Barnes) Date: Thu, 30 Nov 2017 22:59:05 +0000 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: <5A20857A.9000901@canterbury.ac.nz> References: <20171130155024.GG22248@ando.pearwood.info> <5A20857A.9000901@canterbury.ac.nz> Message-ID: On 30/11/2017 22:26, Greg Ewing wrote: > Steven D'Aprano wrote: >> On Wed, Nov 29, 2017 at 07:33:54PM +0000, Steve Barnes wrote: >> >>> Just a thought but what about a syntax something along the lines of: >>> >>> a, b, *remainder = iterable >>> >>> Where remainder becomes the iterable with the first two values >>> consumed by assigning to a & b. >> >> Guido's time machine strikes again. That has worked since 3.3 if not >> older. > > This is not quite the same thing -- the rest of the items > are extracted and put in a new list. I think Steve Barnes is > suggesting that the iterator should be left alone and > bound to the * target instead. > > That would be an incompatible change. > That is what I was thinking of but surely it would be more efficient way to do this for generators and large iterators. Since the practical difference between remainder being the (partly exhausted) iterator and it being a list, (itself an iterator), containing the results of expanding the remainder of the original iterator is slight. The only limitation would be that the syntax a, b, *remainder, c, d = iterator would be illegal. -- Steve (Gadget) Barnes Any opinions in this message are my personal opinions and do not reflect those of my employer. --- This email has been checked for viruses by AVG. http://www.avg.com From kulakov.ilya at gmail.com Thu Nov 30 18:34:21 2017 From: kulakov.ilya at gmail.com (Ilya Kulakov) Date: Thu, 30 Nov 2017 15:34:21 -0800 Subject: [Python-ideas] Stub class for Generic to further improve PEP 560 In-Reply-To: References: <2FCB1F0E-24AF-4ACC-922F-D5E44B618C37@gmail.com> <3ECD02CE-F378-4A75-BEA6-4F9AE1FF55AF@gmail.com> Message-ID: <5E326196-7E95-433D-8F1C-A1537278F31A@gmail.com> > Anyway, my expectation is that going along this way (i.e. removing all runtime API apart from a necessary minimum) > will give a minor speed-up as compared to PEP 560 at the cost of a breaking change (even for small number of developers). I don't think the change will be breaking: usage of this class will be entirely voluntarily and does not replace typing.Generic > PEP 560 already gives overhead of 80% as compared to normal classes in worst case scenario > (empty body with a single generic base). This is actually less than for ABCs (they can give up to 120% in worst case scenario). GenericMeta inherits from ABCMeta. Do you mean that it will be removed after 560 lands? > Moreover, performance is not a single motivation for PEP 560, there are other arguments such as metaclass conflicts which will > not be solved without the machinery proposed by the PEP. Perhaps you can consider designing Generic / GenericMeta in a way that will allow end user to create GenericStub-alike class without much trouble? -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Thu Nov 30 23:44:00 2017 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 01 Dec 2017 17:44:00 +1300 Subject: [Python-ideas] How assignment should work with generators? In-Reply-To: References: <20171130155024.GG22248@ando.pearwood.info> <5A20857A.9000901@canterbury.ac.nz> Message-ID: <5A20DE10.2070509@canterbury.ac.nz> Steve Barnes wrote: > Since the practical > difference between remainder being the (partly exhausted) iterator and > it being a list, (itself an iterator), A list is *not* an iterator, it's a sequence. There's a huge difference. Items of a sequence can be accessed at random, it can be iterated over multiple times, if it's mutable its contents can be changed, etc. None of that can be done to an iterator. Maybe it would have been a defensible design decision when *-unpacking was introduced, but it's too late to change it now. Even if it weren't, there are arguments against it. It would be inconsistent with the usage of * in an argument list, where the function always receives a new tuple, never an iterator over something passed as a * argument. -- Greg