Moving typing out of the stdlib in Python 3.7?
[A copy from https://github.com/python/typing/issues/495 to get more people's attention to this issue.] I'm wondering if we should remove typing from the stdlib. Now's the time to think about this, as the feature freeze for 3.7 is about 12 weeks away. Cons: - People have to depend on a PyPI package to use typing (but they do anyway for typing_extensions) - It's a backward incompatibility for users of Python 3.5 and 3.6 (but the typing module was always provisional) Pros: - The typing module can evolve much faster outside the stdlib - We could get rid of typing_extensions (and maybe even mypy_extensions) If we don't do this I worry that we're entering a period where many new typesystem features end up in typing_extensions and users will be confused about which items are in typing and which in typing_extensions (not to mention mypy_extensions). Anything new to be added to typing (e.g. Const, Final, Literal, or changing ABCs to Protocols) would have to be added to typing_extensions instead, and users would be confused about which features exist in which module. Moving typing out of the stdlib can make things potentially simpler, at the cost of an extra pip install (but they'll need one anyway for mypy). Thoughts? -- --Guido van Rossum (python.org/~guido)
On Sat, Nov 4, 2017 at 1:35 AM, Guido van Rossum <guido@python.org> wrote:
If the lazy evaluation of annotations (PEP 563) also lands in 3.7, then this would be a very minor downside. You'd need to pip-install typing as well as mypy *for the actual type checking*, but at run time, you could ignore both (all those List[...] annotations would be stored unevaluated). Otherwise, it'd mean that any project that makes use of type hints would require typing as a run-time dependency. ChrisA
I could see the typing module staying but: - typing_extension (or the new official external module) append types in the typing module; - if typing_extension is not installed or too old and missing some types, any missing type being accessed returns some kind of mock object that avoid crashing; - any running type checker encountering mocked types has to show a warning - you can pip install typing (and it's mocking abilities) in python 2. This way somebody without the 3rd party module can still use the code without crashing it. Having types out of date or missing a brand new one won't crash the code, only have a warning in mypy or pycharm. This solves the compatibility problem, but also another major problem currently: several projects I worked on had a fake typing module I could use it in python 2 and 3 without having to manually install typing. Sometime I also create scripts and leave type annotations, and my users really don't want to install the exact typing extension to make it work. And I don't want to remove the annotations manually after ward. Le 03/11/2017 à 15:51, Chris Angelico a écrit :
I use typing quite systematically nowadays, even for projects that don't use mypy (for historical reasons: bringing an older code base to zero mypy issues can be quite time-consuming). For instance, adding typing annotation can help autocompletion under PyCharm (and hopefully other IDEs). With these annotations, PyCharm is also able to signal typing issues either directly in the editor, or when running a code check. I'm quite OK with removing the typing module from the stdlib as it can easily be added to my projects dependencies, and I can definitively understand the benefits of a faster release cycle, but I'm worried that this could hinder adoption of these practices by certain people. S. On Fri, Nov 3, 2017 at 3:35 PM, Guido van Rossum <guido@python.org> wrote:
-- Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier - http://linkedin.com/in/sfermigier Founder & CEO, Abilian - Enterprise Social Software - http://www.abilian.com/ Chairman, Free&OSS Group / Systematic Cluster - http://www.gt-logiciel-libre.org/ Co-Chairman, National Council for Free & Open Source Software (CNLL) - http://cnll.fr/ Founder & Organiser, PyData Paris - http://pydata.fr/ --- “You never change things by fighting the existing reality. To change something, build a new model that makes the existing model obsolete.” — R. Buckminster Fuller
On Fri, 3 Nov 2017 16:04:43 +0100 Stéfane Fermigier <sf@fermigier.com> wrote:
I don't think casual or beginner users of Python really have to worry about typing annotations. As I understand it, they become really useful on middle- to large-scale projects (disclaimer: I've never used them myself; the kind of typing the Numba project does -- which I don't participate in anymore -- is quite different). Regards Antoine.
On 4 November 2017 at 00:35, Guido van Rossum <guido@python.org> wrote:
Perhaps typing could switch to being a bundled module, such that it had its own version, independent of the Python standard library version, but was still present by default in new installations? Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sat, Nov 4, 2017 at 7:05 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
This is beginning to sound like the most attractive solution. We could possibly do away with typing_extensions. Are there precedents of how to bundle a module in this way? Or is it going to be another special case like pip? -- --Guido van Rossum (python.org/~guido)
On 5 November 2017 at 06:22, Guido van Rossum <guido@python.org> wrote:
You'd be blazing new trails, but you'd be blazing them with a single-file pure Python module without any binary dependencies outside CPython itself, which is the simplest possible precedent setting scenario we could hope for. I can think of a couple of possible approaches you could take, depending on how you want to manage development on the typing module itself. = External dev = * Typing moves entirely out of the CPython source tree, with a separate issue tracker, repository, branch structure, etc (which you already have) * We add a "_bundled" directory to the CPython source tree in 3.7+ and put a typing sdist archive there * the release process PEP gains a "Confirm the bundled modules are up to date" step * the CPython build process is updated to use a venv to generate wheel files from bundled sdists * regrtest gains a "bundled" resource (and/or dedicated command line option) * when asked to test bundled modules, regrtest spins up a test venv, installs pip and the bundled modules, then runs the tests for those modules * ensurepip gains the ability to also install bundled wheel files * the Windows and Mac OS X installers start including the typing wheel file = In tree dev = * Similar to external dev for testing and distribution * Source code would stay in the main CPython tree and track CPython branch structure * The sdist file would be built by CPython's build process, rather than being checked in The external dev model is probably a better fit for your use case, though, since you'd like something that you can readily keep consistent across all supported 3.x versions (or at least 3.7+), and that's harder to do when you have multiple copies of the source code to keep in sync (vs having a single development branch where you run your CI testing across all supported Python versions). The external dev option also establishes a more useful precedent, since we could use it to officially migrate ongoing distutils maintenance into the setuptools project, and then bring it back via bundling of setuptools. Similarly, PEP 434 (which already exempts IDLE from the usual "No new features in maintenance releases" rules) could potentially be switched over to handling IDLE as an independently updated bundled application. The above idea is somewhat similar to what I suggested for the recommended modules list in https://mail.python.org/pipermail/python-ideas/2017-October/047508.html, but the difference is that to simplify the testing and installer building process, we'd bundle a suitable source archive in the git repo, rather than requiring PyPI access. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 5 November 2017 at 10:48, Antoine Pitrou <solipsis@pitrou.net> wrote:
I'm not quite sure what you mean? It needs to be "installed", in the sense of being unpacked into site-packages, and the ensurepip mechanism is already able to do that for pip and setuptools, so adding an extra wheel to install wouldn't be too hard. If we don't install like that, people won't be able to easily upgrade typing by just using "pip install --upgrade typing". Wheels don't support simply being added to sys.path the way that eggs did, if that's what you mean. Paul
On 5 November 2017 at 23:30, Paul Moore <p.f.moore@gmail.com> wrote:
What Paul said here. While wheels *can* be designed to support use-without-extraction (and pip's own wheel file is constructed that way so you can use a pip wheel to install itself), the general design principle is that we expect them to be installed prior to use, such that they get proper PEP 376 metadata entries, and so that their subcomponents end up in the right sysconfig directories for the deployment environment. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Le 05/11/2017 à 14:30, Paul Moore a écrit :
Ok, perhaps my question wasn't quite clear. Are you suggesting that people have to run "python -m ensurepip typing" after they installed Python? Or would the typing module be importable as soon as you have an installed Python, like stdlib modules are? Regards Antoine.
On 5 November 2017 at 14:47, Antoine Pitrou <antoine@python.org> wrote:
Ah, I get you now. I'd expect typing to be available exactly the same way as pip is. For me (on Windows) that means that it's available in a standard install. On Unix, I don't know (I think there's certain situations where you need to take extra steps in a custom build to ensure pip is available? I'd expect those steps to also install typing. So basically, yes, typing can be expected to be available in all installations, exactly the same as pip is. Paul
On Sun, 5 Nov 2017 18:22:11 +0000 Paul Moore <p.f.moore@gmail.com> wrote:
I think typing shouldn't require any extra typing (ha) on Unix either. I don't remember what the rationale was for having to type "python -m ensurepip" to get pip installed, but typing is just a library, not an executable tool that may be able to mess with the system state, so I don't think it's worthwhile introducing an extra step for it. Regards Antoine.
On 6 November 2017 at 05:00, Paul Moore <p.f.moore@gmail.com> wrote:
It's the default on Unix as well - you have to do "make install ENSUREPIP=no" to avoid getting it. (And some distros also modify their Python installations so that pip is missing by default) We provide "python -m ensurepip" because we offer ways to skip doing it at install time (and venv creation time), so the explicit command gives folks a straightforward way to change their minds later. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Le 06/11/2017 à 07:07, Nick Coghlan a écrit :
On debian and derivatives (so Ubuntu) you need to install python-pip to be able to use pip. Now it's annoying already. Because you have to write every tutorial to include a special case for them. But at least it's not a required step to run your program. However, if you do code using type hints and typing is not installed, you can't even run the program without installing something. So we finally have Python 3 by default on most Linux system, but still would not be able to assume we can run a modern script on it.
This is a valid concern. Although this particular problem is self-inflicted by Debian, I can understand their rationale behind explicit packaging. They need to have control over the entire package graph. I wonder if there's a way in .deb to specify a required installed package. I'm not calling it a "dependency" since obviously it would rather be "python3-typing" that depends on "python3". Barry, do you know? But even if Debian installs python3-typing by default, will "pip install -U typing" be possible in this scenario? I guess it wouldn't be terrible if that only worked in virtualenvs, although ideally it would work also for the raw host installation. - Ł
On 6 November 2017 at 16:42, Lukasz Langa <lukasz@langa.pl> wrote:
Fedora just lives with the circular dependency between python3 and python3-pip, which ensures both are installed by default (this arrangement allows "python3 -m venv" to still install pip, without actually needing a second copy of pip inside the python3 package) A bundled typing module that "python -m venv" installed by default would probably require similar treatment.
But even if Debian installs python3-typing by default, will "pip install -U typing" be possible in this scenario? I guess it wouldn't be terrible if that only worked in virtualenvs, although ideally it would work also for the raw host installation.
"sudo pip install <anything>" remains a terrible idea on any distro, because it leads to pip and the system package manager getting into a fight over which tool is responsible for managing the affected files. "pip install --user --upgrade typing" should work OK, though. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 6 November 2017 at 06:42, Lukasz Langa <lukasz@langa.pl> wrote:
It's also a problem for virtual environments, which won't include typing by default, and don't see the system installed packages. As Nick says, we could modify venv to always include typing, but most users, and all tools (tox, pew, pipenv, etc) use virtualenv rather than venv still, so this would be of very limited benefit.
I wonder if there's a way in .deb to specify a required installed package. I'm not calling it a "dependency" since obviously it would rather be "python3-typing" that depends on "python3". Barry, do you know?
There's always "include typing in the standard library" :-)
But even if Debian installs python3-typing by default, will "pip install -U typing" be possible in this scenario? I guess it wouldn't be terrible if that only worked in virtualenvs, although ideally it would work also for the raw host installation.
It would push yet more people into doing "sudo pip install -U typing", which is something we've been trying really hard to encourage them to stop doing over on the pip issues list. We strongly recommend using system supplied packages, but I'd be surprised if Debian is any more able to keep up with the latest versions of typing than Python core is. Paul
On Saturday, 4 November 2017 20:22:25 GMT Guido van Rossum wrote:
Is the outcome you want that you ship a version of typing with the python kit, but if you install from pip it overrides the one shipped in the python kit? That would be a matter of being having a suitable sys.path/site config I guess. pip folder before the bundled packages folder. If this is a mechanism that python kitting has then you would be able to bundle other packages like requests or six as well as typing, but because you can use pip to override the one shipped a user can optionally keep up with the latest versions. Barry
On Mon, Nov 6, 2017 at 9:29 AM, Barry Scott <barry@barrys-emacs.org> wrote:
If this were to happen, I would be inclined to put these "bootstrap" modules into their own directory in sys.path, after the rest of the stdlib. Then someone who's paranoid about stdlib shadowing could put pip-installed modules after the bulk of the stdlib (thus preventing any third-party package from overriding "import random", for instance) but still update modules that are specifically intended for updating; plus it'd mean you can get a directory listing of that, and go grab all the "blessed by python.org as an extension of the stdlib" packages. ChrisA
On 6 November 2017 at 09:08, Chris Angelico <rosuav@gmail.com> wrote:
When we say bundled, we mean bundled: the exact same bits you would get from PyPI, installed the same way, and if you upgrade it system wide, there's no trace of the one we bundled. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 11/06/2017 06:48 AM, brent bejot wrote:
On Mon, Nov 6, 2017 at 9:21 AM, Gyro Funch wrote:
Yes. Stability is probably the first (knowing what is support in a particular version). Second would be that nearly all modules in the stdlib are stable and don't need frequent updates -- in fact, I suspect typing is the only exception. -- ~Ethan~
Yes: https://www.python.org/dev/peps/pep-0206/#batteries-included-philosophy I can't find it but there is a great story about a guy needing to do a source code analysis in total isolation (not even a usb key or a cd) and the computer has nothing installed but Python. He gets by because Python is very useful "as is". I can say the same. When I log in on bare debian / centos or if I'm on a locked mac os. When i'm on a place or in a remote place in Africa. When I'm with customers locking down all the network with NDA, security badge and paranoia for desert... I can count on Python to be there for me and hashlib the stuff, etree thingy, zipfile the foo and uuid all the bars. It's a major feature. I can live without typing, but I would hate having to pip install a json parser.
Guido van Rossum wrote:
With my downstream consumer hat on, *if* we go with a bundling solution, please make sure it's easy to unbundle. Downstream consumers may need to do this for policy, security, etc. reasons, so making this too difficult will mean delays in releases, convoluted hacks, policy violations, or all of the above. See as reference some of the hoops Debian has to go through to handle ensurepip. Cheers, -Barry
On Mon, Nov 6, 2017 at 5:27 PM, Barry Warsaw <barry@python.org> wrote:
Having skimmed this thread and some others, I am going in the opposite direction. Moving typing.py out of the stdlib and then bundling it will cause too many problems, both psychological and technical. Therefore I am withdrawing my idea of moving typing.py out of the stdlib, and replacing it with an extension of its (and PEP 484's) provisional status by at least one release cycle. This means that at least for the duration of 3.7 we'll be able to update typing.py pretty aggressively in bugfix releases. See https://github.com/python/peps/pull/451 -- basically we only need a restricted interpretation of provisional, where we can add new features, but won't break backwards compatibility *for documented aspects of the API*. With PEP 560, we can then attempt to speed up the import of typing.py (hopefully 10x by the time 3.8 rolls around). I propose that the schedule for typing.py's provisional status be added to PEP 563 (the __future__ import for annotations); I've got a PR with tentative update to PEP 484 ready (https://github.com/python/peps/pull/451). -- --Guido van Rossum (python.org/~guido)
On Sat, Nov 4, 2017 at 1:35 AM, Guido van Rossum <guido@python.org> wrote:
If the lazy evaluation of annotations (PEP 563) also lands in 3.7, then this would be a very minor downside. You'd need to pip-install typing as well as mypy *for the actual type checking*, but at run time, you could ignore both (all those List[...] annotations would be stored unevaluated). Otherwise, it'd mean that any project that makes use of type hints would require typing as a run-time dependency. ChrisA
I could see the typing module staying but: - typing_extension (or the new official external module) append types in the typing module; - if typing_extension is not installed or too old and missing some types, any missing type being accessed returns some kind of mock object that avoid crashing; - any running type checker encountering mocked types has to show a warning - you can pip install typing (and it's mocking abilities) in python 2. This way somebody without the 3rd party module can still use the code without crashing it. Having types out of date or missing a brand new one won't crash the code, only have a warning in mypy or pycharm. This solves the compatibility problem, but also another major problem currently: several projects I worked on had a fake typing module I could use it in python 2 and 3 without having to manually install typing. Sometime I also create scripts and leave type annotations, and my users really don't want to install the exact typing extension to make it work. And I don't want to remove the annotations manually after ward. Le 03/11/2017 à 15:51, Chris Angelico a écrit :
I use typing quite systematically nowadays, even for projects that don't use mypy (for historical reasons: bringing an older code base to zero mypy issues can be quite time-consuming). For instance, adding typing annotation can help autocompletion under PyCharm (and hopefully other IDEs). With these annotations, PyCharm is also able to signal typing issues either directly in the editor, or when running a code check. I'm quite OK with removing the typing module from the stdlib as it can easily be added to my projects dependencies, and I can definitively understand the benefits of a faster release cycle, but I'm worried that this could hinder adoption of these practices by certain people. S. On Fri, Nov 3, 2017 at 3:35 PM, Guido van Rossum <guido@python.org> wrote:
-- Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier - http://linkedin.com/in/sfermigier Founder & CEO, Abilian - Enterprise Social Software - http://www.abilian.com/ Chairman, Free&OSS Group / Systematic Cluster - http://www.gt-logiciel-libre.org/ Co-Chairman, National Council for Free & Open Source Software (CNLL) - http://cnll.fr/ Founder & Organiser, PyData Paris - http://pydata.fr/ --- “You never change things by fighting the existing reality. To change something, build a new model that makes the existing model obsolete.” — R. Buckminster Fuller
On Fri, 3 Nov 2017 16:04:43 +0100 Stéfane Fermigier <sf@fermigier.com> wrote:
I don't think casual or beginner users of Python really have to worry about typing annotations. As I understand it, they become really useful on middle- to large-scale projects (disclaimer: I've never used them myself; the kind of typing the Numba project does -- which I don't participate in anymore -- is quite different). Regards Antoine.
On 4 November 2017 at 00:35, Guido van Rossum <guido@python.org> wrote:
Perhaps typing could switch to being a bundled module, such that it had its own version, independent of the Python standard library version, but was still present by default in new installations? Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sat, Nov 4, 2017 at 7:05 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
This is beginning to sound like the most attractive solution. We could possibly do away with typing_extensions. Are there precedents of how to bundle a module in this way? Or is it going to be another special case like pip? -- --Guido van Rossum (python.org/~guido)
On 5 November 2017 at 06:22, Guido van Rossum <guido@python.org> wrote:
You'd be blazing new trails, but you'd be blazing them with a single-file pure Python module without any binary dependencies outside CPython itself, which is the simplest possible precedent setting scenario we could hope for. I can think of a couple of possible approaches you could take, depending on how you want to manage development on the typing module itself. = External dev = * Typing moves entirely out of the CPython source tree, with a separate issue tracker, repository, branch structure, etc (which you already have) * We add a "_bundled" directory to the CPython source tree in 3.7+ and put a typing sdist archive there * the release process PEP gains a "Confirm the bundled modules are up to date" step * the CPython build process is updated to use a venv to generate wheel files from bundled sdists * regrtest gains a "bundled" resource (and/or dedicated command line option) * when asked to test bundled modules, regrtest spins up a test venv, installs pip and the bundled modules, then runs the tests for those modules * ensurepip gains the ability to also install bundled wheel files * the Windows and Mac OS X installers start including the typing wheel file = In tree dev = * Similar to external dev for testing and distribution * Source code would stay in the main CPython tree and track CPython branch structure * The sdist file would be built by CPython's build process, rather than being checked in The external dev model is probably a better fit for your use case, though, since you'd like something that you can readily keep consistent across all supported 3.x versions (or at least 3.7+), and that's harder to do when you have multiple copies of the source code to keep in sync (vs having a single development branch where you run your CI testing across all supported Python versions). The external dev option also establishes a more useful precedent, since we could use it to officially migrate ongoing distutils maintenance into the setuptools project, and then bring it back via bundling of setuptools. Similarly, PEP 434 (which already exempts IDLE from the usual "No new features in maintenance releases" rules) could potentially be switched over to handling IDLE as an independently updated bundled application. The above idea is somewhat similar to what I suggested for the recommended modules list in https://mail.python.org/pipermail/python-ideas/2017-October/047508.html, but the difference is that to simplify the testing and installer building process, we'd bundle a suitable source archive in the git repo, rather than requiring PyPI access. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 5 November 2017 at 10:48, Antoine Pitrou <solipsis@pitrou.net> wrote:
I'm not quite sure what you mean? It needs to be "installed", in the sense of being unpacked into site-packages, and the ensurepip mechanism is already able to do that for pip and setuptools, so adding an extra wheel to install wouldn't be too hard. If we don't install like that, people won't be able to easily upgrade typing by just using "pip install --upgrade typing". Wheels don't support simply being added to sys.path the way that eggs did, if that's what you mean. Paul
On 5 November 2017 at 23:30, Paul Moore <p.f.moore@gmail.com> wrote:
What Paul said here. While wheels *can* be designed to support use-without-extraction (and pip's own wheel file is constructed that way so you can use a pip wheel to install itself), the general design principle is that we expect them to be installed prior to use, such that they get proper PEP 376 metadata entries, and so that their subcomponents end up in the right sysconfig directories for the deployment environment. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Le 05/11/2017 à 14:30, Paul Moore a écrit :
Ok, perhaps my question wasn't quite clear. Are you suggesting that people have to run "python -m ensurepip typing" after they installed Python? Or would the typing module be importable as soon as you have an installed Python, like stdlib modules are? Regards Antoine.
On 5 November 2017 at 14:47, Antoine Pitrou <antoine@python.org> wrote:
Ah, I get you now. I'd expect typing to be available exactly the same way as pip is. For me (on Windows) that means that it's available in a standard install. On Unix, I don't know (I think there's certain situations where you need to take extra steps in a custom build to ensure pip is available? I'd expect those steps to also install typing. So basically, yes, typing can be expected to be available in all installations, exactly the same as pip is. Paul
On Sun, 5 Nov 2017 18:22:11 +0000 Paul Moore <p.f.moore@gmail.com> wrote:
I think typing shouldn't require any extra typing (ha) on Unix either. I don't remember what the rationale was for having to type "python -m ensurepip" to get pip installed, but typing is just a library, not an executable tool that may be able to mess with the system state, so I don't think it's worthwhile introducing an extra step for it. Regards Antoine.
On 6 November 2017 at 05:00, Paul Moore <p.f.moore@gmail.com> wrote:
It's the default on Unix as well - you have to do "make install ENSUREPIP=no" to avoid getting it. (And some distros also modify their Python installations so that pip is missing by default) We provide "python -m ensurepip" because we offer ways to skip doing it at install time (and venv creation time), so the explicit command gives folks a straightforward way to change their minds later. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Le 06/11/2017 à 07:07, Nick Coghlan a écrit :
On debian and derivatives (so Ubuntu) you need to install python-pip to be able to use pip. Now it's annoying already. Because you have to write every tutorial to include a special case for them. But at least it's not a required step to run your program. However, if you do code using type hints and typing is not installed, you can't even run the program without installing something. So we finally have Python 3 by default on most Linux system, but still would not be able to assume we can run a modern script on it.
This is a valid concern. Although this particular problem is self-inflicted by Debian, I can understand their rationale behind explicit packaging. They need to have control over the entire package graph. I wonder if there's a way in .deb to specify a required installed package. I'm not calling it a "dependency" since obviously it would rather be "python3-typing" that depends on "python3". Barry, do you know? But even if Debian installs python3-typing by default, will "pip install -U typing" be possible in this scenario? I guess it wouldn't be terrible if that only worked in virtualenvs, although ideally it would work also for the raw host installation. - Ł
On 6 November 2017 at 16:42, Lukasz Langa <lukasz@langa.pl> wrote:
Fedora just lives with the circular dependency between python3 and python3-pip, which ensures both are installed by default (this arrangement allows "python3 -m venv" to still install pip, without actually needing a second copy of pip inside the python3 package) A bundled typing module that "python -m venv" installed by default would probably require similar treatment.
But even if Debian installs python3-typing by default, will "pip install -U typing" be possible in this scenario? I guess it wouldn't be terrible if that only worked in virtualenvs, although ideally it would work also for the raw host installation.
"sudo pip install <anything>" remains a terrible idea on any distro, because it leads to pip and the system package manager getting into a fight over which tool is responsible for managing the affected files. "pip install --user --upgrade typing" should work OK, though. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 6 November 2017 at 06:42, Lukasz Langa <lukasz@langa.pl> wrote:
It's also a problem for virtual environments, which won't include typing by default, and don't see the system installed packages. As Nick says, we could modify venv to always include typing, but most users, and all tools (tox, pew, pipenv, etc) use virtualenv rather than venv still, so this would be of very limited benefit.
I wonder if there's a way in .deb to specify a required installed package. I'm not calling it a "dependency" since obviously it would rather be "python3-typing" that depends on "python3". Barry, do you know?
There's always "include typing in the standard library" :-)
But even if Debian installs python3-typing by default, will "pip install -U typing" be possible in this scenario? I guess it wouldn't be terrible if that only worked in virtualenvs, although ideally it would work also for the raw host installation.
It would push yet more people into doing "sudo pip install -U typing", which is something we've been trying really hard to encourage them to stop doing over on the pip issues list. We strongly recommend using system supplied packages, but I'd be surprised if Debian is any more able to keep up with the latest versions of typing than Python core is. Paul
On Saturday, 4 November 2017 20:22:25 GMT Guido van Rossum wrote:
Is the outcome you want that you ship a version of typing with the python kit, but if you install from pip it overrides the one shipped in the python kit? That would be a matter of being having a suitable sys.path/site config I guess. pip folder before the bundled packages folder. If this is a mechanism that python kitting has then you would be able to bundle other packages like requests or six as well as typing, but because you can use pip to override the one shipped a user can optionally keep up with the latest versions. Barry
On Mon, Nov 6, 2017 at 9:29 AM, Barry Scott <barry@barrys-emacs.org> wrote:
If this were to happen, I would be inclined to put these "bootstrap" modules into their own directory in sys.path, after the rest of the stdlib. Then someone who's paranoid about stdlib shadowing could put pip-installed modules after the bulk of the stdlib (thus preventing any third-party package from overriding "import random", for instance) but still update modules that are specifically intended for updating; plus it'd mean you can get a directory listing of that, and go grab all the "blessed by python.org as an extension of the stdlib" packages. ChrisA
On 6 November 2017 at 09:08, Chris Angelico <rosuav@gmail.com> wrote:
When we say bundled, we mean bundled: the exact same bits you would get from PyPI, installed the same way, and if you upgrade it system wide, there's no trace of the one we bundled. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 11/06/2017 06:48 AM, brent bejot wrote:
On Mon, Nov 6, 2017 at 9:21 AM, Gyro Funch wrote:
Yes. Stability is probably the first (knowing what is support in a particular version). Second would be that nearly all modules in the stdlib are stable and don't need frequent updates -- in fact, I suspect typing is the only exception. -- ~Ethan~
Yes: https://www.python.org/dev/peps/pep-0206/#batteries-included-philosophy I can't find it but there is a great story about a guy needing to do a source code analysis in total isolation (not even a usb key or a cd) and the computer has nothing installed but Python. He gets by because Python is very useful "as is". I can say the same. When I log in on bare debian / centos or if I'm on a locked mac os. When i'm on a place or in a remote place in Africa. When I'm with customers locking down all the network with NDA, security badge and paranoia for desert... I can count on Python to be there for me and hashlib the stuff, etree thingy, zipfile the foo and uuid all the bars. It's a major feature. I can live without typing, but I would hate having to pip install a json parser.
Guido van Rossum wrote:
With my downstream consumer hat on, *if* we go with a bundling solution, please make sure it's easy to unbundle. Downstream consumers may need to do this for policy, security, etc. reasons, so making this too difficult will mean delays in releases, convoluted hacks, policy violations, or all of the above. See as reference some of the hoops Debian has to go through to handle ensurepip. Cheers, -Barry
On Mon, Nov 6, 2017 at 5:27 PM, Barry Warsaw <barry@python.org> wrote:
Having skimmed this thread and some others, I am going in the opposite direction. Moving typing.py out of the stdlib and then bundling it will cause too many problems, both psychological and technical. Therefore I am withdrawing my idea of moving typing.py out of the stdlib, and replacing it with an extension of its (and PEP 484's) provisional status by at least one release cycle. This means that at least for the duration of 3.7 we'll be able to update typing.py pretty aggressively in bugfix releases. See https://github.com/python/peps/pull/451 -- basically we only need a restricted interpretation of provisional, where we can add new features, but won't break backwards compatibility *for documented aspects of the API*. With PEP 560, we can then attempt to speed up the import of typing.py (hopefully 10x by the time 3.8 rolls around). I propose that the schedule for typing.py's provisional status be added to PEP 563 (the __future__ import for annotations); I've got a PR with tentative update to PEP 484 ready (https://github.com/python/peps/pull/451). -- --Guido van Rossum (python.org/~guido)
participants (14)
-
Antoine Pitrou -
Antoine Pitrou -
Barry Scott -
Barry Warsaw -
brent bejot -
Chris Angelico -
Ethan Furman -
Guido van Rossum -
Gyro Funch -
Lukasz Langa -
Michel Desmoulin -
Nick Coghlan -
Paul Moore -
Stéfane Fermigier