On Sat, Nov 4, 2017 at 1:49 AM, Guido van Rossum wrote:

On Fri, Nov 3, 2017 at 7:45 AM, Chris Angelico wrote:

...

On Sat, Nov 4, 2017 at 1:35 AM, Guido van Rossum wrote:

...

[A copy from https://github.com/python/typing/issues/495 to get more people's attention to this issue.]

I'm wondering if we should remove typing from the stdlib. Now's the time to think about this, as the feature freeze for 3.7 is about 12 weeks away.

Cons:

People have to depend on a PyPI package to use typing (but they do anyway for typing_extensions)

If the lazy evaluation of annotations (PEP 563) also lands in 3.7, then this would be a very minor downside. You'd need to pip-install typing as well as mypy *for the actual type checking*, but at run time, you could ignore both (all those List[...] annotations would be stored unevaluated). Otherwise, it'd mean that any project that makes use of type hints would require typing as a run-time dependency.

This would not work if you use TypeVar, NewType, or any kind of type alias involving things imported from typing (e.g. Union or TypedDict). Also cast().

Ah, I forgot about those - they're not just used in the annotations. Oh well, was a nice idea. ChrisA

On 5 November 2017 at 06:22, Guido van Rossum wrote:

On Sat, Nov 4, 2017 at 7:05 AM, Nick Coghlan wrote:

...

Perhaps typing could switch to being a bundled module, such that it had its own version, independent of the Python standard library version, but was still present by default in new installations?

This is beginning to sound like the most attractive solution. We could possibly do away with typing_extensions. Are there precedents of how to bundle a module in this way? Or is it going to be another special case like pip?

You'd be blazing new trails, but you'd be blazing them with a single-file pure Python module without any binary dependencies outside CPython itself, which is the simplest possible precedent setting scenario we could hope for. I can think of a couple of possible approaches you could take, depending on how you want to manage development on the typing module itself. = External dev = * Typing moves entirely out of the CPython source tree, with a separate issue tracker, repository, branch structure, etc (which you already have) * We add a "_bundled" directory to the CPython source tree in 3.7+ and put a typing sdist archive there * the release process PEP gains a "Confirm the bundled modules are up to date" step * the CPython build process is updated to use a venv to generate wheel files from bundled sdists * regrtest gains a "bundled" resource (and/or dedicated command line option) * when asked to test bundled modules, regrtest spins up a test venv, installs pip and the bundled modules, then runs the tests for those modules * ensurepip gains the ability to also install bundled wheel files * the Windows and Mac OS X installers start including the typing wheel file = In tree dev = * Similar to external dev for testing and distribution * Source code would stay in the main CPython tree and track CPython branch structure * The sdist file would be built by CPython's build process, rather than being checked in The external dev model is probably a better fit for your use case, though, since you'd like something that you can readily keep consistent across all supported 3.x versions (or at least 3.7+), and that's harder to do when you have multiple copies of the source code to keep in sync (vs having a single development branch where you run your CI testing across all supported Python versions). The external dev option also establishes a more useful precedent, since we could use it to officially migrate ongoing distutils maintenance into the setuptools project, and then bring it back via bundling of setuptools. Similarly, PEP 434 (which already exempts IDLE from the usual "No new features in maintenance releases" rules) could potentially be switched over to handling IDLE as an independently updated bundled application. The above idea is somewhat similar to what I suggested for the recommended modules list in https://mail.python.org/pipermail/python-ideas/2017-October/047508.html, but the difference is that to simplify the testing and installer building process, we'd bundle a suitable source archive in the git repo, rather than requiring PyPI access. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 5 November 2017 at 10:48, Antoine Pitrou wrote:

On Sun, 5 Nov 2017 13:46:59 +1000 Nick Coghlan wrote:

...

* ensurepip gains the ability to also install bundled wheel files

Why? Why wouldn't you put the wheel directly in site-packages on install?

I'm not quite sure what you mean? It needs to be "installed", in the sense of being unpacked into site-packages, and the ensurepip mechanism is already able to do that for pip and setuptools, so adding an extra wheel to install wouldn't be too hard. If we don't install like that, people won't be able to easily upgrade typing by just using "pip install --upgrade typing". Wheels don't support simply being added to sys.path the way that eggs did, if that's what you mean. Paul

Le 05/11/2017 à 14:30, Paul Moore a écrit :

On 5 November 2017 at 10:48, Antoine Pitrou wrote:

...

On Sun, 5 Nov 2017 13:46:59 +1000 Nick Coghlan wrote:

...

* ensurepip gains the ability to also install bundled wheel files

Why? Why wouldn't you put the wheel directly in site-packages on install?

I'm not quite sure what you mean? It needs to be "installed", in the sense of being unpacked into site-packages, and the ensurepip mechanism is already able to do that for pip and setuptools, so adding an extra wheel to install wouldn't be too hard.

Ok, perhaps my question wasn't quite clear. Are you suggesting that people have to run "python -m ensurepip typing" after they installed Python? Or would the typing module be importable as soon as you have an installed Python, like stdlib modules are? Regards Antoine.

On Sun, 5 Nov 2017 18:22:11 +0000 Paul Moore wrote:

On 5 November 2017 at 14:47, Antoine Pitrou wrote:

...

Le 05/11/2017 à 14:30, Paul Moore a écrit :

...

On 5 November 2017 at 10:48, Antoine Pitrou wrote:

...

On Sun, 5 Nov 2017 13:46:59 +1000 Nick Coghlan wrote:

...

* ensurepip gains the ability to also install bundled wheel files

Why? Why wouldn't you put the wheel directly in site-packages on install?

I'm not quite sure what you mean? It needs to be "installed", in the sense of being unpacked into site-packages, and the ensurepip mechanism is already able to do that for pip and setuptools, so adding an extra wheel to install wouldn't be too hard.

Ok, perhaps my question wasn't quite clear. Are you suggesting that people have to run "python -m ensurepip typing" after they installed Python? Or would the typing module be importable as soon as you have an installed Python, like stdlib modules are?

Ah, I get you now. I'd expect typing to be available exactly the same way as pip is. For me (on Windows) that means that it's available in a standard install. On Unix, I don't know (I think there's certain situations where you need to take extra steps in a custom build to ensure pip is available? I'd expect those steps to also install typing.

I think typing shouldn't require any extra typing (ha) on Unix either. I don't remember what the rationale was for having to type "python -m ensurepip" to get pip installed, but typing is just a library, not an executable tool that may be able to mess with the system state, so I don't think it's worthwhile introducing an extra step for it. Regards Antoine.

On 6 November 2017 at 05:00, Paul Moore wrote:

On 5 November 2017 at 18:40, Antoine Pitrou wrote:

...

I think typing shouldn't require any extra typing (ha) on Unix either. I don't remember what the rationale was for having to type "python -m ensurepip" to get pip installed, but typing is just a library, not an executable tool that may be able to mess with the system state, so I don't think it's worthwhile introducing an extra step for it.

Yes, I agree. I didn't realise that "give me pip" was an extra step on Unix. I don't know what the arguments for doing it like that on Unix were, but they certainly don't seem to apply to typing.

It's the default on Unix as well - you have to do "make install ENSUREPIP=no" to avoid getting it. (And some distros also modify their Python installations so that pip is missing by default) We provide "python -m ensurepip" because we offer ways to skip doing it at install time (and venv creation time), so the explicit command gives folks a straightforward way to change their minds later. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 5 Nov, 2017, at 10:30 PM, Michel Desmoulin wrote:

Le 06/11/2017 à 07:07, Nick Coghlan a écrit :

...

It's the default on Unix as well - you have to do "make install ENSUREPIP=no" to avoid getting it. (And some distros also modify their Python installations so that pip is missing by default)

On debian and derivatives (so Ubuntu) you need to install python-pip to be able to use pip.

Now it's annoying already. Because you have to write every tutorial to include a special case for them. But at least it's not a required step to run your program.

However, if you do code using type hints and typing is not installed, you can't even run the program without installing something. So we finally have Python 3 by default on most Linux system, but still would not be able to assume we can run a modern script on it.

This is a valid concern. Although this particular problem is self-inflicted by Debian, I can understand their rationale behind explicit packaging. They need to have control over the entire package graph. I wonder if there's a way in .deb to specify a required installed package. I'm not calling it a "dependency" since obviously it would rather be "python3-typing" that depends on "python3". Barry, do you know? But even if Debian installs python3-typing by default, will "pip install -U typing" be possible in this scenario? I guess it wouldn't be terrible if that only worked in virtualenvs, although ideally it would work also for the raw host installation. - Ł

On 6 November 2017 at 06:42, Lukasz Langa wrote:

...

Now it's annoying already. Because you have to write every tutorial to include a special case for them. But at least it's not a required step to run your program.

However, if you do code using type hints and typing is not installed, you can't even run the program without installing something. So we finally have Python 3 by default on most Linux system, but still would not be able to assume we can run a modern script on it.

This is a valid concern. Although this particular problem is self-inflicted by Debian, I can understand their rationale behind explicit packaging. They need to have control over the entire package graph.

It's also a problem for virtual environments, which won't include typing by default, and don't see the system installed packages. As Nick says, we could modify venv to always include typing, but most users, and all tools (tox, pew, pipenv, etc) use virtualenv rather than venv still, so this would be of very limited benefit.

I wonder if there's a way in .deb to specify a required installed package. I'm not calling it a "dependency" since obviously it would rather be "python3-typing" that depends on "python3". Barry, do you know?

There's always "include typing in the standard library" :-)

But even if Debian installs python3-typing by default, will "pip install -U typing" be possible in this scenario? I guess it wouldn't be terrible if that only worked in virtualenvs, although ideally it would work also for the raw host installation.

It would push yet more people into doing "sudo pip install -U typing", which is something we've been trying really hard to encourage them to stop doing over on the pip issues list. We strongly recommend using system supplied packages, but I'd be surprised if Debian is any more able to keep up with the latest versions of typing than Python core is. Paul

On Mon, 6 Nov 2017 16:07:56 +1000 Nick Coghlan wrote:

On 6 November 2017 at 05:00, Paul Moore wrote:

...

On 5 November 2017 at 18:40, Antoine Pitrou wrote:

...

I think typing shouldn't require any extra typing (ha) on Unix either. I don't remember what the rationale was for having to type "python -m ensurepip" to get pip installed, but typing is just a library, not an executable tool that may be able to mess with the system state, so I don't think it's worthwhile introducing an extra step for it.

Yes, I agree. I didn't realise that "give me pip" was an extra step on Unix. I don't know what the arguments for doing it like that on Unix were, but they certainly don't seem to apply to typing.

It's the default on Unix as well - you have to do "make install ENSUREPIP=no" to avoid getting it. (And some distros also modify their Python installations so that pip is missing by default)

Oh, thank you. I stand corrected then :-) Regards Antoine.

On Mon, Nov 6, 2017 at 9:29 AM, Barry Scott wrote:

On Saturday, 4 November 2017 20:22:25 GMT Guido van Rossum wrote:

...

On Sat, Nov 4, 2017 at 7:05 AM, Nick Coghlan wrote:

...

Perhaps typing could switch to being a bundled module, such that it had its own version, independent of the Python standard library version, but was still present by default in new installations?

This is beginning to sound like the most attractive solution. We could possibly do away with typing_extensions. Are there precedents of how to bundle a module in this way? Or is it going to be another special case like pip?

Is the outcome you want that you ship a version of typing with the python kit, but if you install from pip it overrides the one shipped in the python kit?

That would be a matter of being having a suitable sys.path/site config I guess. pip folder before the bundled packages folder.

If this is a mechanism that python kitting has then you would be able to bundle other packages like requests or six as well as typing, but because you can use pip to override the one shipped a user can optionally keep up with the latest versions.

If this were to happen, I would be inclined to put these "bootstrap" modules into their own directory in sys.path, after the rest of the stdlib. Then someone who's paranoid about stdlib shadowing could put pip-installed modules after the bulk of the stdlib (thus preventing any third-party package from overriding "import random", for instance) but still update modules that are specifically intended for updating; plus it'd mean you can get a directory listing of that, and go grab all the "blessed by python.org as an extension of the stdlib" packages. ChrisA

On Mon, Nov 6, 2017 at 5:06 PM, Nick Coghlan wrote:

On 6 November 2017 at 09:08, Chris Angelico wrote:

...

On Mon, Nov 6, 2017 at 9:29 AM, Barry Scott wrote:

...

If this is a mechanism that python kitting has then you would be able to bundle other packages like requests or six as well as typing, but because you can use pip to override the one shipped a user can optionally keep up with the latest versions.

If this were to happen, I would be inclined to put these "bootstrap" modules into their own directory in sys.path, after the rest of the stdlib. Then someone who's paranoid about stdlib shadowing could put pip-installed modules after the bulk of the stdlib (thus preventing any third-party package from overriding "import random", for instance) but still update modules that are specifically intended for updating; plus it'd mean you can get a directory listing of that, and go grab all the "blessed by python.org as an extension of the stdlib" packages.

When we say bundled, we mean bundled: the exact same bits you would get from PyPI, installed the same way, and if you upgrade it system wide, there's no trace of the one we bundled.

Oh, that's even better! :+1: ChrisA

On Mon, Nov 6, 2017 at 9:21 AM, Gyro Funch wrote:

On 11/4/2017 2:22 PM, Guido van Rossum wrote:

...

On Sat, Nov 4, 2017 at 7:05 AM, Nick Coghlan mailto:ncoghlan@gmail.com> wrote:

Perhaps typing could switch to being a bundled module, such that it had its own version, independent of the Python standard library version, but was still present by default in new installations?

This is beginning to sound like the most attractive solution. We could possibly do away with typing_extensions. Are there precedents of how to bundle a module in this way? Or is it going to be another special case like pip?

-- --Guido van Rossum (python.org/~guido http://python.org/~guido)

Would this mean that other packages in the stdlib with development cycles faster than those of python could use the same bundling mechanism?

This is pretty much how I thought things worked for all built-in packages until this thread came up. Is there any reason to not do this for all of stdlib?

This is pretty much how I thought things worked for all built-in packages until this thread came up.  Is there any reason to not do this for all of stdlib?

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

Yes: https://www.python.org/dev/peps/pep-0206/#batteries-included-philosophy I can't find it but there is a great story about a guy needing to do a source code analysis in total isolation (not even a usb key or a cd) and the computer has nothing installed but Python. He gets by because Python is very useful "as is". I can say the same. When I log in on bare debian / centos or if I'm on a locked mac os. When i'm on a place or in a remote place in Africa. When I'm with customers locking down all the network with NDA, security badge and paranoia for desert... I can count on Python to be there for me and hashlib the stuff, etree thingy, zipfile the foo and uuid all the bars. It's a major feature. I can live without typing, but I would hate having to pip install a json parser.

On Mon, Nov 6, 2017 at 5:27 PM, Barry Warsaw wrote:

Guido van Rossum wrote:

...

This is beginning to sound like the most attractive solution. We could possibly do away with typing_extensions. Are there precedents of how to bundle a module in this way? Or is it going to be another special case like pip?

With my downstream consumer hat on, *if* we go with a bundling solution, please make sure it's easy to unbundle. Downstream consumers may need to do this for policy, security, etc. reasons, so making this too difficult will mean delays in releases, convoluted hacks, policy violations, or all of the above.

See as reference some of the hoops Debian has to go through to handle ensurepip.

Having skimmed this thread and some others, I am going in the opposite direction. Moving typing.py out of the stdlib and then bundling it will cause too many problems, both psychological and technical. Therefore I am withdrawing my idea of moving typing.py out of the stdlib, and replacing it with an extension of its (and PEP 484's) provisional status by at least one release cycle. This means that at least for the duration of 3.7 we'll be able to update typing.py pretty aggressively in bugfix releases. See https://github.com/python/peps/pull/451 -- basically we only need a restricted interpretation of provisional, where we can add new features, but won't break backwards compatibility *for documented aspects of the API*. With PEP 560, we can then attempt to speed up the import of typing.py (hopefully 10x by the time 3.8 rolls around). I propose that the schedule for typing.py's provisional status be added to PEP 563 (the __future__ import for annotations); I've got a PR with tentative update to PEP 484 ready (https://github.com/python/peps/pull/451). -- --Guido van Rossum (python.org/~guido)