Re: Support for import hooks for static, type checkers and IDEs
Aha, thanks for your reply, Bernat! It seems like it hasn't been sent to typing-sig, though. Hopefully, the automatic quotation will keep everyone on the same page. Your presentation on the topic is just what I needed. (and the terriers are adorable!)
Can you detail how you achieve this? As far as I'm aware pylance doesn't do this, am I right Eric?
Effectively, we scan through names in sys.builtin_module_names and binary modules found in sys.path, import them and try reconstructing a stub definition for each of them using the introspection facilities of the language. It's the second kind of stubs we operate with internally, in addition to .pyi stubs. These don't have type annotations, but e.g. preserve docstrings of the definitions. We also use them for navigation e.g. when one invokes "Go to definition" on a built-in function, such as "print". In theory, the same mechanism could be reused for modules that become available dynamically through import hooks, if it's what you're implying. On Mon, May 9, 2022 at 9:26 PM Bernat Gabor <gaborjbernat@gmail.com> wrote:
generate stubs for binary modules
Can you detail how you achieve this? As far as I'm aware pylance doesn't do this, am I right Eric?
When should this process be repeated? For .pth files, we rely on regular filesystem events from under site-packages. Am I right that STATIC_EXTRA.json should appear under "{package}-{version}.dist-info" directory, so we can expect that if it ever changes it will trigger another filesystem event in a known location?
Yes, in my vision, the file would be part of the dist-info, and as such trackable as any other file for changes.
Aren't editable installs, as they are, a mechanism too complicated just
for exposing the content of "src" directory in sys.path?
Exposing the src directory is what we historically did, and gets you 80% there, but is not actually a true representation of what the user would want in the other 20%. Editable installations at their core are a method to develop a library without needing to reinstall it after every change. In the trivial case, yes it can be achieved by just injecting the src folder onto the sys.path via some mechanism (PYTHONPATH, cwd change, PTH file, sitecustomize.py, etc). However, there are more complicated cases when this approach will not work:
- selective inclusion of what gets packaged (only package these business logic files) - selective exclusion of what gets packaged (don't package test files) - any kind of code generation during the package build (either C-extension or generate python code from a JSON schema)
The goal we want to achieve in the new editable installations is to allow supporting these additional use cases, even if with some limitations (e.g. updates to files get picked up automatically but adding/removing new files would still require a reinstallation - through tooling could hide this away). My presentation from PyCon covered some of these, see https://gaborbernat.github.io/pep-660-662-2022 (recording should also be available in a few weeks).
On Mon, May 9, 2022 at 9:45 AM Mikhail Golubev < mikhail.golubev@jetbrains.com> wrote:
Hi everyone! Sorry for joining the conversation *that* late.
In short, PyCharm does support the current means of editable installs (.pth files) and does it very similar to what Eric described for Pyright. Namely, we have an interpreter "introspection" phase when we dynamically evaluate the content of sys.path with a helper script, as well as generate stubs for binary modules, retrieve the list of installed packages, etc. Because extra roots registered with .pth files automatically become visible in sys.path, we recognize them without any special handling. Although I'm not a huge fan of the idea, it is possible to add an extra step in this phase for checking either some physical file with metadata, such as STATIC_EXTRA.json, suggested by Bernat, or even another runtime value in addition to sys.path.
Two questions, though.
When should this process be repeated? For .pth files, we rely on regular filesystem events from under site-packages. Am I right that STATIC_EXTRA.json should appear under "{package}-{version}.dist-info" directory, so we can expect that if it ever changes it will trigger another filesystem event in a known location?
who prefer to put the library into a src folder and use editable
installations to manifest the content of the src folder to the import system (either via pth files, symlinks or import hooks).
This is a bit confusing to me. Aren't editable installs, as they are, a mechanism too complicated just for exposing the content of "src" directory in sys.path? I believe that existing tooling offers plenty of options for altering the content of sys.path before running an executable script or tests, e.g. in PyCharm we automatically add the directories marked as "source roots" in PYTHONPATH before launching project run configurations, in tox.ini it's possible to prepend PYTHONPATH with an extra directory before launching tests, etc. There must be something else editable installs are necessary for and what "flat" projects, without the "src" subfolder, cannot achieve automatically. Is it extra steps in a package's setup.py, or working with multi-package projects where a few libraries are being developed simultaneously and share the same virtual environment? Could someone give me more context, please?
On Sun, May 8, 2022 at 4:42 AM Bernat Gabor <gaborjbernat@gmail.com> wrote:
If there are serious discussions about deprecating ".pth" files, have
there been any proposals for a replacement mechanism? Bernat, do you think that the "STATIC_EXTRA.json" mechanism you have in mind could be that replacement?
No that's not my goal here. I think the primary reason people stay away from import hooks is that IDEs don't support them. I know many people in my work team spoke up against adopting PEP-660 when they found out it would lose auto-imports. Very-very few people take the effort to work on improving the existing systems. The majority just work around the problem by using the old solutions (e.g. PTH) file with all their problems. Every so often they ran into an issue caused by it, but then just blame wasting a few hours for not having better solutions available. I think the same is true with loading python modules from a database. Bigger companies might have a proprietary custom plugin that works around the issue, and the rest just live with what they have, they are well aware that if company Y wants to load code from DB X pylance adding support for it is likely to close to zero. After all, it's a custom DB and custom checker.
Finally, for me personally as a packaging tools maintainer, it would be very important that what we come up here does not impact the interpreter at runtime (that should keep using the dynamic import hook). However, for ease of development of libraries, I'd like libraries to have a mechanism to feed their snapshot of dynamic code to the static checker. Otherwise extending and writing a library would become even harder, and we're already struggling with finding maintainers for projects. I'm OK with this better solution taking a longer time than our current workaround ones (after all until we get there we can keep using the current workarounds).
Bernat
On Sat, May 7, 2022 at 7:26 PM Bernat Gabor <gaborjbernat@gmail.com> wrote:
Sadly, while the db/dynamic code stub files can be used to work around the problem; for files that live on the disk just not under the site-packages folder, they're less useful.
What I mean that if I have a SRC folder on the disk and I'm using import hooks to expose them to the python interpreter as they would be installed into the purelib/platlib locations I cannot use the stub mechanism to expose these files to the static checkers. At best I could take all python files and copy them via a pyi files to a stub package. I'd like to get to a world where editable installations (even when implemented via import hooks) mostly work in IDEs. This will make significantly easier the life of these library developers. And while I agree users of the library are likely two order of magnitude bigger than maintainers, I think we can also agree that for a sustainable ecosystem the life of maintainers should also be easy as possible. Maintaining a library is already hard let's not make it harder by taking away the convenience of autocomplete in IDEs.
On Sat, 7 May 2022, 19:02 Guido van Rossum, <guido@python.org> wrote:
On Sat, May 7, 2022 at 17:30 Bernat Gabor <gaborjbernat@gmail.com> wrote:
TBH, if all we're trying to solve is editable installs, I am assuming > that those would be used by a vanishingly small proportion of users > (otherwise we're doing something else wrong with package installs), so I > would personally be okay if an editable install wasn't visible to my IDE or > type checker, or if those tools were to fall back to typeshed or a PEP 561 > stub package.
Generally, they are two types of people when developing libraries:
- who put everything in the current working directory and use the fact that the current working directory is added to the sys.path automatically - these people do not need editable installations as the lib appears to be installed without requiring installation, - who prefer to put the library into a src folder and use editable installations to manifest the content of the src folder to the import system (either via pth files, symlinks or import hooks).
In my experience, there's roughly a 60 to 40 divide in favour of the first. Especially more experienced/advanced users tend to prefer the src layout over the inline approach. Based on this I don't think it's accurate to say that it's used by a vanishingly small proportion of users. I personally use it for all my projects, and for why src layout can be better than the inline source see Paul Ganssle blog post here https://blog.ganssle.io/tag/tox.html explaining some of the problems (what's in your source directory is not necessarily what'll be put in your purelib/platlib during installation).
Ah, that’s not what I meant. I meant very few users are developing packages. E.g. there are millions of pandas users but few of them are modifying pandas. (I hope. :-)
Also, I do not think that what we want to solve is editable
installations per se. What I'd like to solve here is supporting import hooks for static checkers. Editable installations are just a good example of where import hooks can be used.
Agreed.
Sadly, while the db/dynamic code stub files can be used to work around
the problem; for disks that live on the disk just not under the site-packages folder, they're less useful.
Not sure I follow.
—Guido
All the best,
Bernat
On Sat, May 7, 2022 at 6:11 PM Guido van Rossum <guido@python.org> wrote:
> On Sat, May 7, 2022 at 4:14 PM Eric Traut <eric@traut.com> wrote: > >> Most static analysis tools already support “.pth” files. Jia >> mentioned that pyre doesn’t yet, but it should be straightforward for them >> to add this. My recommendation is that we rally behind “.pth” files. I >> think this approach is more pragmatic than introducing yet another >> mechanism that needs to be implemented by all of the tools in the ecosystem. >> > > Hm. You're the first person I've met in a long time who regards PTH > files as something we should do more of -- IIUC they are almost universally > considered a necessary evil. Especially the fact that they can be used to > do other stuff than add entries to sys.path. But also that -- a long > sys.path tends to slow down module search, and it seems just wrong that > adding N packages would append N sys.path entries: if we wanted each > package to have its own entry we should have designed it as a mapping, not > a search path. > > TBH, if all we're trying to solve is editable installs, I am > assuming that those would be used by a vanishingly small proportion of > users (otherwise we're doing something else wrong with package installs), > so I would personally be okay if an editable install wasn't visible to my > IDE or type checker, or if those tools were to fall back to typeshed or a > PEP 561 stub package. > > Regarding your claim that almost nobody loads code from a database, > apparently that's actually very popular in the banking sector ( > https://calpaterson.com/bank-python.html) -- and those people have > a *lot* of Python code. > > I believe that VS Code is using (or going to use, or may use?) a > Virtual Filesystem to access user code. Would it really be so terrible if > there were some corner of the VFS namespace to map to the Python module > (spec) namespace? It could be served from a Python process that can give > you the source of a module given its full name (without executing the > module, of course -- the import loader machinery should be able to > guarantee that already). > > -- > --Guido van Rossum (python.org/~guido) > *Pronouns: he/him **(why is my pronoun here?)* > <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/> > _______________________________________________ > Typing-sig mailing list -- typing-sig@python.org > To unsubscribe send an email to typing-sig-leave@python.org > https://mail.python.org/mailman3/lists/typing-sig.python.org/ > Member address: gaborjbernat@gmail.com > --
--Guido (mobile)
_______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: mikhail.golubev@jetbrains.com
-- Mikhail Golubev Software Developer JetBrains http://www.jetbrains.com The drive to develop
-- Mikhail Golubev Software Developer JetBrains http://www.jetbrains.com The drive to develop
Yeah, while that could be reused it does not seem viable for other static checkers that don't want to run the python interpreter and import modules. So for general usability, I think we're still back to some form of extra static paths loaded by the static checker (but ignored by the runtime interpreter, that keeps using the import hooks). Also, this solution would also provide type annotations which would be an improvement. PS. You could in theory also get type annotations during this runtime inspection, I guess you don't do it for now as this mechanism only is used on binary modules that don't have type annotations. On Wed, May 11, 2022 at 5:54 AM Mikhail Golubev < mikhail.golubev@jetbrains.com> wrote:
Aha, thanks for your reply, Bernat! It seems like it hasn't been sent to typing-sig, though. Hopefully, the automatic quotation will keep everyone on the same page. Your presentation on the topic is just what I needed. (and the terriers are adorable!)
Can you detail how you achieve this? As far as I'm aware pylance doesn't do this, am I right Eric?
Effectively, we scan through names in sys.builtin_module_names and binary modules found in sys.path, import them and try reconstructing a stub definition for each of them using the introspection facilities of the language. It's the second kind of stubs we operate with internally, in addition to .pyi stubs. These don't have type annotations, but e.g. preserve docstrings of the definitions. We also use them for navigation e.g. when one invokes "Go to definition" on a built-in function, such as "print". In theory, the same mechanism could be reused for modules that become available dynamically through import hooks, if it's what you're implying.
On Mon, May 9, 2022 at 9:26 PM Bernat Gabor <gaborjbernat@gmail.com> wrote:
generate stubs for binary modules
Can you detail how you achieve this? As far as I'm aware pylance doesn't do this, am I right Eric?
When should this process be repeated? For .pth files, we rely on regular filesystem events from under site-packages. Am I right that STATIC_EXTRA.json should appear under "{package}-{version}.dist-info" directory, so we can expect that if it ever changes it will trigger another filesystem event in a known location?
Yes, in my vision, the file would be part of the dist-info, and as such trackable as any other file for changes.
Aren't editable installs, as they are, a mechanism too complicated just
for exposing the content of "src" directory in sys.path?
Exposing the src directory is what we historically did, and gets you 80% there, but is not actually a true representation of what the user would want in the other 20%. Editable installations at their core are a method to develop a library without needing to reinstall it after every change. In the trivial case, yes it can be achieved by just injecting the src folder onto the sys.path via some mechanism (PYTHONPATH, cwd change, PTH file, sitecustomize.py, etc). However, there are more complicated cases when this approach will not work:
- selective inclusion of what gets packaged (only package these business logic files) - selective exclusion of what gets packaged (don't package test files) - any kind of code generation during the package build (either C-extension or generate python code from a JSON schema)
The goal we want to achieve in the new editable installations is to allow supporting these additional use cases, even if with some limitations (e.g. updates to files get picked up automatically but adding/removing new files would still require a reinstallation - through tooling could hide this away). My presentation from PyCon covered some of these, see https://gaborbernat.github.io/pep-660-662-2022 (recording should also be available in a few weeks).
On Mon, May 9, 2022 at 9:45 AM Mikhail Golubev < mikhail.golubev@jetbrains.com> wrote:
Hi everyone! Sorry for joining the conversation *that* late.
In short, PyCharm does support the current means of editable installs (.pth files) and does it very similar to what Eric described for Pyright. Namely, we have an interpreter "introspection" phase when we dynamically evaluate the content of sys.path with a helper script, as well as generate stubs for binary modules, retrieve the list of installed packages, etc. Because extra roots registered with .pth files automatically become visible in sys.path, we recognize them without any special handling. Although I'm not a huge fan of the idea, it is possible to add an extra step in this phase for checking either some physical file with metadata, such as STATIC_EXTRA.json, suggested by Bernat, or even another runtime value in addition to sys.path.
Two questions, though.
When should this process be repeated? For .pth files, we rely on regular filesystem events from under site-packages. Am I right that STATIC_EXTRA.json should appear under "{package}-{version}.dist-info" directory, so we can expect that if it ever changes it will trigger another filesystem event in a known location?
who prefer to put the library into a src folder and use editable
installations to manifest the content of the src folder to the import system (either via pth files, symlinks or import hooks).
This is a bit confusing to me. Aren't editable installs, as they are, a mechanism too complicated just for exposing the content of "src" directory in sys.path? I believe that existing tooling offers plenty of options for altering the content of sys.path before running an executable script or tests, e.g. in PyCharm we automatically add the directories marked as "source roots" in PYTHONPATH before launching project run configurations, in tox.ini it's possible to prepend PYTHONPATH with an extra directory before launching tests, etc. There must be something else editable installs are necessary for and what "flat" projects, without the "src" subfolder, cannot achieve automatically. Is it extra steps in a package's setup.py, or working with multi-package projects where a few libraries are being developed simultaneously and share the same virtual environment? Could someone give me more context, please?
On Sun, May 8, 2022 at 4:42 AM Bernat Gabor <gaborjbernat@gmail.com> wrote:
If there are serious discussions about deprecating ".pth" files, have
there been any proposals for a replacement mechanism? Bernat, do you think that the "STATIC_EXTRA.json" mechanism you have in mind could be that replacement?
No that's not my goal here. I think the primary reason people stay away from import hooks is that IDEs don't support them. I know many people in my work team spoke up against adopting PEP-660 when they found out it would lose auto-imports. Very-very few people take the effort to work on improving the existing systems. The majority just work around the problem by using the old solutions (e.g. PTH) file with all their problems. Every so often they ran into an issue caused by it, but then just blame wasting a few hours for not having better solutions available. I think the same is true with loading python modules from a database. Bigger companies might have a proprietary custom plugin that works around the issue, and the rest just live with what they have, they are well aware that if company Y wants to load code from DB X pylance adding support for it is likely to close to zero. After all, it's a custom DB and custom checker.
Finally, for me personally as a packaging tools maintainer, it would be very important that what we come up here does not impact the interpreter at runtime (that should keep using the dynamic import hook). However, for ease of development of libraries, I'd like libraries to have a mechanism to feed their snapshot of dynamic code to the static checker. Otherwise extending and writing a library would become even harder, and we're already struggling with finding maintainers for projects. I'm OK with this better solution taking a longer time than our current workaround ones (after all until we get there we can keep using the current workarounds).
Bernat
On Sat, May 7, 2022 at 7:26 PM Bernat Gabor <gaborjbernat@gmail.com> wrote:
Sadly, while the db/dynamic code stub files can be used to work around the problem; for files that live on the disk just not under the site-packages folder, they're less useful.
What I mean that if I have a SRC folder on the disk and I'm using import hooks to expose them to the python interpreter as they would be installed into the purelib/platlib locations I cannot use the stub mechanism to expose these files to the static checkers. At best I could take all python files and copy them via a pyi files to a stub package. I'd like to get to a world where editable installations (even when implemented via import hooks) mostly work in IDEs. This will make significantly easier the life of these library developers. And while I agree users of the library are likely two order of magnitude bigger than maintainers, I think we can also agree that for a sustainable ecosystem the life of maintainers should also be easy as possible. Maintaining a library is already hard let's not make it harder by taking away the convenience of autocomplete in IDEs.
On Sat, 7 May 2022, 19:02 Guido van Rossum, <guido@python.org> wrote:
On Sat, May 7, 2022 at 17:30 Bernat Gabor <gaborjbernat@gmail.com> wrote:
> > TBH, if all we're trying to solve is editable installs, I am >> assuming that those would be used by a vanishingly small proportion of >> users (otherwise we're doing something else wrong with package installs), >> so I would personally be okay if an editable install wasn't visible to my >> IDE or type checker, or if those tools were to fall back to typeshed or a >> PEP 561 stub package. > > > Generally, they are two types of people when developing libraries: > > - who put everything in the current working directory and use > the fact that the current working directory is added to the sys.path > automatically - these people do not need editable installations as the lib > appears to be installed without requiring installation, > - who prefer to put the library into a src folder and use > editable installations to manifest the content of the src folder to the > import system (either via pth files, symlinks or import hooks). > > In my experience, there's roughly a 60 to 40 divide in favour of the > first. Especially more experienced/advanced users tend to prefer the src > layout over the inline approach. Based on this I don't think it's accurate > to say that it's used by a vanishingly small proportion of users. I > personally use it for all my projects, and for why src layout can be better > than the inline source see Paul Ganssle blog post here > https://blog.ganssle.io/tag/tox.html explaining some of the > problems (what's in your source directory is not necessarily what'll be put > in your purelib/platlib during installation). >
Ah, that’s not what I meant. I meant very few users are developing packages. E.g. there are millions of pandas users but few of them are modifying pandas. (I hope. :-)
Also, I do not think that what we want to solve is editable > installations per se. What I'd like to solve here is supporting import > hooks for static checkers. Editable installations are just a good example > of where import hooks can be used. >
Agreed.
Sadly, while the db/dynamic code stub files can be used to work > around the problem; for disks that live on the disk just not under the > site-packages folder, they're less useful. >
Not sure I follow.
—Guido
All the best, > > Bernat > > On Sat, May 7, 2022 at 6:11 PM Guido van Rossum <guido@python.org> > wrote: > >> On Sat, May 7, 2022 at 4:14 PM Eric Traut <eric@traut.com> wrote: >> >>> Most static analysis tools already support “.pth” files. Jia >>> mentioned that pyre doesn’t yet, but it should be straightforward for them >>> to add this. My recommendation is that we rally behind “.pth” files. I >>> think this approach is more pragmatic than introducing yet another >>> mechanism that needs to be implemented by all of the tools in the ecosystem. >>> >> >> Hm. You're the first person I've met in a long time who regards PTH >> files as something we should do more of -- IIUC they are almost universally >> considered a necessary evil. Especially the fact that they can be used to >> do other stuff than add entries to sys.path. But also that -- a long >> sys.path tends to slow down module search, and it seems just wrong that >> adding N packages would append N sys.path entries: if we wanted each >> package to have its own entry we should have designed it as a mapping, not >> a search path. >> >> TBH, if all we're trying to solve is editable installs, I am >> assuming that those would be used by a vanishingly small proportion of >> users (otherwise we're doing something else wrong with package installs), >> so I would personally be okay if an editable install wasn't visible to my >> IDE or type checker, or if those tools were to fall back to typeshed or a >> PEP 561 stub package. >> >> Regarding your claim that almost nobody loads code from a database, >> apparently that's actually very popular in the banking sector ( >> https://calpaterson.com/bank-python.html) -- and those people have >> a *lot* of Python code. >> >> I believe that VS Code is using (or going to use, or may use?) a >> Virtual Filesystem to access user code. Would it really be so terrible if >> there were some corner of the VFS namespace to map to the Python module >> (spec) namespace? It could be served from a Python process that can give >> you the source of a module given its full name (without executing the >> module, of course -- the import loader machinery should be able to >> guarantee that already). >> >> -- >> --Guido van Rossum (python.org/~guido) >> *Pronouns: he/him **(why is my pronoun here?)* >> <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/> >> > _______________________________________________ >> Typing-sig mailing list -- typing-sig@python.org >> To unsubscribe send an email to typing-sig-leave@python.org >> https://mail.python.org/mailman3/lists/typing-sig.python.org/ >> Member address: gaborjbernat@gmail.com >> > -- --Guido (mobile)
_______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: mikhail.golubev@jetbrains.com
-- Mikhail Golubev Software Developer JetBrains http://www.jetbrains.com The drive to develop
-- Mikhail Golubev Software Developer JetBrains http://www.jetbrains.com The drive to develop
Hi All, I'm new to this list so forgive me if this has been previously discussed. Is there any work currently towards allowing mapping dependent types of variadic generic TypeVarTuples? For instance, consider this function: def pair_with_ids(*args): return tuple((id(x), x) for x in args) The single-arg version of this function I could type as: T = TypeVar('T') def pair_with_ids(x: T) -> tuple[tuple[int, T]]: return ((id(x), x),) Something similar could be done for any fixed number of parameters, e.g. T1 = TypeVar('T1') T2 = TypeVar('T2') def pair_with_ids(x1: T2, x2: T2) -> tuple[tuple[int, T1], tuple[int, T2]]: return ((id(x1), x1), (id(x2), x2)) I would like to be able to type the whole thing at once with something like: Ts = TypeVarTuple('Ts') def pair_with_ids(*args: *Ts) -> tuple[*tuple[int, Ts]]: return tuple((id(x), x) for x in args) I realize this breaks the "TypeVarTuple is always starred" rule, but I'm wondering if any kind of dynamic type mapping like this has been considered before. The idea here being that you always need a star applying to a TypeVarTuple but not necessarily _immediately_ before. The star could be applied to any generic that has a single free type variable filled by a TypeVarTuple. Thanks, James
Hey James! I think this is the functionality provided by the Map operator that we included in early drafts of PEP 646, but removed to keep the size of the PEP manageable. We intend to introduce this in a future PEP - a (very) early draft of which is available here <https://docs.google.com/document/d/1szTVcFyLznoDT7phtT-6Fpvp27XaBw9DmbTLHrB6BE4/edit> . Best, Matthew On Wed, 11 May 2022 at 20:05, James Murphy via Typing-sig < typing-sig@python.org> wrote:
Hi All,
I'm new to this list so forgive me if this has been previously discussed.
Is there any work currently towards allowing mapping dependent types of variadic generic TypeVarTuples? For instance, consider this function:
def pair_with_ids(*args): return tuple((id(x), x) for x in args)
The single-arg version of this function I could type as:
T = TypeVar('T')
def pair_with_ids(x: T) -> tuple[tuple[int, T]]: return ((id(x), x),)
Something similar could be done for any fixed number of parameters, e.g.
T1 = TypeVar('T1')T2 = TypeVar('T2')def pair_with_ids(x1: T2, x2: T2) -> tuple[tuple[int, T1], tuple[int, T2]]: return ((id(x1), x1), (id(x2), x2))
I would like to be able to type the whole thing at once with something like:
Ts = TypeVarTuple('Ts')
def pair_with_ids(*args: *Ts) -> tuple[*tuple[int, Ts]]: return tuple((id(x), x) for x in args)
I realize this breaks the "TypeVarTuple is always starred" rule, but I'm wondering if any kind of dynamic type mapping like this has been considered before.
The idea here being that you always need a star applying to a TypeVarTuple but not necessarily _immediately_ before. The star could be applied to any generic that has a single free type variable filled by a TypeVarTuple.
Thanks,
James
_______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: mrahtz@google.com
On Thu, May 12, 2022, at 6:01 AM, Matthew Rahtz via Typing-sig wrote:
Hey James!
I think this is the functionality provided by the Map operator that we included in early drafts of PEP 646, but removed to keep the size of the PEP manageable. We intend to introduce this in a future PEP - a (very) early draft of which is available here <https://docs.google.com/document/d/1szTVcFyLznoDT7phtT-6Fpvp27XaBw9DmbTLHrB6BE4/edit>.
wow this is exciting. In SQLAlchemy , we were not able to make any use of pep-646 due to mapping and easy interoperability with Tuple not being present. I was wondering, will this "map" operator allow for the production of arbitrary Tuple types from a variable set of arguments? For an explicit example, see the series of overloads we are using at https://github.com/sqlalchemy/sqlalchemy/blob/9e7bed9df601ead02fd96bf2fc787b... where we must hardcode a series of overloads defining one to ten arguments. This pattern can also be seen in projects such as SQLModel at https://github.com/tiangolo/sqlmodel/blob/4d200517934c38b44c8ce372425a1db679... . in both of the above code examples, there are two things happening that originally I had hoped pep-646 would have solved, but does not. One is that we can easily map between different TypeVarTuple types without hardcoding for the number of elements. The other is that we can allow for Tuple-specific operations such as __getitem__ with an integer index working with types (that is, (1, 'foo')[0] is int and (1, 'foo')[1] is str). We currently need to use Tuple directly in order to have a working __getitem__ scheme, and the generated code approach seen in the links above in order to map from *varargs to tuples. I haven't looked at the spec yet but just putting out some things that I was dealing with in case they are helpful.
Best, Matthew
On Wed, 11 May 2022 at 20:05, James Murphy via Typing-sig <typing-sig@python.org> wrote:
Hi All,
I'm new to this list so forgive me if this has been previously discussed.
Is there any work currently towards allowing mapping dependent types of variadic generic TypeVarTuples? For instance, consider this function:
def pair_with_ids(*args): return tuple((id(x), x) for x in args)
The single-arg version of this function I could type as:
T = TypeVar('T')
def pair_with_ids(x: T) -> tuple[tuple[int, T]]: return ((id(x), x),)
Something similar could be done for any fixed number of parameters, e.g.
T1 = TypeVar('T1') T2 = TypeVar('T2')
def pair_with_ids(x1: T2, x2: T2) -> tuple[tuple[int, T1], tuple[int, T2]]: return ((id(x1), x1), (id(x2), x2))
I would like to be able to type the whole thing at once with something like:
Ts = TypeVarTuple('Ts')
def pair_with_ids(*args: *Ts) -> tuple[*tuple[int, Ts]]: return tuple((id(x), x) for x in args)
I realize this breaks the "TypeVarTuple is always starred" rule, but I'm wondering if any kind of dynamic type mapping like this has been considered before.
The idea here being that you always need a star applying to a TypeVarTuple but not necessarily _immediately_ before. The star could be applied to any generic that has a single free type variable filled by a TypeVarTuple.
Thanks,
James
_______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: mrahtz@google.com
Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: mike_mp@zzzcomputing.com
I was wondering, will this "map" operator allow for the production of arbitrary Tuple types from a variable set of arguments? For an explicit example, see the series of overloads we are using at https://github.com/sqlalchemy/sqlalchemy/blob/9e7bed9df601ead02fd96bf2fc787b... where we must hardcode a series of overloads defining one to ten arguments. This pattern can also be seen in projects such as SQLModel at https://github.com/tiangolo/sqlmodel/blob/4d200517934c38b44c8ce372425a1db679... .
Yes, you would be able to express these individual overloads as just one signature using the map operator. (We cut it out to keep the base PEP "simple".) On Wed, May 18, 2022 at 11:19 AM Mike Bayer <mike_mp@zzzcomputing.com> wrote:
On Thu, May 12, 2022, at 6:01 AM, Matthew Rahtz via Typing-sig wrote:
Hey James!
I think this is the functionality provided by the Map operator that we included in early drafts of PEP 646, but removed to keep the size of the PEP manageable. We intend to introduce this in a future PEP - a (very) early draft of which is available here <https://docs.google.com/document/d/1szTVcFyLznoDT7phtT-6Fpvp27XaBw9DmbTLHrB6BE4/edit> .
wow this is exciting. In SQLAlchemy , we were not able to make any use of pep-646 due to mapping and easy interoperability with Tuple not being present.
I was wondering, will this "map" operator allow for the production of arbitrary Tuple types from a variable set of arguments? For an explicit example, see the series of overloads we are using at https://github.com/sqlalchemy/sqlalchemy/blob/9e7bed9df601ead02fd96bf2fc787b... where we must hardcode a series of overloads defining one to ten arguments. This pattern can also be seen in projects such as SQLModel at https://github.com/tiangolo/sqlmodel/blob/4d200517934c38b44c8ce372425a1db679... .
in both of the above code examples, there are two things happening that originally I had hoped pep-646 would have solved, but does not. One is that we can easily map between different TypeVarTuple types without hardcoding for the number of elements. The other is that we can allow for Tuple-specific operations such as __getitem__ with an integer index working with types (that is, (1, 'foo')[0] is int and (1, 'foo')[1] is str). We currently need to use Tuple directly in order to have a working __getitem__ scheme, and the generated code approach seen in the links above in order to map from *varargs to tuples.
I haven't looked at the spec yet but just putting out some things that I was dealing with in case they are helpful.
Best, Matthew
On Wed, 11 May 2022 at 20:05, James Murphy via Typing-sig < typing-sig@python.org> wrote:
Hi All,
I'm new to this list so forgive me if this has been previously discussed.
Is there any work currently towards allowing mapping dependent types of variadic generic TypeVarTuples? For instance, consider this function:
def pair_with_ids(*args): return tuple((id(x), x) for x in args)
The single-arg version of this function I could type as:
T = TypeVar('T')
def pair_with_ids(x: T) -> tuple[tuple[int, T]]: return ((id(x), x),)
Something similar could be done for any fixed number of parameters, e.g.
T1 = TypeVar('T1')T2 = TypeVar('T2')def pair_with_ids(x1: T2, x2: T2) -> tuple[tuple[int, T1], tuple[int, T2]]: return ((id(x1), x1), (id(x2), x2))
I would like to be able to type the whole thing at once with something like:
Ts = TypeVarTuple('Ts')
def pair_with_ids(*args: *Ts) -> tuple[*tuple[int, Ts]]: return tuple((id(x), x) for x in args)
I realize this breaks the "TypeVarTuple is always starred" rule, but I'm wondering if any kind of dynamic type mapping like this has been considered before.
The idea here being that you always need a star applying to a TypeVarTuple but not necessarily _immediately_ before. The star could be applied to any generic that has a single free type variable filled by a TypeVarTuple.
Thanks,
James
_______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: mrahtz@google.com
_______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: mike_mp@zzzcomputing.com
_______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: gohanpra@gmail.com
-- S Pradeep Kumar
Has someone considered the following intuitive syntax using 'for ... in ...' instead of the Map operator? In my opinion, it's a bit more verbose, but also much more intuitive as well as powerful at the same time. Ts = TypeVarTuple('Ts') def pair_with_ids(*args: *Ts) -> tuple[tuple[int, T] for T in Ts]: ...
What would this evaluate to at runtime? tuple.__getitem__(<generator>)? AbyxDev; https://abyx.dev *From: *jakob.leben@gmail.com *Sent: *Tuesday, October 4, 2022 11:25 AM *To: *typing-sig@python.org *Subject: *[Typing-sig] Re: Dependent mapping of TypeVarTuples? Has someone considered the following intuitive syntax using 'for ... in ...' instead of the Map operator? In my opinion, it's a bit more verbose, but also much more intuitive as well as powerful at the same time. Ts = TypeVarTuple('Ts') def pair_with_ids(*args: *Ts) -> tuple[tuple[int, T] for T in Ts]: ... _______________________________________________ Typing-sig mailing list -- typing-sig@python.org To unsubscribe send an email to typing-sig-leave@python.org https://mail.python.org/mailman3/lists/typing-sig.python.org/ Member address: ken@abyx.dev
participants (8)
-
AbyxDev -
Bernat Gabor -
jakob.leben＠gmail.com -
James Murphy -
Matthew Rahtz -
Mike Bayer -
Mikhail Golubev -
S Pradeep Kumar