Concurrent.futures: no type discovery for PyCharm
I am using concurrent.futures to parallelize independent tasks on multiple cores once in a while. Each time I have a difficulty remembering the specific syntax and have to look it up in old code or google. I would much prefer to be able to find the source through the PyCharm and have autocompletion. It takes adding two lines to the __init__.py of concurrent.futures: (insert on line 19) from .process import ProcessPoolExecutor from .thread import ThreadPoolExecutor I would also guess that it would make the __getattr__ redundant? Am I missing something or can this change be done this way and would indeed be an improvement? Best Regards, -- Ilya Kamenshchikov
alright, so would an import under TYPE_CHECKING guard be an option? like: from typing import TYPE_CHECKING if TYPE_CHECKING: from .process import ProcessPoolExecutor from .thread import ThreadPoolExecutor Perhaps we can have both clarity and performance.
"import typing" is slow too. 2019年4月21日(日) 1:43 Ilya Kamenshchikov <ikamenshchikov@gmail.com>:
alright, so would an import under TYPE_CHECKING guard be an option? like:
from typing import TYPE_CHECKING if TYPE_CHECKING: from .process import ProcessPoolExecutor from .thread import ThreadPoolExecutor
Perhaps we can have both clarity and performance.
On 4/20/2019 2:08 PM, Inada Naoki wrote:
"import typing" is slow too.
2019年4月21日(日) 1:43 Ilya Kamenshchikov <ikamenshchikov@gmail.com <mailto:ikamenshchikov@gmail.com>>:
alright, so would an import under TYPE_CHECKING guard be an option? like:
from typingimport TYPE_CHECKING if TYPE_CHECKING: from .processimport ProcessPoolExecutor from .threadimport ThreadPoolExecutor
Perhaps we can have both clarity and performance.
How about: from faketyping import TYPE_CHECKING where faketyping.py: TYPE_CHECKING = None I don't know enough about how TYPE_CHECKING (or typing) is optionally enabled to come up with an exactly correct proposal.
On Sat, Apr 20, 2019 at 2:11 PM Inada Naoki <songofacandy@gmail.com> wrote:
"import typing" is slow too.
Many static analysis tools will also accept: TYPE_CHECKING = False if TYPE_CHECKING: ... At least mypy and pylint both treat all variables named TYPE_CHECKING as true, regardless of where they came from. I'm not sure if this is intentional or because they're cutting corners, but it works... -n -- Nathaniel J. Smith -- https://vorpus.org
On Sat, Apr 20, 2019 at 2:10 PM Inada Naoki <songofacandy@gmail.com> wrote:
"import typing" is slow too.
But is it so slow as to not do the right thing here and use the 'typing' module as expected? If you have so much work you need to launch some threads or processes to deal with it then a single import isn't going to be your biggest bottleneck. -Brett
2019年4月21日(日) 1:43 Ilya Kamenshchikov <ikamenshchikov@gmail.com>:
alright, so would an import under TYPE_CHECKING guard be an option? like:
from typing import TYPE_CHECKING if TYPE_CHECKING: from .process import ProcessPoolExecutor from .thread import ThreadPoolExecutor
Perhaps we can have both clarity and performance.
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org
I see the chicken and egg problem here. If we are talking about typing module usage -- typeshed is the type hints provider. If PyCharm doesn't want to use it -- it is not CPython problem. I think there is no need to change python code itself but used tooling. On Mon, Apr 22, 2019 at 11:06 PM Brett Cannon <brett@python.org> wrote:
On Sat, Apr 20, 2019 at 2:10 PM Inada Naoki <songofacandy@gmail.com> wrote:
"import typing" is slow too.
But is it so slow as to not do the right thing here and use the 'typing' module as expected? If you have so much work you need to launch some threads or processes to deal with it then a single import isn't going to be your biggest bottleneck.
-Brett
2019年4月21日(日) 1:43 Ilya Kamenshchikov <ikamenshchikov@gmail.com>:
alright, so would an import under TYPE_CHECKING guard be an option? like:
from typing import TYPE_CHECKING if TYPE_CHECKING: from .process import ProcessPoolExecutor from .thread import ThreadPoolExecutor
Perhaps we can have both clarity and performance.
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.co...
-- Thanks, Andrew Svetlov
On 22Apr2019 1521, Andrew Svetlov wrote:
I see the chicken and egg problem here. If we are talking about typing module usage -- typeshed is the type hints provider. If PyCharm doesn't want to use it -- it is not CPython problem.
I think there is no need to change python code itself but used tooling.
It's not typeshed related, it's most likely because Python 3.7 Lib/concurrent/future/__init__.py switched from always importing the subclasses to doing it lazily in a module __getattr__ function. I assume for performance, since either submodule may have deep import chains. Presumably PyCharm has not yet added support for this, and so it simply doesn't know how to resolve ThreadPoolExecutor or ProcessPoolExecutor without actually executing code (which most static analysers will hesitate to do, since you don't know if that code is "os.system('rm -rf /')" until it's too late). Perhaps for the sake of IDEs and static analysers we could make a policy for standard library modules to include an "if False:" or "TYPE_CHECKING = False; if TYPE_CHECKING:" block that includes the import statement when adding lazy module attribute resolution? Cheers, Steve
On Tue, Apr 23, 2019 at 4:40 AM Brett Cannon <brett@python.org> wrote:
On Sat, Apr 20, 2019 at 2:10 PM Inada Naoki <songofacandy@gmail.com> wrote:
"import typing" is slow too.
But is it so slow as to not do the right thing here and use the 'typing' module as expected?
I don't know it is not a "right thing" yet. It feel it is just a workaround for PyCharm at the moment. __dir__ and __all__ has ProcessPoolExecutor and ThreadPoolExecutor for interactive shell. So Python REPL can complete them. But we didn't discussed about "static hinting" version of __all__ in PEP 562. If we decide it's a "right way", we can update example code in PEP 562. But when we use lazy import, we want to make import faster. Adding more 3~5ms import time seems not so happy solution. Maybe, can we add TYPE_CHECKING=False in builtins?
If you have so much work you need to launch some threads or processes to deal with it then a single import isn't going to be your biggest bottleneck.
Importing futures module doesn't mean the app really need thread or processes. That's why we defer importing ThreadPoolExecutor and ProcessPoolExecutor. And people who want apps like vim starts quickly (~200ms), we want avoid every "significant overhead" as possible. Not only "the biggest bottleneck" is the problem. -- Inada Naoki <songofacandy@gmail.com>
I agree that `from typing import TYPE_CHECKING` is not desirable from the import time reduction perspective.
From my understanding code completion *can* be based on type hinting to avoid actual code execution. That's why I've mentioned that typeshed already has the correct type information.
if TYPE_CHECKING: import ... requires mypy modification. if False: import ... Works right now for stdlib (mypy ignores stdlib code but uses typeshed anyway) but looks a little cryptic. Requires a comprehensive comment at least. On Tue, Apr 23, 2019 at 1:59 AM Inada Naoki <songofacandy@gmail.com> wrote:
On Tue, Apr 23, 2019 at 4:40 AM Brett Cannon <brett@python.org> wrote:
On Sat, Apr 20, 2019 at 2:10 PM Inada Naoki <songofacandy@gmail.com> wrote:
"import typing" is slow too.
But is it so slow as to not do the right thing here and use the 'typing' module as expected?
I don't know it is not a "right thing" yet. It feel it is just a workaround for PyCharm at the moment.
__dir__ and __all__ has ProcessPoolExecutor and ThreadPoolExecutor for interactive shell. So Python REPL can complete them. But we didn't discussed about "static hinting" version of __all__ in PEP 562.
If we decide it's a "right way", we can update example code in PEP 562.
But when we use lazy import, we want to make import faster. Adding more 3~5ms import time seems not so happy solution.
Maybe, can we add TYPE_CHECKING=False in builtins?
If you have so much work you need to launch some threads or processes to deal with it then a single import isn't going to be your biggest bottleneck.
Importing futures module doesn't mean the app really need thread or processes. That's why we defer importing ThreadPoolExecutor and ProcessPoolExecutor.
And people who want apps like vim starts quickly (~200ms), we want avoid every "significant overhead" as possible. Not only "the biggest bottleneck" is the problem.
-- Inada Naoki <songofacandy@gmail.com> _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.co...
-- Thanks, Andrew Svetlov
On Tue, Apr 23, 2019, 05:09 Andrew Svetlov <andrew.svetlov@gmail.com> wrote:
I agree that `from typing import TYPE_CHECKING` is not desirable from the import time reduction perspective.
From my understanding code completion *can* be based on type hinting to avoid actual code execution. That's why I've mentioned that typeshed already has the correct type information.
if TYPE_CHECKING: import ...
requires mypy modification.
if False: import ...
Works right now for stdlib (mypy ignores stdlib code but uses typeshed anyway) but looks a little cryptic. Requires a comprehensive comment at least.
Last time I looked at this, I'm pretty sure `if False` broke at least one popular static analysis tool (ie it was clever enough to ignore everything inside `if False`) – I think either pylint or jedi? I'd suggest checking any clever hacks against at least: mypy, pylint/astroid, jedi, pyflakes, and pycharm. They all have their own static analysis engines, and each one has its own idiosyncratic quirks. We've struggled with this a *lot* in trio, and eventually ended up giving up on all forms of dynamic export cleverness; we've even banned the use of __all__ entirely. Static analysis has gotten good enough that users won't accept it not working, but it hasn't gotten good enough to handle anything but the simplest static exports in a reliable way: https://github.com/python-trio/trio/pull/316 https://github.com/python-trio/trio/issues/542 The stdlib has more leeway because when tools don't work on the stdlib then they tend to eventually add workarounds. I'm just saying, think twice before diving into clever hacks to workaround static analysis limits, and if you're going to do it then be careful to be thorough. You're basically relying on undocumented bugs, and it gets really messy really quickly. -n
In any case I think this should be filed (by the OP) as an issue against JetBrains' PyCharm issue tracker. Who knows they may be able to special-case this in a jiffy. I don't think we should add any clever hacks to the stdlib for this. On Tue, Apr 23, 2019 at 9:59 AM Nathaniel Smith <njs@pobox.com> wrote:
On Tue, Apr 23, 2019, 05:09 Andrew Svetlov <andrew.svetlov@gmail.com> wrote:
I agree that `from typing import TYPE_CHECKING` is not desirable from the import time reduction perspective.
From my understanding code completion *can* be based on type hinting to avoid actual code execution. That's why I've mentioned that typeshed already has the correct type information.
if TYPE_CHECKING: import ...
requires mypy modification.
if False: import ...
Works right now for stdlib (mypy ignores stdlib code but uses typeshed anyway) but looks a little cryptic. Requires a comprehensive comment at least.
Last time I looked at this, I'm pretty sure `if False` broke at least one popular static analysis tool (ie it was clever enough to ignore everything inside `if False`) – I think either pylint or jedi?
I'd suggest checking any clever hacks against at least: mypy, pylint/astroid, jedi, pyflakes, and pycharm. They all have their own static analysis engines, and each one has its own idiosyncratic quirks.
We've struggled with this a *lot* in trio, and eventually ended up giving up on all forms of dynamic export cleverness; we've even banned the use of __all__ entirely. Static analysis has gotten good enough that users won't accept it not working, but it hasn't gotten good enough to handle anything but the simplest static exports in a reliable way: https://github.com/python-trio/trio/pull/316 https://github.com/python-trio/trio/issues/542
The stdlib has more leeway because when tools don't work on the stdlib then they tend to eventually add workarounds. I'm just saying, think twice before diving into clever hacks to workaround static analysis limits, and if you're going to do it then be careful to be thorough. You're basically relying on undocumented bugs, and it gets really messy really quickly.
-n _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him/his **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
How would we answer the same question if it was not a part of stdlib? I am not sure it is fair to expect of Pycharm to parse / execute the __getattr__ on modules, as more elaborate implementation could even contain different types per some condition at the runtime or anything at all. The code: TYPE_CHECKING = False if TYPE_CHECKING: from .process import ProcessPoolExecutor from .thread import ThreadPoolExecutor works for type checking in PyCharm and is fast. This is how stdlib can be an example to how side libraries can be implemented. If we can agree that this is the only clear, performant and sufficient code - then perhaps modifying mypy is a reasonable price to pay. Perhaps this particular case can be just patched locally by PyCharm /JetBrains, but what is a general solution to this class of problems? Best Regards, -- Ilya Kamenshchikov On Tue, Apr 23, 2019 at 7:05 PM Guido van Rossum <guido@python.org> wrote:
In any case I think this should be filed (by the OP) as an issue against JetBrains' PyCharm issue tracker. Who knows they may be able to special-case this in a jiffy. I don't think we should add any clever hacks to the stdlib for this.
On Tue, Apr 23, 2019 at 9:59 AM Nathaniel Smith <njs@pobox.com> wrote:
On Tue, Apr 23, 2019, 05:09 Andrew Svetlov <andrew.svetlov@gmail.com> wrote:
I agree that `from typing import TYPE_CHECKING` is not desirable from the import time reduction perspective.
From my understanding code completion *can* be based on type hinting to avoid actual code execution. That's why I've mentioned that typeshed already has the correct type information.
if TYPE_CHECKING: import ...
requires mypy modification.
if False: import ...
Works right now for stdlib (mypy ignores stdlib code but uses typeshed anyway) but looks a little cryptic. Requires a comprehensive comment at least.
Last time I looked at this, I'm pretty sure `if False` broke at least one popular static analysis tool (ie it was clever enough to ignore everything inside `if False`) – I think either pylint or jedi?
I'd suggest checking any clever hacks against at least: mypy, pylint/astroid, jedi, pyflakes, and pycharm. They all have their own static analysis engines, and each one has its own idiosyncratic quirks.
We've struggled with this a *lot* in trio, and eventually ended up giving up on all forms of dynamic export cleverness; we've even banned the use of __all__ entirely. Static analysis has gotten good enough that users won't accept it not working, but it hasn't gotten good enough to handle anything but the simplest static exports in a reliable way: https://github.com/python-trio/trio/pull/316 https://github.com/python-trio/trio/issues/542
The stdlib has more leeway because when tools don't work on the stdlib then they tend to eventually add workarounds. I'm just saying, think twice before diving into clever hacks to workaround static analysis limits, and if you're going to do it then be careful to be thorough. You're basically relying on undocumented bugs, and it gets really messy really quickly.
-n _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him/his **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
The general solution is import typing if typing.TYPE_CHECKING: <whatever> The hack of starting with TYPE_CHECKING = False happens to work but is not endorsed by PEP 484 so is not guaranteed for the future. Note that 3rd party code is rarely in such a critical part for script startup that the cost of `import typing` is too much. But the stdlib often *is* in the critical path for script startup, and some consider the time spent in that import too much (startup time should be in the order of tens of msec so every msec counts -- but once you start importing 3rd party code you basically can't make it that fast regardless). Anyway, the stdlib should almost never be used as an example for non-stdlib code -- there are many reasons for this that I don't want to have to repeat here. On Tue, Apr 23, 2019 at 12:33 PM Ilya Kamenshchikov < ikamenshchikov@gmail.com> wrote:
How would we answer the same question if it was not a part of stdlib? I am not sure it is fair to expect of Pycharm to parse / execute the __getattr__ on modules, as more elaborate implementation could even contain different types per some condition at the runtime or anything at all. The code:
TYPE_CHECKING = False if TYPE_CHECKING: from .process import ProcessPoolExecutor from .thread import ThreadPoolExecutor
works for type checking in PyCharm and is fast.
This is how stdlib can be an example to how side libraries can be implemented. If we can agree that this is the only clear, performant and sufficient code - then perhaps modifying mypy is a reasonable price to pay.
Perhaps this particular case can be just patched locally by PyCharm /JetBrains, but what is a general solution to this class of problems?
Best Regards, -- Ilya Kamenshchikov
On Tue, Apr 23, 2019 at 7:05 PM Guido van Rossum <guido@python.org> wrote:
In any case I think this should be filed (by the OP) as an issue against JetBrains' PyCharm issue tracker. Who knows they may be able to special-case this in a jiffy. I don't think we should add any clever hacks to the stdlib for this.
On Tue, Apr 23, 2019 at 9:59 AM Nathaniel Smith <njs@pobox.com> wrote:
On Tue, Apr 23, 2019, 05:09 Andrew Svetlov <andrew.svetlov@gmail.com> wrote:
I agree that `from typing import TYPE_CHECKING` is not desirable from the import time reduction perspective.
From my understanding code completion *can* be based on type hinting to avoid actual code execution. That's why I've mentioned that typeshed already has the correct type information.
if TYPE_CHECKING: import ...
requires mypy modification.
if False: import ...
Works right now for stdlib (mypy ignores stdlib code but uses typeshed anyway) but looks a little cryptic. Requires a comprehensive comment at least.
Last time I looked at this, I'm pretty sure `if False` broke at least one popular static analysis tool (ie it was clever enough to ignore everything inside `if False`) – I think either pylint or jedi?
I'd suggest checking any clever hacks against at least: mypy, pylint/astroid, jedi, pyflakes, and pycharm. They all have their own static analysis engines, and each one has its own idiosyncratic quirks.
We've struggled with this a *lot* in trio, and eventually ended up giving up on all forms of dynamic export cleverness; we've even banned the use of __all__ entirely. Static analysis has gotten good enough that users won't accept it not working, but it hasn't gotten good enough to handle anything but the simplest static exports in a reliable way: https://github.com/python-trio/trio/pull/316 https://github.com/python-trio/trio/issues/542
The stdlib has more leeway because when tools don't work on the stdlib then they tend to eventually add workarounds. I'm just saying, think twice before diving into clever hacks to workaround static analysis limits, and if you're going to do it then be careful to be thorough. You're basically relying on undocumented bugs, and it gets really messy really quickly.
-n _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him/his **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him/his **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
Mypy doesn't use source code of stlib for analysis and instead uses stub files from typeshed. IIUC PyCharm can also do that (i.e. use the typeshed stubs). The whole idea of typeshed is to avoid changing stlib solely for the sake of static analysis. Please open an issue on typeshed an/or PyCharm tracker. -- Ivan On Tue, 23 Apr 2019 at 20:38, Ilya Kamenshchikov <ikamenshchikov@gmail.com> wrote:
How would we answer the same question if it was not a part of stdlib? I am not sure it is fair to expect of Pycharm to parse / execute the __getattr__ on modules, as more elaborate implementation could even contain different types per some condition at the runtime or anything at all. The code:
TYPE_CHECKING = False if TYPE_CHECKING: from .process import ProcessPoolExecutor from .thread import ThreadPoolExecutor
works for type checking in PyCharm and is fast.
This is how stdlib can be an example to how side libraries can be implemented. If we can agree that this is the only clear, performant and sufficient code - then perhaps modifying mypy is a reasonable price to pay.
Perhaps this particular case can be just patched locally by PyCharm /JetBrains, but what is a general solution to this class of problems?
Best Regards, -- Ilya Kamenshchikov
On Tue, Apr 23, 2019 at 7:05 PM Guido van Rossum <guido@python.org> wrote:
In any case I think this should be filed (by the OP) as an issue against JetBrains' PyCharm issue tracker. Who knows they may be able to special-case this in a jiffy. I don't think we should add any clever hacks to the stdlib for this.
On Tue, Apr 23, 2019 at 9:59 AM Nathaniel Smith <njs@pobox.com> wrote:
On Tue, Apr 23, 2019, 05:09 Andrew Svetlov <andrew.svetlov@gmail.com> wrote:
I agree that `from typing import TYPE_CHECKING` is not desirable from the import time reduction perspective.
From my understanding code completion *can* be based on type hinting to avoid actual code execution. That's why I've mentioned that typeshed already has the correct type information.
if TYPE_CHECKING: import ...
requires mypy modification.
if False: import ...
Works right now for stdlib (mypy ignores stdlib code but uses typeshed anyway) but looks a little cryptic. Requires a comprehensive comment at least.
Last time I looked at this, I'm pretty sure `if False` broke at least one popular static analysis tool (ie it was clever enough to ignore everything inside `if False`) – I think either pylint or jedi?
I'd suggest checking any clever hacks against at least: mypy, pylint/astroid, jedi, pyflakes, and pycharm. They all have their own static analysis engines, and each one has its own idiosyncratic quirks.
We've struggled with this a *lot* in trio, and eventually ended up giving up on all forms of dynamic export cleverness; we've even banned the use of __all__ entirely. Static analysis has gotten good enough that users won't accept it not working, but it hasn't gotten good enough to handle anything but the simplest static exports in a reliable way: https://github.com/python-trio/trio/pull/316 https://github.com/python-trio/trio/issues/542
The stdlib has more leeway because when tools don't work on the stdlib then they tend to eventually add workarounds. I'm just saying, think twice before diving into clever hacks to workaround static analysis limits, and if you're going to do it then be careful to be thorough. You're basically relying on undocumented bugs, and it gets really messy really quickly.
-n _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him/his **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/levkivskyi%40gmail.com
Ok thanks for explaining. I will proceed by trying it with typeshed. Best Regards, -- Ilya Kamenshchikov On Tue, Apr 23, 2019 at 9:44 PM Ivan Levkivskyi <levkivskyi@gmail.com> wrote:
Mypy doesn't use source code of stlib for analysis and instead uses stub files from typeshed. IIUC PyCharm can also do that (i.e. use the typeshed stubs). The whole idea of typeshed is to avoid changing stlib solely for the sake of static analysis. Please open an issue on typeshed an/or PyCharm tracker.
-- Ivan
On Tue, 23 Apr 2019 at 20:38, Ilya Kamenshchikov <ikamenshchikov@gmail.com> wrote:
How would we answer the same question if it was not a part of stdlib? I am not sure it is fair to expect of Pycharm to parse / execute the __getattr__ on modules, as more elaborate implementation could even contain different types per some condition at the runtime or anything at all. The code:
TYPE_CHECKING = False if TYPE_CHECKING: from .process import ProcessPoolExecutor from .thread import ThreadPoolExecutor
works for type checking in PyCharm and is fast.
This is how stdlib can be an example to how side libraries can be implemented. If we can agree that this is the only clear, performant and sufficient code - then perhaps modifying mypy is a reasonable price to pay.
Perhaps this particular case can be just patched locally by PyCharm /JetBrains, but what is a general solution to this class of problems?
Best Regards, -- Ilya Kamenshchikov
On Tue, Apr 23, 2019 at 7:05 PM Guido van Rossum <guido@python.org> wrote:
In any case I think this should be filed (by the OP) as an issue against JetBrains' PyCharm issue tracker. Who knows they may be able to special-case this in a jiffy. I don't think we should add any clever hacks to the stdlib for this.
On Tue, Apr 23, 2019 at 9:59 AM Nathaniel Smith <njs@pobox.com> wrote:
On Tue, Apr 23, 2019, 05:09 Andrew Svetlov <andrew.svetlov@gmail.com> wrote:
I agree that `from typing import TYPE_CHECKING` is not desirable from the import time reduction perspective.
From my understanding code completion *can* be based on type hinting to avoid actual code execution. That's why I've mentioned that typeshed already has the correct type information.
if TYPE_CHECKING: import ...
requires mypy modification.
if False: import ...
Works right now for stdlib (mypy ignores stdlib code but uses typeshed anyway) but looks a little cryptic. Requires a comprehensive comment at least.
Last time I looked at this, I'm pretty sure `if False` broke at least one popular static analysis tool (ie it was clever enough to ignore everything inside `if False`) – I think either pylint or jedi?
I'd suggest checking any clever hacks against at least: mypy, pylint/astroid, jedi, pyflakes, and pycharm. They all have their own static analysis engines, and each one has its own idiosyncratic quirks.
We've struggled with this a *lot* in trio, and eventually ended up giving up on all forms of dynamic export cleverness; we've even banned the use of __all__ entirely. Static analysis has gotten good enough that users won't accept it not working, but it hasn't gotten good enough to handle anything but the simplest static exports in a reliable way: https://github.com/python-trio/trio/pull/316 https://github.com/python-trio/trio/issues/542
The stdlib has more leeway because when tools don't work on the stdlib then they tend to eventually add workarounds. I'm just saying, think twice before diving into clever hacks to workaround static analysis limits, and if you're going to do it then be careful to be thorough. You're basically relying on undocumented bugs, and it gets really messy really quickly.
-n _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org
-- --Guido van Rossum (python.org/~guido) *Pronouns: he/him/his **(why is my pronoun here?)* <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/levkivskyi%40gmail.com
participants (9)
-
Andrew Svetlov -
Brett Cannon -
Glenn Linderman -
Guido van Rossum -
Ilya Kamenshchikov -
Inada Naoki -
Ivan Levkivskyi -
Nathaniel Smith -
Steve Dower