I love many of the ancillary tools that help improve the quality of my
Python code, including flake8, coverage, and mypy. Each of these
usually produces some great diagnostics, but none of them are perfect,
so they also produce false positives that have to be suppressed.
Each of these tools supports "pragmas", i.e. comments you can add to a
line of code to suppress a warning. E.g. with flake8 I can say:
import readline, rlcompleter # noqa: F401
to suppress the flake8 warnings about unused imports. These modules
work by side-effect, which of course a static analyzer can't know.
Similarly, when mypy complains about a line it's confused on, I can add
a comment to suppress the useless warning:
try:
from pathlib import Path, PurePath
except ImportError:
from pathlib2 import Path, PurePath # type: ignore
And when I want to boost coverage, but I know that some lines aren't
covered under some versions of Python, I can do something like:
self.send_error(HTTPStatus.NOT_FOUND) # pragma: nocover
These are all well and good until I have to *combine* suppressions.
E.g. in the pathlib2 pragma to mypy, I also get a flake8 warning and
I've tried just about every combination of pragma comment I can think
of, but I've not been able to make both tools happy. I've resorted to
refactoring the code into an entire module for flake8 to ignore and added:
# flake8: noqa
to suppress all warnings in the file. I actually have hit on a few
places where I need to suppress warnings from all three tools on the
same line, and I basically can't do it.
The specifics aren't as important as the general use case: multiple
tools competing for the same valuable real-estate.
I have no ideas how to improve the situation, and of course any solution
would involve some coordination between all of these tools, but it's
beginning to feel like a losing battle. Is there a better way?
Cheers,
-Barry
Here are some thoughts––maybe even a proposal––for type-related
terminology, because clear terminology makes discussion and reasoning
easier, and helps avoid errors.
(And related to the PEP 560 thread, the question of what should go into
class attributes like __bases__).
Terminology regarding types is confusing enough, and of all the terminology
I've seen, I like these most:
* concrete type: something that concretely implements data storage or
functionality. Usually this is a normal class.
* abstract type: an assumption or set of assumptions made about an instance.
While both kinds can (and probably should) have a corresponding runtime
representation, there's not that much that we can currently assume about
the second kind––abstract types. They could be almost anything.
Anyway, regarding PEP 560, in my speculated naming, the __bases__
attribute of a class would contain both concrete and abstract bases. Those
with any concrete method implementations should go in the mro.
But then there's the problem that "abstract base classes" may contain both
concrete and abstract methods.
What do we call such a "type"? Maybe we have both "concrete" and "strictly
concrete" types. Perhaps we also have both "abstract" and "strictly
abstract" types. An ABC with some concrete default implementations might
then be both a concrete type and an abstract type.
Note that in the above bullet point "definition" of concrete type, I
intentionally left out the requirement that the type can be instantiated.
The other two bullet points are:
* strictly concrete type: a concrete type that is not abstract––it
concretely implements everything that it represents / describes. This is
almost always a normal class, so it might be also known as "class".
* strictly abstract type: an abstract type that is not concrete––it does
not implement any functionality or storage.
There might be a way to improve terminology from this, but I feel that
what I sketched here is usable but still not very ambiguous.
––Koos
--
+ Koos Zevenhoven + http://twitter.com/k7hoven +
This is an idea I have been playing with and seems to hold some
promise. I think we should use a module instance as the standard
global namespace rather than directly using its dict. I have a
prototype version of CPython that does this, not working 100% yet
though.
Major changes:
- In the frameobject, remove f_builtins and f_globals, add
f_namespace that refers to the module. Make f_globals and
f_builtins into properties.
- Change ceval to use f_namespace, rather than carrying around
globals and builtins. Change functions that take globals as a
dict so they also accept a module object.
- Change the module object to keep track of the builtins for it.
- Change funcobject (e.g. PyFunction_NewWithQualName) to
accept 'globals' as a module object
- When given a dict and we now expect a module object, create an
anonymous module to wrap it. This part is a bit tricky due to
reference cycles and speed requirements. However, my hope is that
if the internals of CPython can be made to pass modules instead of
dicts around, these anonymous modules will become a rare case and
so won't matter if they are a little slower.
So, what is the purpose of all this trouble?
- I believe quite a lot of Python internals can be simpler. For
example, importlib is complicated by the fact that a dict is
passed around when most of the logic would prefer to have the
module. Grubbing in sys.modules to lookup the module object is
ugly. The expression "exec(code, module)" is elegant to me.
- I believe it will be possible to make global variable access work
similar to fast locals. I.e. have each module contain a indexed
list of global names, and use an array lookup rather than a dict
lookup in the normal case. For backwards compatibility, my idea
is to keep a flag on the module object noting if fast global
behavior is okay. If someone grabs the module dict, e.g. using
module.__dict__ , vars() or frame.f_globals then we clear the flag
and revert to the slow dict behavior. Also, if a piece of code is
executed in a different namespace, we have to revert to the slow
case. That does not seem so hard to do.
- I want to have properties for module globals and have them work
from within the module. I.e. changing something from a global
variable to a property should not require changes to other module
code. I.e. LOAD_GLOBAL should trigger a property get.
Regards,
Neil
Previously I posted PEP 560 two weeks ago, while several other PEPs were
also posted, so it didn't get much of attention. Here I post the PEP 560
again, now including the full text for convenience of commenting.
--
Ivan
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
PEP: 560
Title: Core support for generic types
Author: Ivan Levkivskyi <levkivskyi(a)gmail.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 03-Sep-2017
Python-Version: 3.7
Post-History: 09-Sep-2017
Abstract
========
Initially PEP 484 was designed in such way that it would not introduce
*any* changes to the core CPython interpreter. Now type hints and
the ``typing`` module are extensively used by the community, e.g. PEP 526
and PEP 557 extend the usage of type hints, and the backport of ``typing``
on PyPI has 1M downloads/month. Therefore, this restriction can be removed.
It is proposed to add two special methods ``__class_getitem__`` and
``__subclass_base__`` to the core CPython for better support of
generic types.
Rationale
=========
The restriction to not modify the core CPython interpreter lead to some
design decisions that became questionable when the ``typing`` module started
to be widely used. There are three main points of concerns:
performance of the ``typing`` module, metaclass conflicts, and the large
number of hacks currently used in ``typing``.
Performance:
------------
The ``typing`` module is one of the heaviest and slowest modules in
the standard library even with all the optimizations made. Mainly this is
because subscripted generic types (see PEP 484 for definition of terms
used in this PEP) are class objects (see also [1]_). The three main ways how
the performance can be improved with the help of the proposed special
methods:
- Creation of generic classes is slow since the ``GenericMeta.__new__`` is
very slow; we will not need it anymore.
- Very long MROs for generic classes will be twice shorter; they are present
because we duplicate the ``collections.abc`` inheritance chain
in ``typing``.
- Time of instantiation of generic classes will be improved
(this is minor however).
Metaclass conflicts:
--------------------
All generic types are instances of ``GenericMeta``, so if a user uses
a custom metaclass, then it is hard to make a corresponding class generic.
This is particularly hard for library classes that a user doesn't control.
A workaround is to always mix-in ``GenericMeta``::
class AdHocMeta(GenericMeta, LibraryMeta):
pass
class UserClass(LibraryBase, Generic[T], metaclass=AdHocMeta):
...
but this is not always practical or even possible. With the help of the
proposed special attributes the ``GenericMeta`` metaclass will not be
needed.
Hacks and bugs that will be removed by this proposal:
-----------------------------------------------------
- ``_generic_new`` hack that exists since ``__init__`` is not called on
instances with a type differing form the type whose ``__new__`` was
called,
``C[int]().__class__ is C``.
- ``_next_in_mro`` speed hack will be not necessary since subscription will
not create new classes.
- Ugly ``sys._getframe`` hack, this one is particularly nasty, since it
looks
like we can't remove it without changes outside ``typing``.
- Currently generics do dangerous things with private ABC caches
to fix large memory consumption that grows at least as O(N\ :sup:`2`),
see [2]_. This point is also important because it was recently proposed to
re-implement ``ABCMeta`` in C.
- Problems with sharing attributes between subscripted generics,
see [3]_. Current solution already uses ``__getattr__`` and
``__setattr__``,
but it is still incomplete, and solving this without the current proposal
will be hard and will need ``__getattribute__``.
- ``_no_slots_copy`` hack, where we clean-up the class dictionary on every
subscription thus allowing generics with ``__slots__``.
- General complexity of the ``typing`` module, the new proposal will not
only allow to remove the above mentioned hacks/bugs, but also simplify
the implementation, so that it will be easier to maintain.
Specification
=============
The idea of ``__class_getitem__`` is simple: it is an exact analog of
``__getitem__`` with an exception that it is called on a class that
defines it, not on its instances, this allows us to avoid
``GenericMeta.__getitem__`` for things like ``Iterable[int]``.
The ``__class_getitem__`` is automatically a class method and
does not require ``@classmethod`` decorator (similar to
``__init_subclass__``) and is inherited like normal attributes.
For example::
class MyList:
def __getitem__(self, index):
return index + 1
def __class_getitem__(cls, item):
return f"{cls.__name__}[{item.__name__}]"
class MyOtherList(MyList):
pass
assert MyList()[0] == 1
assert MyList[int] == "MyList[int]"
assert MyOtherList()[0] == 1
assert MyOtherList[int] == "MyOtherList[int]"
Note that this method is used as a fallback, so if a metaclass defines
``__getitem__``, then that will have the priority.
If an object that is not a class object appears in the bases of a class
definition, the ``__subclass_base__`` is searched on it. If found,
it is called with the original tuple of bases as an argument. If the result
of the call is not ``None``, then it is substituted instead of this object.
Otherwise (if the result is ``None``), the base is just removed. This is
necessary to avoid inconsistent MRO errors, that are currently prevented by
manipulations in ``GenericMeta.__new__``. After creating the class,
the original bases are saved in ``__orig_bases__`` (currently this is also
done by the metaclass).
NOTE: These two method names are reserved for exclusive use by
the ``typing`` module and the generic types machinery, and any other use is
strongly discouraged. The reference implementation (with tests) can be found
in [4]_, the proposal was originally posted and discussed on
the ``typing`` tracker, see [5]_.
Backwards compatibility and impact on users who don't use ``typing``:
=====================================================================
This proposal may break code that currently uses the names
``__class_getitem__`` and ``__subclass_base__``.
This proposal will support almost complete backwards compatibility with
the current public generic types API; moreover the ``typing`` module is
still
provisional. The only two exceptions are that currently
``issubclass(List[int], List)`` returns True, with this proposal it will
raise
``TypeError``. Also ``issubclass(collections.abc.Iterable,
typing.Iterable)``
will return ``False``, which is probably desirable, since currently we have
a (virtual) inheritance cycle between these two classes.
With the reference implementation I measured negligible performance effects
(under 1% on a micro-benchmark) for regular (non-generic) classes.
References
==========
.. [1] Discussion following Mark Shannon's presentation at Language Summit
(https://github.com/python/typing/issues/432)
.. [2] Pull Request to implement shared generic ABC caches
(https://github.com/python/typing/pull/383)
.. [3] An old bug with setting/accessing attributes on generic types
(https://github.com/python/typing/issues/392)
.. [4] The reference implementation
(https://github.com/ilevkivskyi/cpython/pull/2/files)
.. [5] Original proposal
(https://github.com/python/typing/issues/468)
Copyright
=========
This document has been placed in the public domain.
General summary so far
#######################
This debate has been very civil so far and quite productive, so thanks
to everyone involved.
Several issues and solution proposals have been mentioned, so I will
make a summary here.
Issue 1: running Python is a different command different OS when several
versions are installed
==================================================================================================
Proposal A:
-----------
Suffix Python executable on Windows like on Unix, so that people will
type pythonX.X if they want a specify version.
Pros: easy and discoverable.
Cons: you need a lot of stuff in the system path.
Proposal B:
------------
Provide the "py" command on Unix, so that people will type "py -x.x"
Pros: no more issues with system path. Just add this command, and
register python installations to be listed by "py".
Cons: very few people know about "py", even on windows. This would need
lots of communication. Harder to change Mac and majors Linux distros
setup than just the next windows installer.
Issue 2: there is no one and only one obvious way to do venv / pip
===================================================================
Because pip and venv are tied to a Python version, there are numerous
ways to call them.
Just for pip, you have "pip", "py -x.x -m pip", "pythonx.x -m pip" and
"pipx".
Proposal A:
-----------
Decide on one form, and promote it everywhere.
Pros: simple.
Cons: long and tedious process, with a huge inertia before all 3rd party
to it. Would need a "PEP 8 for Python command". Requires "issue 1" to be
solved before.
Proposal B:
------------
Provide a pipenv-like behavior
pipenv is a tool created by the author of "requests" that automatically
"do the right thing" by default. The first time you install a package,
it creates a virtualenv for the current directory (but don't store it in
the current directory). It will then install all packages in there.
"pipenv shell" starts a shell in the virtualenv. Dependencies are
automatically dumped in a requirement files separating dev and prod.
Pros: bypass the multiple installers issue by prompting you what version
you wish the first time it creates a virtualenv. Solve the problem of
admin rights and system path.
Cons: lots of work. Either a new tool or breaking compat with pip. If
debian didn't install pip by default, this may not help.
Issue 3: pip and virtualenv are not available everywhere
===================================================================
Proposal A:
------------
Make a mock "pip" commands for when it's not installed. venv already
does that in some cases (see attachements), alhought it does give you
the wrong command so it should be fixed.
The gist of it: if the users try to run pip install while it's missing,
it prints a warning explaining it's not installed and the command you
can run to solve the problem plus a link to more information in the
documentation.
Pros: low friction to implement.
Cons: requires to put up a hook somewhere so that each linux distro can
input "the proper commands to use" for their system when they package
python. More a workaround than a solution.
Proposal B:
------------
Make pip and venv part of the standard and request debian that they
provide it.
Pros: straight forward.
Cons: holy war in the making.
.................
Can I have +1 and -1 from each of you on each point ?
Should I start a new thread for each of them ?
Favorite personnal combination
================================
1 - So far, I like "providing py on unix" better, but I think it's too
hard to do. So provide suffix on windows seems more achievable.
So +1 for having pythonX.X on windows.
2 - Decide on one form, and promote it everywhere is again the simplest
IMO. I love pipenv but the strings to pull to get something like this
means it probably would not happen.
If we stick to suffix for issue 1, this mean we can promote:
pythonX.X -m pip
pythonX.X -m venv
Everywhere. In the docs. In the tutorials. In teaching classes.
This will help a lot to remove all confusions and make the Python start
up experience way easier.
3 - getting angry debian devs is probably not wise. The "mock" pip/venv
command also plays very well with the 2 other solutions.
It means we can promote everywhere:
pythonX.X -m cmd
It will work most of the time. And when it doesn't, give clear
contextual steps on
I have written a short PEP as a complement/alternative to PEP 549.
I will be grateful for comments and suggestions. The PEP should
appear online soon.
--
Ivan
***********************************************************
PEP: 562
Title: Module __getattr__
Author: Ivan Levkivskyi <levkivskyi(a)gmail.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 09-Sep-2017
Python-Version: 3.7
Post-History: 09-Sep-2017
Abstract
========
It is proposed to support ``__getattr__`` function defined on modules to
provide basic customization of module attribute access.
Rationale
=========
It is sometimes convenient to customize or otherwise have control over
access to module attributes. A typical example is managing deprecation
warnings. Typical workarounds are assigning ``__class__`` of a module object
to a custom subclass of ``types.ModuleType`` or substituting ``sys.modules``
item with a custom wrapper instance. It would be convenient to simplify this
procedure by recognizing ``__getattr__`` defined directly in a module that
would act like a normal ``__getattr__`` method, except that it will be
defined
on module *instances*. For example::
# lib.py
from warnings import warn
deprecated_names = ["old_function", ...]
def _deprecated_old_function(arg, other):
...
def __getattr__(name):
if name in deprecated_names:
warn(f"{name} is deprecated", DeprecationWarning)
return globals()[f"_deprecated_{name}"]
raise AttributeError(f"module {__name__} has no attribute {name}")
# main.py
from lib import old_function # Works, but emits the warning
There is a related proposal PEP 549 that proposes to support instance
properties for a similar functionality. The difference is this PEP proposes
a faster and simpler mechanism, but provides more basic customization.
An additional motivation for this proposal is that PEP 484 already defines
the use of module ``__getattr__`` for this purpose in Python stub files,
see [1]_.
Specification
=============
The ``__getattr__`` function at the module level should accept one argument
which is a name of an attribute and return the computed value or raise
an ``AttributeError``::
def __getattr__(name: str) -> Any: ...
This function will be called only if ``name`` is not found in the module
through the normal attribute lookup.
The reference implementation for this PEP can be found in [2]_.
Backwards compatibility and impact on performance
=================================================
This PEP may break code that uses module level (global) name
``__getattr__``.
The performance implications of this PEP are minimal, since ``__getattr__``
is called only for missing attributes.
References
==========
.. [1] PEP 484 section about ``__getattr__`` in stub files
(https://www.python.org/dev/peps/pep-0484/#stub-files)
.. [2] The reference implementation
(https://github.com/ilevkivskyi/cpython/pull/3/files)
Copyright
=========
This document has been placed in the public domain.
I needed to declare a type that would mean "Any but None" and didn't find how.
Current implementation allows to expand set of allowed types, not narrow it.
Perhaps typing needs the rest of set operators in addition to Union?
Best Regards,
Ilya Kulakov
Hi,
The dangers of eval and exec are obvious and well known to advanced users,
but the availability as built-in functions makes it too tempting for
beginners or even medium-level programmers. You can see questions about
these function pretty often in stackoverflow (roughly once a day
<https://stackoverflow.com/search?tab=newest&q=eval%20python>, though
sometimes the uses are legitimate).
Maybe we could start a ten-year process of deprecating the use of
`builtins.eval` (in the docs, and then with warnings)? `builtins.eval` will
be a wrapper to the real evaluation function, moved to `unsafe.eval` or
something obvious like that, so all you need to do to port your code is to
add `from unsafe import unsafe_eval as eval, unsafe_exec as exec` at the
top of the file; it will be a nice warning to the reader.
The fact that it is a wrapper will slightly slow it down and make the stack
traces noisier - both are good things, IMO.
Also, it is unfortunate that `ast.literal_eval` is less accessible than
`builtins.eval`. Giving it an alias in builtins might make it easier for
programmers (and less scary - "ast" might sound like I need a PhD to use
it).
What do you think?
Elazar
The discussions on moving typing to a standalone library prompted me to
take a look at the documentation for typing which in turn lead to an
idea that should be reasonably simple to implement but greatly add value
to typing for some applications.
If a group of iterators were to be added to the typing module it would
be reasonably simple to automatically add and assert to any decorated
modules to ensure that such modules were always called with the
documented types.
I am thinking of decorators such as:
- @typing.assert_params_mismatch - this would provide a wrapper that
had auto-generated asserts that all the parameters were of designated types.
- @typing.debug_assert_params_mismatch - this would provide a wrapper
that had auto-generated asserts that all the parameters were of
designated types only if a DEBUG environmental variable was set or similar.
- @typing.warn_params_mismatch - this would provide a wrapper that had
auto-generated warnings issued to stderr if any of the parameters were
of not of the designated types.
- @typing.coerce_params - this would add a wrapper to attempt to
convert any mismatched parameters to the specified type(s).
I also think that this might increase the uptake of typing by giving
some clear benefits outside of documentation and static type checking.
Pushing this out for comment before considering a more formal, (PEP),
proposal.
--
Steve (Gadget) Barnes
Any opinions in this message are my personal opinions and do not reflect
those of my employer.
---
This email has been checked for viruses by AVG.
http://www.avg.com