webmaster has already heard from 4 people who cannot install it.
I sent them to the bug tracker or to python-list but they seem
not to have gone either place. Is there some guide I should be
sending them to, 'how to debug installation problems'?
Laura
Hi folks,
I have a confession to make - I dropped the ball on the HTTPS
verification backport proposals in PEP 493, and let the upstream and
downstream approval processes get out of sequence.
As a result, the RHEL 7.2 beta released back in September incorporates
the HTTPS verification feature backport based on the current PEP 493
draft, even though that hasn't formally been pronounced as an Active
recommendation by python-dev yet.
Accordingly, I'm belatedly submitting it for pronouncement now:
https://www.python.org/dev/peps/pep-0493/
There's currently no BDFL-Delegate assigned, so if Guido doesn't want
to handle it, we'll need to address that question first.
Our last discussion back in July seemed to show that folks either
didn't care about the question (because they're using unmodified
upstream versions so the PEP didn't affect them), or else thought the
approach described in the PEP was reasonable, so I'm hoping the
consequences of my mistake won't be too severe.
Regards,
Nick.
P.S. I'm aware that this looks like presenting a fait accompli at a
point where it's too late to realistically say "No", but the truth is
that preparation for the Python in Education miniconf at PyCon
Australia ramped up immediately after the July discussion, and then I
personally got confused as to the scope of what was being included in
7.2 (I mistakenly thought it was just PEP 466 for now, with 476+493
being deferred to a later release, but it's actually the whole package
of 466+476+493). That's my fault for trying to keep track of too many
things at once (and thus failing at some of them), not anyone else's.
================================
PEP: 493
Title: HTTPS verification recommendations for Python 2.7 redistributors
Version: $Revision$
Last-Modified: $Date$
Author: Nick Coghlan <ncoghlan(a)gmail.com>,
Robert Kuska <rkuska(a)redhat.com>,
Marc-André Lemburg <mal(a)lemburg.com>
Status: Draft
Type: Informational
Content-Type: text/x-rst
Created: 10-May-2015
Post-History: 06-Jul-2015
Abstract
========
PEP 476 updated Python's default handling of HTTPS certificates to be
appropriate for communication over the public internet. The Python 2.7 long
term maintenance series was judged to be in scope for this change, with the
new behaviour introduced in the Python 2.7.9 maintenance release.
This PEP provides recommendations to downstream redistributors wishing to
provide a smoother migration experience when helping their users to manage
this change in Python's default behaviour.
Rationale
=========
PEP 476 changed Python's default behaviour to better match the needs and
expectations of developers operating over the public internet, a category
which appears to include most new Python developers. It is the position of
the authors of this PEP that this was a correct decision.
However, it is also the case that this change *does* cause problems for
infrastructure administrators operating private intranets that rely on
self-signed certificates, or otherwise encounter problems with the new default
certificate verification settings.
The long term answer for such environments is to update their internal
certificate management to at least match the standards set by the public
internet, but in the meantime, it is desirable to offer these administrators
a way to continue receiving maintenance updates to the Python 2.7 series,
without having to gate that on upgrades to their certificate management
infrastructure.
PEP 476 did attempt to address this question, by covering how to revert the
new settings process wide by monkeypatching the ``ssl`` module to restore the
old behaviour. Unfortunately, the ``sitecustomize.py`` based technique proposed
to allow system administrators to disable the feature by default in their
Standard Operating Environment definition has been determined to be
insufficient in at least some cases. The specific case of interest to the
authors of this PEP is the one where a Linux distributor aims to provide
their users with a
`smoother migration path
<https://bugzilla.redhat.com/show_bug.cgi?id=1173041>`__
than the standard one provided by consuming upstream CPython 2.7 releases
directly, but other potential challenges have also been pointed out with
updating embedded Python runtimes and other user level installations of Python.
Rather than allowing a plethora of mutually incompatibile migration techniques
to bloom, this PEP proposes two alternative approaches that redistributors
may take when addressing these problems. Redistributors may choose to implement
one, both, or neither of these approaches based on their assessment of the
needs of their particular userbase.
These designs are being proposed as a recommendation for redistributors, rather
than as new upstream features, as they are needed purely to support legacy
environments migrating from older versions of Python 2.7. Neither approach
is being proposed as an upstream Python 2.7 feature, nor as a feature in any
version of Python 3 (whether published directly by the Python Software
Foundation or by a redistributor).
Requirements for capability detection
=====================================
As these recommendations are intended to cover backports to earlier Python
versions, the Python version number cannot be used as a reliable means for
detecting them. Instead, the recommendations are defined to allow the presence
or absence of the feature to be determined using the following technique::
python -c "import ssl; ssl._relevant_attribute"
This will fail with `AttributeError` (and hence a non-zero return code) if the
relevant capability is not available.
The marker attributes are prefixed with an underscore to indicate the
implementation dependent nature of these capabilities - not all Python
distributions will offer them, only those that are providing a multi-stage
migration process from the legacy HTTPS handling to the new default behaviour.
Recommendation for an environment variable based security downgrade
===================================================================
Some redistributors may wish to provide a per-application option to disable
certificate verification in selected applications that run on or embed CPython
without needing to modify the application itself.
In these cases, a configuration mechanism is needed that provides:
* an opt-out model that allows certificate verification to be selectively
turned off for particular applications after upgrading to a version of
Python that verifies certificates by default
* the ability for all users to configure this setting on a per-application
basis, rather than on a per-system, or per-Python-installation basis
This approach may be used for any redistributor provided version of Python 2.7,
including those that advertise themselves as providing Python 2.7.9 or later.
Required marker attribute
-------------------------
The required marker attribute on the ``ssl`` module when implementing this
recommendation is::
_https_verify_envvar = 'PYTHONHTTPSVERIFY'
This not only makes it straightforward to detect the presence (or absence) of
the capability, it also makes it possible to programmatically determine the
relevant environment variable name.
Recommended modifications to the Python standard library
--------------------------------------------------------
The recommended approach to providing a per-application configuration setting
for HTTPS certificate verification that doesn't require modifications to the
application itself is to:
* modify the ``ssl`` module to read the ``PYTHONHTTPSVERIFY`` environment
variable when the module is first imported into a Python process
* set the ``ssl._create_default_https_context`` function to be an alias for
``ssl._create_unverified_context`` if this environment variable is present
and set to ``'0'``
* otherwise, set the ``ssl._create_default_https_context`` function to be an
alias for ``ssl.create_default_context`` as usual
Example implementation
----------------------
::
_https_verify_envvar = 'PYTHONHTTPSVERIFY'
def _get_https_context_factory():
config_setting = os.environ.get(_https_verify_envvar)
if config_setting == '0':
return _create_unverified_context
return create_default_context
_create_default_https_context = _get_https_context_factory()
Security Considerations
-----------------------
Relative to an unmodified version of CPython 2.7.9 or later, this approach
does introduce a new downgrade attack against the default security settings
that potentially allows a sufficiently determined attacker to revert Python
to the vulnerable configuration used in CPython 2.7.8 and earlier releases.
However, such an attack requires the ability to modify the execution
environment of a Python process prior to the import of the ``ssl`` module,
and any attacker with such access would already be able to modify the
behaviour of the underlying OpenSSL implementation.
Recommendation for backporting to earlier Python versions
=========================================================
Some redistributors, most notably Linux distributions, may choose to backport
the PEP 476 HTTPS verification changes to modified Python versions based on
earlier Python 2 maintenance releases. In these cases, a configuration
mechanism is needed that provides:
* an opt-in model that allows the decision to enable HTTPS certificate
verification to be made independently of the decision to upgrade to the
Python version where the feature was first backported
* the ability for system administrators to set the default behaviour of Python
applications and scripts run directly in the system Python installation
* the ability for the redistributor to consider changing the default behaviour
of *new* installations at some point in the future without impacting existing
installations that have been explicitly configured to skip verifying HTTPS
certificates by default
This approach should not be used for any Python installation that advertises
itself as providing Python 2.7.9 or later, as most Python users will have the
reasonable expectation that all such environments will validate HTTPS
certificates by default.
Required marker attribute
-------------------------
The required marker attribute on the ``ssl`` module when implementing this
recommendation is::
_cert_verification_config = '<path to configuration file>'
This not only makes it straightforward to detect the presence (or absence) of
the capability, it also makes it possible to programmatically determine the
relevant configuration file name.
Recommended modifications to the Python standard library
--------------------------------------------------------
The recommended approach to backporting the PEP 476 modifications to an earlier
point release is to implement the following changes relative to the default
PEP 476 behaviour implemented in Python 2.7.9+:
* modify the ``ssl`` module to read a system wide configuration file when the
module is first imported into a Python process
* define a platform default behaviour (either verifying or not verifying HTTPS
certificates) to be used if this configuration file is not present
* support selection between the following three modes of operation:
* ensure HTTPS certificate verification is enabled
* ensure HTTPS certificate verification is disabled
* delegate the decision to the redistributor providing this Python version
* set the ``ssl._create_default_https_context`` function to be an alias for
either ``ssl.create_default_context`` or ``ssl._create_unverified_context``
based on the given configuration setting.
Recommended file location
-------------------------
This approach is currently only defined for \*nix system Python installations.
The recommended configuration file name is
``/etc/python/cert-verification.cfg``.
The ``.cfg`` filename extension is recommended for consistency with the
``pyvenv.cfg`` used by the ``venv`` module in Python 3's standard library.
Recommended file format
-----------------------
The configuration file should use a ConfigParser ini-style format with a
single section named ``[https]`` containing one required setting ``verify``.
Permitted values for ``verify`` are:
* ``enable``: ensure HTTPS certificate verification is enabled by default
* ``disable``: ensure HTTPS certificate verification is disabled by default
* ``platform_default``: delegate the decision to the redistributor providing
this particular Python version
If the ``[https]`` section or the ``verify`` setting are missing, or if the
``verify`` setting is set to an unknown value, it should be treated as if the
configuration file is not present.
Example implementation
----------------------
::
_cert_verification_config = '/etc/python/cert-verification.cfg'
def _get_https_context_factory():
# Check for a system-wide override of the default behaviour
context_factories = {
'enable': create_default_context,
'disable': _create_unverified_context,
'platform_default': _create_unverified_context, # For now :)
}
import ConfigParser
config = ConfigParser.RawConfigParser()
config.read(_cert_verification_config)
try:
verify_mode = config.get('https', 'verify')
except (ConfigParser.NoSectionError, ConfigParser.NoOptionError):
verify_mode = 'platform_default'
default_factory = context_factories.get('platform_default')
return context_factories.get(verify_mode, default_factory)
_create_default_https_context = _get_https_context_factory()
Security Considerations
-----------------------
The specific recommendations for the backporting case are designed to work for
privileged, security sensitive processes, even those being run in the following
locked down configuration:
* run from a locked down administrator controlled directory rather than a normal
user directory (preventing ``sys.path[0]`` based privilege escalation attacks)
* run using the ``-E`` switch (preventing ``PYTHON*`` environment variable based
privilege escalation attacks)
* run using the ``-s`` switch (preventing user site directory based privilege
escalation attacks)
* run using the ``-S`` switch (preventing ``sitecustomize`` based privilege
escalation attacks)
The intent is that the *only* reason HTTPS verification should be getting
turned off system wide when using this approach is because:
* an end user is running a redistributor provided version of CPython rather
than running upstream CPython directly
* that redistributor has decided to provide a smoother migration path to
verifying HTTPS certificates by default than that being provided by the
upstream project
* either the redistributor or the local infrastructure administrator has
determined that it is appropriate to override the default upstream behaviour
(at least for the time being)
Using an administrator controlled configuration file rather than an environment
variable has the essential feature of providing a smoother migration path, even
for applications being run with the ``-E`` switch.
Combining the recommendations
=============================
If a redistributor chooses to implement both recommendations, then the
environment variable should take precedence over the system-wide configuration
setting. This allows the setting to be changed for a given user, virtual
environment or application, regardless of the system-wide default behaviour.
In this case, if the ``PYTHONHTTPSVERIFY`` environment variable is defined, and
set to anything *other* than ``'0'``, then HTTPS certificate verification
should be enabled.
Example implementation
----------------------
::
_https_verify_envvar = 'PYTHONHTTPSVERIFY'
_cert_verification_config = '/etc/python/cert-verification.cfg'
def _get_https_context_factory():
# Check for am environmental override of the default behaviour
config_setting = os.environ.get(_https_verify_envvar)
if config_setting is not None:
if config_setting == '0':
return _create_unverified_context
return create_default_context
# Check for a system-wide override of the default behaviour
context_factories = {
'enable': create_default_context,
'disable': _create_unverified_context,
'platform_default': _create_unverified_context, # For now :)
}
import ConfigParser
config = ConfigParser.RawConfigParser()
config.read(_cert_verification_config)
try:
verify_mode = config.get('https', 'verify')
except (ConfigParser.NoSectionError, ConfigParser.NoOptionError):
verify_mode = 'platform_default'
default_factory = context_factories.get('platform_default')
return context_factories.get(verify_mode, default_factory)
_create_default_https_context = _get_https_context_factory()
Copyright
=========
This document has been placed into the public domain.
--
Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia
Hi python-dev,
I've seen that on and off CPython had attempts to measure benchmarks over
time to avoid performance regressions (i.e.: https://speed.python.org), but
had nothing concrete so far, so, I ended up creating a hosted service for
that (https://www.speedtin.com) and I'd like to help in setting up a
structure to run the benchmarks from https://hg.python.org/benchmarks/ and
properly upload them to SpeedTin (if CPython devs are Ok with that) -- note
that I don't really have server to run the benchmarks, only to host the
data (but https://speed.python.org seems to indicate that such a server is
available...).
There's a sample report at:
https://www.speedtin.com/reports/1_CPython27x_Performance_Over_Time/ (it
has real data from running using the PyPy benchmarks as I only discovered
about the benchmarks from https://hg.python.org/benchmarks/ later on --
also, it doesn't seem to support Python 3 right now, so, it's probably not
that useful for the current Python dev, but it does have some nice insight
on CPython 2.7.x performance over time).
Later on, the idea is being able to compare across different Python
implementations which use the same benchmark set... (although that needs
other implementations to also post to the data to SpeedTin).
Note that uploading the data to SpeedTin should be pretty straightforward
(by using https://github.com/fabioz/pyspeedtin, so, the main issue would be
setting up o machine to run the benchmarks).
Best Regards,
Fabio
Slots like PyTypeObject.tp_setattr, PySequenceMethods.sq_ass_item,
PyMappingMethods.mp_ass_subscript are used not only for setting
attribute/item value, but also for deleting attribute/item (if value is
NULL). This is not documented and should be. [1] Correspondingly public
API functions like PyObject_SetAttr, PyObject_SetItem,
PySequence_SetItem, PySequence_SetSlice, PyMapping_SetItemString can be
used for deleting. But all these functions have special counterparts for
deleting: PyObject_DelAttr etc.
The question is wherever deleting ability of Set-functions is
intentional, should we document this or deprecate and then delete?
[1] http://bugs.python.org/issue25701
Hello Python Developers!
I'm writing a book about descriptors, and I'm hoping to make it a
comprehensive guide, covering just about everything there is to know about
descriptors, including many tips to get past certain knowledge barriers and
how to avoid certain pitfalls. I'm hoping for it to become the definitive
guide for those who want to really understand descriptors. Raymond's
Descriptor HowTo Guide is great, but focused. I want to expand on what that
guide teaches.
Anyway, I am currently reading through my initial rough draft and making
corrections for a revised draft. Once I'm done with that, I'd like a few
volunteers to read through it. First, to find spelling, grammatical, or
consistency mistakes, though those should be few in number, but also to
give input as to whether something seems too repetitive, if some more
information should be given in a spot, and if I'm even blatantly wrong
about anything (hopefully not).
Anyone who helps will be able to receive a free ebook of the final copy as
well as mention in the Special Thanks.
I appreciate any help anyone can give to assist in making this book as
great as it can be.
Thanks,
Jacob Zimmerman
Twitter: @jacobz_20
Blog: http://programmingideaswithjake.wordpress.com
Hi,
I am new here wishing to contribute to python.
I tried to go through the tutorial for seting up Cpython, while I cannot pass the configuration. My system is Mac OS and I have installed the command tool of Xcode and hg. But on terminal it said that no such file or directory when I type “./configure --with-pydebug”. What was wrong with my action? How can I fix the issue?
This may a little question, but thanks for your help and tutoring!
Sincerely,
Chongzhao Mao
Hi
I have come across some dubious code in Objects/weakrefobject.c which looks like a bug to me, but wanted to run it past others.
This was discovered from looking at crash dumps from a multithreaded python app (using Python 2.7.9, but the same weakref code exists in 3.5 and hg tip).
The code that worries me is at the end of the "weakref_richcompare" function:
return PyObject_RichCompare(PyWeakref_GET_OBJECT(self),
PyWeakref_GET_OBJECT(other), op);
At this point the code has established that the referents are still alive, and it is trying to compare the referents. However it has not acquired a strong reference to the referents, so I think it is possible for one of them to be deleted half way through this comparison. This can lead to a crash, because PyObject_RichCompare assumes that the PyObject*'s it was passed will remain usable for the duration of the call.
The crashes I have seen involve data corruption consistent with one of these PyObject's being deleted and the memory reused for something else, eg:
00 python27!try_3way_compare+0x15 [objects\object.c @ 712]
01 python27!try_3way_to_rich_compare+0xb [objects\object.c @ 901]
02 python27!do_richcmp+0x2c [objects\object.c @ 935]
03 python27!PyObject_RichCompare+0x99 [objects\object.c @ 982]
04 python27!weakref_richcompare+0x6a [objects\weakrefobject.c @ 212]
(In this example v->ob_type was 0x5f637865 which is ASCII "exc_", not a valid pointer at all)
Other places in weakrefobject.c seem to have a similar weakness, eg in weakref_hash and weakref_repr.
I have not been able to produce a small test case to exhibit this crash, but from this inspection of the code it looks like a bug - am I understanding this correctly?
Thanks,
Luke.
**********************************************************************************************
Important Note
This email (including any attachments) contains information which is confidential and may be subject to legal privilege. If you are not the intended recipient you must not use, distribute or copy this email. If you have received this email in error please notify the
sender immediately and delete this email. Any views expressed in this email are not necessarily the views of IRESS Limited.
It is the duty of the recipient to virus scan and otherwise test the information provided before loading onto any computer system.
IRESS Limited does not warrant that the information is free of a virus or any other defect or error.
**********************************************************************************************
My mailer just barfed trying to read that summary.
The problem is that the mail comes out with the encoding:
Content-Type: text/plain; charset="us-ascii"
but then wants to print 'http://bugs.python.org/issue25709 opened
by Árpád Kósa'
Can we change the encoding to utf-8 ?
Laura
