From python-checkins at python.org  Sun Feb  1 04:06:09 2015
From: python-checkins at python.org (nick.coghlan)
Date: Sun, 01 Feb 2015 03:06:09 +0000
Subject: [Python-checkins] =?utf-8?q?peps=3A_PEP_462_now_proposes_Kallithe?=
	=?utf-8?q?a_as_the_review_component?=
Message-ID: <20150201030551.39290.76025@psf.io>

https://hg.python.org/peps/rev/33f7f6d4fcb9
changeset:   5683:33f7f6d4fcb9
user:        Nick Coghlan <ncoghlan at gmail.com>
date:        Sun Feb 01 12:59:34 2015 +1000
summary:
  PEP 462 now proposes Kallithea as the review component

files:
  pep-0462.txt |  188 ++++++++++++++++++++------------------
  pep-0474.txt |   57 +++++++++++-
  2 files changed, 157 insertions(+), 88 deletions(-)


diff --git a/pep-0462.txt b/pep-0462.txt
--- a/pep-0462.txt
+++ b/pep-0462.txt
@@ -8,7 +8,7 @@
 Content-Type: text/x-rst
 Requires: 474
 Created: 23-Jan-2014
-Post-History: 25-Jan-2014, 27-Jan-2014
+Post-History: 25-Jan-2014, 27-Jan-2014, 01-Feb-2015
 
 
 Abstract
@@ -26,8 +26,8 @@
 PEP Deferral
 ============
 
-This PEP is currently deferred pending updates to redesign it around
-the proposal in PEP 474 to create a Kallithea-based forge.python.org service.
+This PEP is currently deferred pending acceptance or rejection of the
+Kallithea-based forge.python.org proposal in PEP 474.
 
 
 Rationale for changes to the core development workflow
@@ -139,9 +139,11 @@
 * For core developers, the branch wrangling for bug fixes is delicate and
   easy to get wrong. Conflicts on the NEWS file and push races when
   attempting to upload changes add to the irritation of something most of
-  us aren't being paid to spend time on. The time we spend actually getting
-  a change merged is time we're not spending coding additional changes,
-  writing or updating documentation or reviewing contributions from others.
+  us aren't being paid to spend time on (and for those that are, contributing
+  to CPython is likely to be only one of our responsibilities). The time we
+  spend actually getting a change merged is time we're not spending coding
+  additional changes, writing or updating documentation or reviewing
+  contributions from others.
 * Red buildbots make life difficult for other developers (since a local
   test failure may *not* be due to anything that developer did), release
   managers (since they may need to enlist assistance cleaning up test
@@ -172,13 +174,17 @@
 * Rietveld (also hosted on bugs.python.org) for code review
 * Buildbot (buildbot.python.org) for automated testing
 
-This proposal does *not* currently suggest replacing any of these tools,
-although implementing it effectively may require modifications to some or
-all of them.
+This proposal suggests replacing the use of Rietveld for code review with
+the more full-featured Kallithea-based forge.python.org service proposed in
+PEP 474. Guido has indicated that the original Rietveld implementation was
+primarily intended as a public demonstration application for Google App
+Engine, and switching to Kallithea will address some of the issues with
+identifying intended target branches that arise when working with patch files
+on Roundup and the associated reviews in the integrated Rietveld instance.
 
-It does however suggest the addition of new tools in order to automate
-additional parts of the workflow, as well as a critical review of these
-tools to see which, if any, may be candidates for replacement.
+It also suggests the addition of new tools in order to automate
+additional parts of the workflow, as well as a critical review of the
+remaining tools to see which, if any, may be candidates for replacement.
 
 
 Proposal
@@ -224,12 +230,12 @@
 the queue, and rebuilds the queue without the failing patch.
 
 If a developer looks at a test which failed on merge and determines that it
-was due to an intermittent failure, then they can resubmit the patch for
+was due to an intermittent failure, they can then resubmit the patch for
 another attempt at merging.
 
 To adapt this process to CPython, it should be feasible to have Zuul monitor
-Rietveld for approved patches (which would require a feature addition in
-Rietveld), submit them to Buildbot for testing on the stable buildbots, and
+Kallithea for approved pull requests (which may require a feature addition in
+Kallithea), submit them to Buildbot for testing on the stable buildbots, and
 then merge the changes appropriately in Mercurial. This idea poses a few
 technical challenges, which have their own section below.
 
@@ -280,7 +286,7 @@
 still allowing the rest of the automation flows to be worked out (such as
 how to push a patch into the merge queue).
 
-The one downside to this approach is that Zuul wouldn't have complete
+The key downside to this approach is that Zuul wouldn't have complete
 control of the merge process as it usually expects, so there would
 potentially be additional coordination needed around that.
 
@@ -288,14 +294,15 @@
 deployment proves to have more trouble with test reliability than is
 anticipated.
 
-It would also be possible to tweak the merge gating criteria such that it doesn't
-run the test suite if it detects that the patch hasn't modified any files
-outside the "Docs" tree, and instead only checks that the documentation
+It would also be possible to tweak the merge gating criteria such that it
+doesn't run the test suite if it detects that the patch hasn't modified any
+files outside the "Docs" tree, and instead only checks that the documentation
 builds without errors.
 
 As yet another alternative, it may be reasonable to move some parts of the
-documentation (such as the tutorial) out of the main source repository and
-manage them using the simpler pull request based model.
+documentation (such as the tutorial and the HOWTO guides) out of the main
+source repository and manage them using the simpler pull request based model
+described in PEP 474.
 
 
 Perceived Benefits
@@ -346,10 +353,11 @@
 Finally, a more stable default branch in CPython makes it easier for
 other Python projects to conduct continuous integration directly against the
 main repo, rather than having to wait until we get into the release
-candidate phase. At the moment, setting up such a system isn't particularly
-attractive, as it would need to include an additional mechanism to wait
-until CPython's own Buildbot fleet had indicate that the build was in a
-usable state.
+candidate phase of a new release. At the moment, setting up such a system
+isn't particularly attractive, as it would need to include an additional
+mechanism to wait until CPython's own Buildbot fleet indicated that the
+build was in a usable state. With the proposed merge gating system, the
+trunk always remains usable.
 
 
 Technical Challenges
@@ -361,20 +369,17 @@
 in some of our existing tools.
 
 
-Rietveld/Roundup vs Gerrit
---------------------------
+Kallithea vs Gerrit
+-------------------
 
-Rietveld does not currently include a voting/approval feature that is
+Kallithea does not currently include a voting/approval feature that is
 equivalent to Gerrit's. For CPython, we wouldn't need anything as
 sophisticated as Gerrit's voting system - a simple core-developer-only
 "Approved" marker to trigger action from Zuul should suffice. The
 core-developer-or-not flag is available in Roundup, as is the flag
 indicating whether or not the uploader of a patch has signed a PSF
-Contributor Licensing Agreement, which may require further additions to
-the existing integration between the two tools.
-
-Rietveld may also require some changes to allow the uploader of a patch
-to indicate which branch it is intended for.
+Contributor Licensing Agreement, which may require further development to
+link contributor accounts between the Kallithea instance and Roundup.
 
 We would likely also want to improve the existing patch handling,
 in particular looking at how the Roundup/Reitveld integration handles cases
@@ -386,29 +391,33 @@
 Some of the existing Zuul triggers work by monitoring for particular comments
 (in particular, recheck/reverify comments to ask Zuul to try merging a
 change again if it was previously rejected due to an unrelated intermittent
-failure). We will likely also want similar explicit triggers for Rietveld.
+failure). We will likely also want similar explicit triggers for Kallithea.
 
 The current Zuul plugins for Gerrit work by monitoring the Gerrit activity
-stream for particular events. If Rietveld has no equivalent, we will need
+stream for particular events. If Kallithea has no equivalent, we will need
 to add something suitable for the events we would like to trigger on.
 
 There would also be development effort needed to create a Zuul plugin
-that monitors Rietveld activity rather than Gerrit.
+that monitors Kallithea activity rather than Gerrit.
 
 
 Mercurial vs Gerrit/git
 -----------------------
 
 Gerrit uses git as the actual storage mechanism for patches, and
-automatically handles merging of approved patches. By contrast, Rietveld
-works directly on patches, and is completely decoupled from any specific
-version control system.
+automatically handles merging of approved patches. By contrast, Kallithea
+use the RhodeCode created `vcs <https://pythonhosted.org/vcs/>` library as
+an abstraction layer over specific DVCS implementations (with Mercurial and
+git backends currently available).
 
 Zuul is also directly integrated with git for patch manipulation - as far
-as I am aware, this part of the design isn't pluggable. However, at PyCon
-US 2014, the Mercurial core developers at the sprints expressed some
-interest in collaborating with the core development team and the Zuul
+as I am aware, this part of the design currently isn't pluggable. However,
+at PyCon US 2014, the Mercurial core developers at the sprints expressed
+some interest in collaborating with the core development team and the Zuul
 developers on enabling the use of Zuul with Mercurial in addition to git.
+As Zuul is itself a Python application, migrating it to use the same DVCS
+abstraction library as RhodeCode and Kallithea may be a viable path towards
+achieving that.
 
 
 Buildbot vs Jenkins
@@ -463,27 +472,28 @@
 maintenance branches.
 
 Python 2.7 can be handled easily enough by treating it as a separate patch
-queue. This would just require a change in Rietveld to indicate which
-branch was the intended target of the patch.
+queue. This would be handled natively in Kallithea by submitting separate
+pull requests in order to update the Python 2.7 maintenance branch.
 
 The Python 3.x maintenance branches are potentially more complicated. My
 current recommendation is to simply stop using Mercurial merges to manage
 them, and instead treat them as independent heads, similar to the Python
-2.7 branch. Patches that apply cleanly to both the active maintenance branch
-and to default would then just be submitted to both queues, while other
-changes might involve submitting separate patches for the active maintenance
-branch and for default. This approach also has the benefit of adjusting
-cleanly to the intermittent periods where we have two active Python 3
-maintenance branches.
+2.7 branch. Separate pull requests would need to be submitted for the active
+Python 3 maintenance branch and the default development branch. The
+downside of this approach is that it increases the risk that a fix is merged
+only to the maintenance branch without also being submitted to the default
+branch, so we may want to design some additional tooling that ensures that
+every maintenance branch pull request either has a corresponding default
+branch pull request prior to being merged, or else has an explicit disclaimer
+indicating that it is only applicable to that branch and doesn't need to be
+ported forward to later branches.
 
-This does suggest some user interface ideas for the branch nomination
-interface for a patch:
+Such an approach has the benefit of adjusting relatively cleanly to the
+intermittent periods where we have two active Python 3 maintenance branches.
 
-* default to "default" on issues that are marked as "enhancement"
-* default to "3.x+" (where 3.x is the oldest branch in regular maintenance)
-  on any other issues
-* also offer the ability to select specific branches in addition to or
-  instead of the default selection
+This issue does suggest some potential user interface ideas for Kallithea,
+where it may be desirable to be able to clone a pull request in order to be
+able to apply it to a second branch.
 
 
 Handling of security branches
@@ -491,7 +501,10 @@
 
 For simplicity's sake, I would suggest leaving the handling of
 security-fix only branches alone: the release managers for those branches
-would continue to backport specific changes manually.
+would continue to backport specific changes manually. The only change is
+that they would be able to use the Kallithea pull request workflow to do the
+backports if they would like others to review the updates prior to merging
+them.
 
 
 Handling of NEWS file updates
@@ -510,9 +523,9 @@
 -------------------------------------
 
 Instability of the nominally stable buildbots has a substantially larger
-impact under this proposal. We would need to ensure we're happy with each
-of those systems gating merges to the development branches, or else move
-then to "unstable" status.
+impact under this proposal. We would need to ensure we're genuinely happy
+with each of those systems gating merges to the development branches, or
+else move then to "unstable" status.
 
 
 Intermittent test failures
@@ -535,29 +548,20 @@
 hence likely even more amenable to automated analysis.
 
 
-Enhancing Mercurial/Rietveld/Roundup integration
-------------------------------------------------
+Custom Mercurial client workflow support
+----------------------------------------
 
 One useful part of the OpenStack workflow is the "git review" plugin,
 which makes it relatively easy to push a branch from a local git clone up
 to Gerrit for review.
 
-It seems that it should be possible to create a plugin that similarly
-integrates Mercurial queues with Rietveld and Roundup, allowing a draft
-patch to be uploaded as easily as running a command like "hg qpost" with a
-suitable .hgqpost configuration file checked in to the source repo.
+PEP 474 mentions a draft `custom Mercurial
+extension <https://bitbucket.org/ncoghlan/cpydev/src/default/cpyhg.py?at=default>`__
+that automates some aspects of the existing CPython core development workflow.
 
-(There's an existing `hg review <http://sjl.bitbucket.org/hg-review/>`__,
-plugin hence the suggestion of ``hg qpost`` as an alternate command)
-
-It would also be good to work directly with the Mercurial folks to come up
-with a tailored CPython-specific tutorial for using Mercurial queues and
-other extensions to work effectively with the CPython repository structure.
-We have some of that already in the developer guide, but I've come to believe
-that we may want to start getting more opinionated as to which extensions
-we recommend using, especially for users that have previously learned
-``git`` and need to know which extensions to enable to gain a similar level
-of flexibility in their local workflow from Mercurial.
+As part of this proposal, that custom extension would be extended to work
+with the new Kallithea based review workflow in addition to the legacy
+Roundup/Rietveld based review workflow.
 
 
 Social Challenges
@@ -604,9 +608,9 @@
 <https://wiki.python.org/moin/CallForTrackers>`__ rather than in a PEP).
 
 Accordingly, proposals that involve setting ourselves up for "SourceForge
-usability and reliability issues, round two" aren't likely to gain any
-traction with either the CPython core development team or with the PSF
-Infrastructure committee. This proposal respects that history by
+usability and reliability issues, round two" will face significant
+opposition from at least some members of the CPython core development team
+(including the author of this PEP). This proposal respects that history by
 recommending only tools that are available for self-hosting as sponsored
 or PSF funded infrastructure, and are also open source Python projects that
 can be customised to meet the needs of the CPython core development team.
@@ -665,8 +669,9 @@
 ==============
 
 Pretty much everything in the PEP. Do we want to adopt merge gating and
-Zuul? Is Rietveld the right place to hook Zuul into our current workflows?
-How do we want to address the various technical challenges?
+Zuul? How do we want to address the various technical challenges?
+Are the Kallithea and Zuul development communities open to the kind
+of collaboration that would be needed to make this effort a success?
 
 Assuming we do want to do it (or something like it), how is the work going
 to get done? Do we try to get it done solely as a volunteer effort? Do we
@@ -678,14 +683,18 @@
 OpenStack infrastructure team, and the available development resources for
 OpenStack currently dwarf those for CPython?
 
+Do those of us working for Python redistributors and major users (including
+me), attempt to make the business case to our superiors for investing
+developer time in supporting this effort?
+
 
 Next Steps
 ==========
 
-Unfortunately, we ran out of time at the PyCon 2014 language summit to really
-discuss these issues. However, the `core-workflow mailing list
-<https://mail.python.org/mailman/listinfo/core-workflow>`__ has now been set
-up to discuss workflow issues in general.
+If pursued, this will be a follow-on project to the Kallithea-based
+forge.python.org proposal in PEP 474. Refer to that PEP for more details
+on the discussion, review and proof-of-concept pilot process currently
+under way.
 
 
 Acknowledgements
@@ -696,6 +705,11 @@
 Taylor for additional technical feedback following publication of the
 initial draft.
 
+Thanks to Bradley Kuhn, Mads Kiellerich and other Kallithea developers for
+the discussions around PEP 474 that led to a significant revision of this
+proposal to be based on using Kallithea for the review component rather than
+the existing Rietveld installation.
+
 
 Copyright
 =========
diff --git a/pep-0474.txt b/pep-0474.txt
--- a/pep-0474.txt
+++ b/pep-0474.txt
@@ -7,7 +7,7 @@
 Type: Process
 Content-Type: text/x-rst
 Created: 19-Jul-2014
-Post-History: 19-Jul-2014, 08-Jan-2015
+Post-History: 19-Jul-2014, 08-Jan-2015, 01-Feb-2015
 
 
 Abstract
@@ -287,6 +287,61 @@
 forge.python.org hosted Mercurial repositories.
 
 
+Pilot Objectives and Timeline
+=============================
+
+This proposal is part of Brett Cannon's `current evaluation
+<https://mail.python.org/pipermail/python-dev/2014-December/137472.html>`__
+of improvement proposals for various aspects of the CPython development
+workflow. Key dates in that timeline are:
+
+* Feb 1: Draft proposal published (for Kallithea, this PEP)
+* Apr 8: Discussion of final proposals at Python Language Summit
+* May 1: Brett's decision on which proposal to accept
+* Sep 13: Python 3.5 released, adopting new workflows for Python 3.6
+
+Prior to the April 8 discussion, it is proposed to have the following aspects
+of this PEP completed:
+
+* a reference implementation operational at kallithea-pilot.python.org,
+  containing at least the developer guide and PEP repositories. This will
+  be a "throwaway" instance, allowing core developers and other contributors
+  to experiement freely without worrying about the long term consequences for
+  the repository history.
+* read-only live mirrors of the Kallithea hosted repositories on GitHub and
+  BitBucket. As with the pilot service itself, these would be temporary repos,
+  to be discarded after the pilot period ends.
+* clear documentation on using those mirrors to create pull requests against
+  Kallithea hosted Mercurial repositories (for the pilot, this will likely
+  *not* include using the native pull request workflows of those hosted
+  services)
+* automatic linking of issue references in code review comments and commit
+  messages to the corresponding issues on bugs.python.org
+* draft updates to PEP 1 explaining the Kallithea based PEP editing and
+  submission workflow
+
+The following items would be needed for a production migration, but there
+doesn't appear to be an obvious way to trial an updated implementation as
+part of the pilot:
+
+* adjusting the PEP publication process and the developer guide publication
+  process to be based on the relocated Mercurial repos
+
+The following items would be objectives of the overall workflow improvement
+process, but may not be completed before the Python Language summit, and are
+also considered "desirable, but not essential" for the initial adoption of
+the new service in September (if this proposal is the one selected):
+
+* allowing the use of python-social-auth to authenticate against the PSF
+  hosted Kallithea instance
+* allowing the use of the GitHub and BitBucket pull request workflows to
+  submit pull requests to the main Kallithea repo
+* allowing easy triggering of forced BuildBot runs based on Kallithea hosted
+  repos and pull requests (prior to the implementation of PEP 462, this
+  would be intended for use with sandbox repos rather than the main CPython
+  repo)
+
+
 Future Implications for CPython Core Development
 ================================================
 

-- 
Repository URL: https://hg.python.org/peps

From python-checkins at python.org  Sun Feb  1 06:57:26 2015
From: python-checkins at python.org (nick.coghlan)
Date: Sun, 01 Feb 2015 05:57:26 +0000
Subject: [Python-checkins] =?utf-8?q?peps=3A_PEP_477_was_implemented_for_2?=
	=?utf-8?q?=2E7=2E9?=
Message-ID: <20150201055721.25863.27054@psf.io>

https://hg.python.org/peps/rev/ca51fb69dbb7
changeset:   5684:ca51fb69dbb7
user:        Nick Coghlan <ncoghlan at gmail.com>
date:        Sun Feb 01 15:57:13 2015 +1000
summary:
  PEP 477 was implemented for 2.7.9

files:
  pep-0477.txt |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/pep-0477.txt b/pep-0477.txt
--- a/pep-0477.txt
+++ b/pep-0477.txt
@@ -5,7 +5,7 @@
 Author: Donald Stufft <donald at stufft.io>
         Nick Coghlan <ncoghlan at gmail.com>
 BDFL-Delegate: Benjamin Peterson <benjamin at python.org>
-Status: Accepted
+Status: Final
 Type: Standards Track
 Content-Type: text/x-rst
 Created: 26-Aug-2014

-- 
Repository URL: https://hg.python.org/peps

From python-checkins at python.org  Sun Feb  1 06:59:50 2015
From: python-checkins at python.org (nick.coghlan)
Date: Sun, 01 Feb 2015 05:59:50 +0000
Subject: [Python-checkins] =?utf-8?q?peps=3A_PEP_461_has_been_implemented_?=
	=?utf-8?q?for_3=2E5?=
Message-ID: <20150201055949.96068.65696@psf.io>

https://hg.python.org/peps/rev/7fdec54e0566
changeset:   5685:7fdec54e0566
user:        Nick Coghlan <ncoghlan at gmail.com>
date:        Sun Feb 01 15:59:42 2015 +1000
summary:
  PEP 461 has been implemented for 3.5

files:
  pep-0461.txt |  2 +-
  pep-0478.txt |  2 +-
  2 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/pep-0461.txt b/pep-0461.txt
--- a/pep-0461.txt
+++ b/pep-0461.txt
@@ -3,7 +3,7 @@
 Version: $Revision$
 Last-Modified: $Date$
 Author: Ethan Furman <ethan at stoneleaf.us>
-Status: Accepted
+Status: Final
 Type: Standards Track
 Content-Type: text/x-rst
 Created: 2014-01-13
diff --git a/pep-0478.txt b/pep-0478.txt
--- a/pep-0478.txt
+++ b/pep-0478.txt
@@ -58,10 +58,10 @@
 Implemented / Final PEPs:
 
 * PEP 465, a new matrix multiplication operator
+* PEP 461, %-formatting for binary strings
 
 Accepted PEPs:
 
-* PEP 461, %-formatting for binary strings
 * PEP 471, os.scandir()
 
 Proposed changes for 3.5:

-- 
Repository URL: https://hg.python.org/peps

From solipsis at pitrou.net  Sun Feb  1 09:19:34 2015
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Sun, 01 Feb 2015 09:19:34 +0100
Subject: [Python-checkins] Daily reference leaks (469ff344f8fd): sum=3
Message-ID: <E1YHpkk-0000Tw-31@ssl.pitrou.net>

results for 469ff344f8fd on branch "default"
--------------------------------------------

test_functools leaked [0, 0, 3] memory blocks, sum=3


Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogTyMukx', '-x']

From python-checkins at python.org  Sun Feb  1 14:55:34 2015
From: python-checkins at python.org (stefan.krah)
Date: Sun, 01 Feb 2015 13:55:34 +0000
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2322445=3A_PyBuffer?=
 =?utf-8?q?=5FIsContiguous=28=29_now_implements_precise_contiguity?=
Message-ID: <20150201135424.25869.75411@psf.io>

https://hg.python.org/cpython/rev/369300948f3f
changeset:   94423:369300948f3f
user:        Stefan Krah <skrah at bytereef.org>
date:        Sun Feb 01 14:53:54 2015 +0100
summary:
  Issue #22445: PyBuffer_IsContiguous() now implements precise contiguity
tests, compatible with NumPy's NPY_RELAXED_STRIDES_CHECKING compilation
flag.  Previously the function reported false negatives for corner cases.

files:
  Lib/test/test_buffer.py |  13 ++++++
  Misc/NEWS               |   4 ++
  Modules/_testbuffer.c   |  55 +++++++++++++++++++---------
  Objects/abstract.c      |  49 +++++++++++++++++++------
  4 files changed, 91 insertions(+), 30 deletions(-)


diff --git a/Lib/test/test_buffer.py b/Lib/test/test_buffer.py
--- a/Lib/test/test_buffer.py
+++ b/Lib/test/test_buffer.py
@@ -1007,6 +1007,7 @@
         # shape, strides, offset
         structure = (
             ([], [], 0),
+            ([1,3,1], [], 0),
             ([12], [], 0),
             ([12], [-1], 11),
             ([6], [2], 0),
@@ -1078,6 +1079,18 @@
         self.assertRaises(BufferError, ndarray, ex, getbuf=PyBUF_ANY_CONTIGUOUS)
         nd = ndarray(ex, getbuf=PyBUF_SIMPLE)
 
+        # Issue #22445: New precise contiguity definition.
+        for shape in [1,12,1], [7,0,7]:
+            for order in 0, ND_FORTRAN:
+                ex = ndarray(items, shape=shape, flags=order|ND_WRITABLE)
+                self.assertTrue(is_contiguous(ex, 'F'))
+                self.assertTrue(is_contiguous(ex, 'C'))
+
+                for flags in requests:
+                    nd = ndarray(ex, getbuf=flags)
+                    self.assertTrue(is_contiguous(nd, 'F'))
+                    self.assertTrue(is_contiguous(nd, 'C'))
+
     def test_ndarray_exceptions(self):
         nd = ndarray([9], [1])
         ndm = ndarray([9], [1], flags=ND_VAREXPORT)
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -1531,6 +1531,10 @@
 C API
 -----
 
+- Issue #22445: PyBuffer_IsContiguous() now implements precise contiguity
+  tests, compatible with NumPy's NPY_RELAXED_STRIDES_CHECKING compilation
+  flag.  Previously the function reported false negatives for corner cases.
+
 - Issue #22079: PyType_Ready() now checks that statically allocated type has
   no dynamically allocated bases.
 
diff --git a/Modules/_testbuffer.c b/Modules/_testbuffer.c
--- a/Modules/_testbuffer.c
+++ b/Modules/_testbuffer.c
@@ -1510,6 +1510,19 @@
         view->shape = NULL;
     }
 
+    /* Ascertain that the new buffer has the same contiguity as the exporter */
+    if (ND_C_CONTIGUOUS(baseflags) != PyBuffer_IsContiguous(view, 'C') ||
+        /* skip cast to 1-d */
+        (view->format != NULL && view->shape != NULL &&
+         ND_FORTRAN_CONTIGUOUS(baseflags) != PyBuffer_IsContiguous(view, 'F')) ||
+        /* cast to 1-d */
+        (view->format == NULL && view->shape == NULL &&
+         !PyBuffer_IsContiguous(view, 'F'))) {
+        PyErr_SetString(PyExc_BufferError,
+            "ndarray: contiguity mismatch in getbuf()");
+            return -1;
+    }
+
     view->obj = (PyObject *)self;
     Py_INCREF(view->obj);
     self->head->exports++;
@@ -2206,6 +2219,8 @@
     for (i = 0; i < base->ndim; i++)
         base->suboffsets[i] = -1;
 
+    nd->head->flags &= ~(ND_C|ND_FORTRAN);
+
     Py_RETURN_NONE;
 }
 
@@ -2469,13 +2484,12 @@
 {
     Py_ssize_t i;
 
-    if (ndim == 1 && shape && shape[0] == 1) {
-        /* This is for comparing strides: For example, the array
-           [175], shape=[1], strides=[-5] is considered contiguous. */
-        return 1;
-    }
 
     for (i = 0; i < ndim; i++) {
+        if (shape && shape[i] <= 1) {
+            /* strides can differ if the dimension is less than 2 */
+            continue;
+        }
         if (a1[i] != a2[i]) {
             return 0;
         }
@@ -2555,30 +2569,35 @@
     PyObject *obj;
     PyObject *order;
     PyObject *ret = NULL;
-    Py_buffer view;
+    Py_buffer view, *base;
     char ord;
 
     if (!PyArg_ParseTuple(args, "OO", &obj, &order)) {
         return NULL;
     }
 
-    if (PyObject_GetBuffer(obj, &view, PyBUF_FULL_RO) < 0) {
-        PyErr_SetString(PyExc_TypeError,
-            "is_contiguous: object does not implement the buffer "
-            "protocol");
+    ord = get_ascii_order(order);
+    if (ord == CHAR_MAX) {
         return NULL;
     }
 
-    ord = get_ascii_order(order);
-    if (ord == CHAR_MAX) {
-        goto release;
+    if (NDArray_Check(obj)) {
+        /* Skip the buffer protocol to check simple etc. buffers directly. */
+        base = &((NDArrayObject *)obj)->head->base;
+        ret = PyBuffer_IsContiguous(base, ord) ? Py_True : Py_False;
     }
-
-    ret = PyBuffer_IsContiguous(&view, ord) ? Py_True : Py_False;
+    else {
+        if (PyObject_GetBuffer(obj, &view, PyBUF_FULL_RO) < 0) {
+            PyErr_SetString(PyExc_TypeError,
+                "is_contiguous: object does not implement the buffer "
+                "protocol");
+            return NULL;
+        }
+        ret = PyBuffer_IsContiguous(&view, ord) ? Py_True : Py_False;
+        PyBuffer_Release(&view);
+    }
+
     Py_INCREF(ret);
-
-release:
-    PyBuffer_Release(&view);
     return ret;
 }
 
diff --git a/Objects/abstract.c b/Objects/abstract.c
--- a/Objects/abstract.c
+++ b/Objects/abstract.c
@@ -367,16 +367,35 @@
     Py_ssize_t sd, dim;
     int i;
 
-    if (view->ndim == 0) return 1;
-    if (view->strides == NULL) return (view->ndim == 1);
+    /* 1) len = product(shape) * itemsize
+       2) itemsize > 0
+       3) len = 0 <==> exists i: shape[i] = 0 */
+    if (view->len == 0) return 1;
+    if (view->strides == NULL) {  /* C-contiguous by definition */
+        /* Trivially F-contiguous */
+        if (view->ndim <= 1) return 1;
+
+        /* ndim > 1 implies shape != NULL */
+        assert(view->shape != NULL);
+
+        /* Effectively 1-d */
+        sd = 0;
+        for (i=0; i<view->ndim; i++) {
+            if (view->shape[i] > 1) sd += 1;
+        }
+        return sd <= 1;
+    }
+
+    /* strides != NULL implies both of these */
+    assert(view->ndim > 0);
+    assert(view->shape != NULL);
 
     sd = view->itemsize;
-    if (view->ndim == 1) return (view->shape[0] == 1 ||
-                               sd == view->strides[0]);
     for (i=0; i<view->ndim; i++) {
         dim = view->shape[i];
-        if (dim == 0) return 1;
-        if (view->strides[i] != sd) return 0;
+        if (dim > 1 && view->strides[i] != sd) {
+            return 0;
+        }
         sd *= dim;
     }
     return 1;
@@ -388,16 +407,22 @@
     Py_ssize_t sd, dim;
     int i;
 
-    if (view->ndim == 0) return 1;
-    if (view->strides == NULL) return 1;
+    /* 1) len = product(shape) * itemsize
+       2) itemsize > 0
+       3) len = 0 <==> exists i: shape[i] = 0 */
+    if (view->len == 0) return 1;
+    if (view->strides == NULL) return 1; /* C-contiguous by definition */
+
+    /* strides != NULL implies both of these */
+    assert(view->ndim > 0);
+    assert(view->shape != NULL);
 
     sd = view->itemsize;
-    if (view->ndim == 1) return (view->shape[0] == 1 ||
-                               sd == view->strides[0]);
     for (i=view->ndim-1; i>=0; i--) {
         dim = view->shape[i];
-        if (dim == 0) return 1;
-        if (view->strides[i] != sd) return 0;
+        if (dim > 1 && view->strides[i] != sd) {
+            return 0;
+        }
         sd *= dim;
     }
     return 1;

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Sun Feb  1 16:18:49 2015
From: python-checkins at python.org (vinay.sajip)
Date: Sun, 01 Feb 2015 15:18:49 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?q?=29=3A_Merged_documentation_update_from_3=2E4=2E?=
Message-ID: <20150201151825.25847.56941@psf.io>

https://hg.python.org/cpython/rev/6405709f4f82
changeset:   94426:6405709f4f82
parent:      94423:369300948f3f
parent:      94425:35fa000406c4
user:        Vinay Sajip <vinay_sajip at yahoo.co.uk>
date:        Sun Feb 01 15:18:14 2015 +0000
summary:
  Merged documentation update from 3.4.

files:
  Doc/howto/logging-cookbook.rst |  54 ++++++++++++++++++++++
  1 files changed, 54 insertions(+), 0 deletions(-)


diff --git a/Doc/howto/logging-cookbook.rst b/Doc/howto/logging-cookbook.rst
--- a/Doc/howto/logging-cookbook.rst
+++ b/Doc/howto/logging-cookbook.rst
@@ -2088,3 +2088,57 @@
 While the above treatment is simplistic, it points the way to how exception
 information can be formatted to your liking. The :mod:`traceback` module may be
 helpful for more specialized needs.
+
+.. _spoken-messages:
+
+Speaking logging messages
+-------------------------
+
+There might be situations when it is desirable to have logging messages rendered
+in an audible rather than a visible format. This is easy to do if you have text-
+to-speech (TTS) functionality available in your system, even if it doesn't have
+a Python binding. Most TTS systems have a command line program you can run, and
+this can be invoked from a handler using :mod:`subprocess`. It's assumed here
+that TTS command line programs won't expect to interact with users or take a
+long time to complete, and that the frequency of logged messages will be not so
+high as to swamp the user with messages, and that it's acceptable to have the
+messages spoken one at a time rather than concurrently, The example implementation
+below waits for one message to be spoken before the next is processed, and this
+might cause other handlers to be kept waiting. Here is a short example showing
+the approach, which assumes that the ``espeak`` TTS package is available::
+
+    import logging
+    import subprocess
+    import sys
+
+    class TTSHandler(logging.Handler):
+        def emit(self, record):
+            msg = self.format(record)
+            # Speak slowly in a female English voice
+            cmd = ['espeak', '-s150', '-ven+f3', msg]
+            p = subprocess.Popen(cmd, stdout=subprocess.PIPE,
+                                 stderr=subprocess.STDOUT)
+            # wait for the program to finish
+            p.communicate()
+
+    def configure_logging():
+        h = TTSHandler()
+        root = logging.getLogger()
+        root.addHandler(h)
+        # the default formatter just returns the message
+        root.setLevel(logging.DEBUG)
+
+    def main():
+        logging.info('Hello')
+        logging.debug('Goodbye')
+
+    if __name__ == '__main__':
+        configure_logging()
+        sys.exit(main())
+
+When run, this script should say "Hello" and then "Goodbye" in a female voice.
+
+The above approach can, of course, be adapted to other TTS systems and even
+other systems altogether which can process messages via external programs run
+from a command line.
+

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Sun Feb  1 16:18:49 2015
From: python-checkins at python.org (vinay.sajip)
Date: Sun, 01 Feb 2015 15:18:49 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Added_a_cookbo?=
 =?utf-8?q?ok_entry_on_logging_audible_messages=2E?=
Message-ID: <20150201151824.39290.39018@psf.io>

https://hg.python.org/cpython/rev/35fa000406c4
changeset:   94425:35fa000406c4
branch:      3.4
parent:      94420:8049954bc0e4
user:        Vinay Sajip <vinay_sajip at yahoo.co.uk>
date:        Sun Feb 01 15:17:34 2015 +0000
summary:
  Added a cookbook entry on logging audible messages.

files:
  Doc/howto/logging-cookbook.rst |  54 ++++++++++++++++++++++
  1 files changed, 54 insertions(+), 0 deletions(-)


diff --git a/Doc/howto/logging-cookbook.rst b/Doc/howto/logging-cookbook.rst
--- a/Doc/howto/logging-cookbook.rst
+++ b/Doc/howto/logging-cookbook.rst
@@ -2088,3 +2088,57 @@
 While the above treatment is simplistic, it points the way to how exception
 information can be formatted to your liking. The :mod:`traceback` module may be
 helpful for more specialized needs.
+
+.. _spoken-messages:
+
+Speaking logging messages
+-------------------------
+
+There might be situations when it is desirable to have logging messages rendered
+in an audible rather than a visible format. This is easy to do if you have text-
+to-speech (TTS) functionality available in your system, even if it doesn't have
+a Python binding. Most TTS systems have a command line program you can run, and
+this can be invoked from a handler using :mod:`subprocess`. It's assumed here
+that TTS command line programs won't expect to interact with users or take a
+long time to complete, and that the frequency of logged messages will be not so
+high as to swamp the user with messages, and that it's acceptable to have the
+messages spoken one at a time rather than concurrently, The example implementation
+below waits for one message to be spoken before the next is processed, and this
+might cause other handlers to be kept waiting. Here is a short example showing
+the approach, which assumes that the ``espeak`` TTS package is available::
+
+    import logging
+    import subprocess
+    import sys
+
+    class TTSHandler(logging.Handler):
+        def emit(self, record):
+            msg = self.format(record)
+            # Speak slowly in a female English voice
+            cmd = ['espeak', '-s150', '-ven+f3', msg]
+            p = subprocess.Popen(cmd, stdout=subprocess.PIPE,
+                                 stderr=subprocess.STDOUT)
+            # wait for the program to finish
+            p.communicate()
+
+    def configure_logging():
+        h = TTSHandler()
+        root = logging.getLogger()
+        root.addHandler(h)
+        # the default formatter just returns the message
+        root.setLevel(logging.DEBUG)
+
+    def main():
+        logging.info('Hello')
+        logging.debug('Goodbye')
+
+    if __name__ == '__main__':
+        configure_logging()
+        sys.exit(main())
+
+When run, this script should say "Hello" and then "Goodbye" in a female voice.
+
+The above approach can, of course, be adapted to other TTS systems and even
+other systems altogether which can process messages via external programs run
+from a command line.
+

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Sun Feb  1 16:18:49 2015
From: python-checkins at python.org (vinay.sajip)
Date: Sun, 01 Feb 2015 15:18:49 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Added_a_cookbo?=
 =?utf-8?q?ok_entry_on_logging_audible_messages=2E?=
Message-ID: <20150201151824.106327.84528@psf.io>

https://hg.python.org/cpython/rev/7ee7d9fac852
changeset:   94424:7ee7d9fac852
branch:      2.7
parent:      94419:c8d7df3cb854
user:        Vinay Sajip <vinay_sajip at yahoo.co.uk>
date:        Sun Feb 01 15:14:03 2015 +0000
summary:
  Added a cookbook entry on logging audible messages.

files:
  Doc/howto/logging-cookbook.rst |  54 ++++++++++++++++++++++
  1 files changed, 54 insertions(+), 0 deletions(-)


diff --git a/Doc/howto/logging-cookbook.rst b/Doc/howto/logging-cookbook.rst
--- a/Doc/howto/logging-cookbook.rst
+++ b/Doc/howto/logging-cookbook.rst
@@ -1114,3 +1114,57 @@
 While the above treatment is simplistic, it points the way to how exception
 information can be formatted to your liking. The :mod:`traceback` module may be
 helpful for more specialized needs.
+
+.. _spoken-messages:
+
+Speaking logging messages
+-------------------------
+
+There might be situations when it is desirable to have logging messages rendered
+in an audible rather than a visible format. This is easy to do if you have text-
+to-speech (TTS) functionality available in your system, even if it doesn't have
+a Python binding. Most TTS systems have a command line program you can run, and
+this can be invoked from a handler using :mod:`subprocess`. It's assumed here
+that TTS command line programs won't expect to interact with users or take a
+long time to complete, and that the frequency of logged messages will be not so
+high as to swamp the user with messages, and that it's acceptable to have the
+messages spoken one at a time rather than concurrently, The example implementation
+below waits for one message to be spoken before the next is processed, and this
+might cause other handlers to be kept waiting. Here is a short example showing
+the approach, which assumes that the ``espeak`` TTS package is available::
+
+    import logging
+    import subprocess
+    import sys
+
+    class TTSHandler(logging.Handler):
+        def emit(self, record):
+            msg = self.format(record)
+            # Speak slowly in a female English voice
+            cmd = ['espeak', '-s150', '-ven+f3', msg]
+            p = subprocess.Popen(cmd, stdout=subprocess.PIPE,
+                                 stderr=subprocess.STDOUT)
+            # wait for the program to finish
+            p.communicate()
+
+    def configure_logging():
+        h = TTSHandler()
+        root = logging.getLogger()
+        root.addHandler(h)
+        # the default formatter just returns the message
+        root.setLevel(logging.DEBUG)
+
+    def main():
+        logging.info('Hello')
+        logging.debug('Goodbye')
+
+    if __name__ == '__main__':
+        configure_logging()
+        sys.exit(main())
+
+When run, this script should say "Hello" and then "Goodbye" in a female voice.
+
+The above approach can, of course, be adapted to other TTS systems and even
+other systems altogether which can process messages via external programs run
+from a command line.
+

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Sun Feb  1 16:26:41 2015
From: python-checkins at python.org (donald.stufft)
Date: Sun, 01 Feb 2015 15:26:41 +0000
Subject: [Python-checkins] =?utf-8?q?peps=3A_Switch_to_proposing_a_full_mi?=
 =?utf-8?q?gration_to_Git=2C_Github=2C_and_Phabricator?=
Message-ID: <20150201152622.96082.96221@psf.io>

https://hg.python.org/peps/rev/0a92b2d4967b
changeset:   5686:0a92b2d4967b
user:        Donald Stufft <donald at stufft.io>
date:        Sun Feb 01 10:26:15 2015 -0500
summary:
  Switch to proposing a full migration to Git, Github, and Phabricator

files:
  pep-0481.txt |  448 +++++++++++++++++++++++---------------
  1 files changed, 271 insertions(+), 177 deletions(-)


diff --git a/pep-0481.txt b/pep-0481.txt
--- a/pep-0481.txt
+++ b/pep-0481.txt
@@ -1,5 +1,5 @@
 PEP: 481
-Title: Migrate Some Supporting Repositories to Git and Github
+Title: Migrate CPython to Git, Github, and Phabricator
 Version: $Revision$
 Last-Modified: $Date$
 Author: Donald Stufft <donald at stufft.io>
@@ -13,156 +13,305 @@
 Abstract
 ========
 
-This PEP proposes migrating to Git and Github for certain supporting
-repositories (such as the repository for Python Enhancement Proposals) in a way
-that is more accessible to new contributors, and easier to manage for core
-developers. This is offered as an alternative to PEP 474 which aims to achieve
-the same overall benefits but while continuing to use the Mercurial DVCS and
-without relying on a commerical entity.
-
-In particular this PEP proposes changes to the following repositories:
-
-* https://hg.python.org/devguide/
-* https://hg.python.org/devinabox/
-* https://hg.python.org/peps/
-
-
-This PEP does not propose any changes to the core development workflow for
-CPython itself.
+This PEP proposes migrating the repository hosting of CPython and the
+supporting repositories to Git and Github. It also proposes adding Phabricator
+as an alternative to Github Pull Requests to handle reviewing changes. This
+particular PEP is offered as an alternative to PEP 474 and PEP 462 which aims
+to achieve the same overall benefits but restricts itself to tools that support
+Mercurial and are completely Open Source.
 
 
 Rationale
 =========
 
-As PEP 474 mentions, there are currently a number of repositories hosted on
-hg.python.org which are not directly used for the development of CPython but
-instead are supporting or ancillary repositories. These supporting repositories
-do not typically have complex workflows or often branches at all other than the
-primary integration branch. This simplicity makes them very good targets for
-the "Pull Request" workflow that is commonly found on sites like Github.
+CPython is an open source project which relies on a number of volunteers
+donating their time. As an open source project it relies on attracting new
+volunteers as well as retaining existing ones in order to continue to have
+a healthy amount of manpower available. In addition to increasing the amount of
+manpower that is available to the project, it also needs to allow for effective
+use of what manpower *is* available.
 
-However whereas PEP 474 proposes to continue to use Mercurial and restricts
-itself to only solutions which are OSS and self-hosted, this PEP expands the
-scope of that to include migrating to Git and using Github.
+The current toolchain of the CPython project is a custom and unique combination
+of tools which mandates a workflow that is similar to one found in a lot of
+older projects, but which is becoming less and less popular as time goes on.
 
-The existing method of contributing to these repositories generally includes
-generating a patch and either uploading them to bugs.python.org or emailing
-them to peps at python.org. This process is unfriendly towards non-comitter
-contributors as well as cumbersome for comitters seeking to accept the patches
-sent by users. In contrast, the Pull Request workflow style enables non
-technical contributors, especially those who do not know their way around the
-DVCS of choice, to contribute using the web based editor. On the committer
-side, the Pull Requests enable them to tell, before merging, whether or not
-a particular Pull Request will break anything. It also enables them to do a
-simple "push button" merge which does not require them to check out the
-changes locally. Another such feature that is useful in particular for docs,
-is the ability to view a "prose" diff. This Github-specific feature enables
-a committer to view a diff of the rendered output which will hide things like
-reformatting a paragraph and show you what the actual "meat" of the change
-actually is.
+The one-off nature of the CPython toolchain and workflow means that any new
+contributor is going to need spend time learning the tools and workflow before
+they can start contributing to CPython. Once a new contributor goes through
+the process of learning the CPython workflow they also are unlikely to be able
+to take that knowledge and apply it to future projects they wish to contribute
+to. This acts as a barrier to contribution which will scare off potential new
+contributors.
 
+In addition the tooling that CPython uses is under-maintained, antiquated,
+and it lacks important features that enable committers to more effectively use
+their time when reviewing and approving changes. The fact that it is
+under-maintained means that bugs are likely to last for longer, if they ever
+get fixed, as well as it's more likely to go down for extended periods of time.
+The fact that it is antiquated means that it doesn't effectively harness the
+capabilities of the modern web platform. Finally the fact that it lacks several
+important features such as a lack of pre-testing commits and the lack of an
+automatic merge tool means that committers have to do needless busy work to
+commit even the simplest of changes.
 
-Why Git?
---------
 
-Looking at the variety of DVCS which are available today, it becomes fairly
-clear that git has the largest mindshare. The Open Hub (previously Ohloh)
-statistics [#openhub-stats]_ show that currently 37% of the repositories
-indexed by Open Hub are using git which is second only to SVN (which has 48%),
-while Mercurial has just 2% of the indexed repositories (beating only bazaar
-which has 1%). In additon to the Open Hub statistics, a look at the top 100
-projects on PyPI (ordered by total download counts) shows that within the
-Python space itself, the majority of projects use git.
+Version Control System
+----------------------
 
-=== ========= ========== ====== === ====
-Git Mercurial Subversion Bazaar CVS None
-=== ========= ========== ====== === ====
-62  22        7          4      1   1
-=== ========= ========== ====== === ====
+The first decision that needs to be made is the VCS of the primary server side
+repository. Currently the CPython repository, as well as a number of supporting
+repositories, uses Mercurial. When evaluating the VCS we must consider the
+capabilities of the VCS itself as well as the network effect and mindshare of
+the community around that VCS.
 
+There are really only two real options for this, Mercurial and Git. Between the
+two of them the technical capabilities are largely equivilant. For this reason
+this PEP will largely ignore the technical arguments about the VCS system and
+will instead focus on the social aspects.
 
-Chosing a DVCS which has the larger mindshare will make it more likely that any
-particular person who has experience with DVCS at all will be able to
-meaningfully contribute without having to learn a new tool.
+It is not possible to get exact numbers for the number of projects or people
+which are using a particular VCS, however we can infer this by looking at
+several sources of information for what VCS projects are using.
 
-In addition to simply making it more likely that any individual will already
-know how to use git, the number of projects and people using it means that the
-resources for learning the tool are likely to be more fully fleshed out.
-When you run into problems, the likelihood that someone else had that problem
-and posted a question and recieved an answer is also far higher.
+The Open Hub (previously Ohloh) statistics [#openhub-stats]_ show that 37% of
+the repositories indexed by The Open Hub are using Git (second only to SVN
+which has 48%) while Mercurial has just 2% (beating only bazaar which has 1%).
+This has Git being just over 18 times as popular as Mercurial on The Open Hub.
 
-Thirdly, by using a more popular tool you also increase your options for
-tooling *around* the DVCS itself. Looking at the various options for hosting
-repositories, it's extremely rare to find a hosting solution (whether OSS or
-commerical) that supports Mercurial but does not support Git. On the flip side,
-there are a number of tools which support Git but do not support Mercurial.
-Therefore the popularity of git increases the flexibility of our options going
-into the future for what toolchain these projects use.
+Another source of information on the popular of the difference VCSs is PyPI
+itself. This source is more targeted at the Python community itself since it
+represents projects developed for Python. Unfortunately PyPI does not have a
+standard location for representing this information, so this requires manual
+processing. If we limit our search to the top 100 projects on PyPI (ordered
+by download counts) we can see that 62% of them use Git while 22% of them use
+Mercurial while 13% use something else. This has Git being just under 3 times
+as popular as Mercurial for the top 100 projects on PyPI.
 
-Also, by moving to the more popular DVCS, we increase the likelihood that the
-knowledge that the person has learned in contributing to these support
-repositories will transfer to projects outside of the immediate CPython project
-such as to the larger Python community which is primarily using Git hosted on
-Github.
+Obviously from these numbers Git is by far the more popular DVCS for open
+source projects and choosing the more popular VCS has a number of positive
+benefits.
 
-In previous years there was concern about how well supported git was on Windows
-in comparison to Mercurial. However, git has grown to support Windows as a
-first class citizen. In addition to that, for Windows users who are not well
-acquainted with the Windows command line, there are GUI options as well.
+For new contributors it increases the likelihood that they will have already
+learned the basics of Git as part of working with another project or if they
+are just now learning Git, that they'll be able to take that knowledge and
+apply it to other projects. Additionally a larger community means more people
+writing how to guides, answering questions, and writing articles about Git
+which makes it easier for a new user to find answers and information about
+the tool they are trying to learn.
 
+Another benefit is that by nature of having a larger community, there will be
+more tooling written *around* it. This increases options for everything from
+GUI clients, helper scripts, repository hosting, etc.
 
-Why Github?
+
+Repository Hosting
+------------------
+
+This PEP proposes allowing GitHub Pull Requests to be submitted, however GitHub
+does not have a way to submit Pull Requests against a repository that is not
+hosted on GitHub. This PEP also proposes that in addition to GitHub Pull
+Requests Phabricator's Differential app can also be used to submit proposed
+changes and Phabricator *does* allow submitting changes against a repository
+that is not hosted on Phabricator.
+
+For this reason this PEP proposes using GitHub as the canonical location of
+the repository with a read-only mirror located in Phabricator. If at some point
+in the future GitHub is no longer desired, then repository hosting can easily
+be moved to solely in Phabricator and the ability to accept GitHub Pull
+Requests dropped.
+
+In addition to hosting the repositories on Github, a read only copy of all
+repositories will also be mirrored onto the PSF Infrastructure.
+
+
+Code Review
 -----------
 
-There are a number of software projects or web services which offer
-functionality similar to that of Github. These range from commerical web
-services such as Bitbucket to self-hosted OSS solutions such as Kallithea or
-Gitlab. This PEP proposes that we move these repositories to Github.
+Currently CPython uses a custom fork of Rietveld which has been modified to
+not run on Google App Engine which is really only able to be maintained
+currently by one person. In addition it is missing out on features that are
+present in many modern code review tools.
 
-There are two primary reasons for selecting Github: Popularity and
-Quality/Polish.
+This PEP proposes allowing both Github Pull Requests and Phabricator changes
+to propose changes and review code. It suggests both so that contributors can
+select which tool best enables them to submit changes, and reviewers can focus
+on reviewing changes in the tooling they like best.
 
-Github is currently the most popular hosted repository hosting according to
-Alexa, where it currently has a global rank of 121. Much like for Git itself,
-by choosing the most popular tool we gain benefits in increasing the likelihood
-that a new contributor will have already experienced the toolchain, the quality
-and availablity of the help, more and better tooling being built around it, and
-the knowledge transfer to other projects. A look again at the top 100 projects
-by download counts on PyPI shows the following hosting locations:
 
-====== ========= =========== ========= =========== ==========
-GitHub BitBucket Google Code Launchpad SourceForge Other/Self
-====== ========= =========== ========= =========== ==========
-62     18        6           4         3           7
-====== ========= =========== ========= =========== ==========
+GitHub Pull Requests
+~~~~~~~~~~~~~~~~~~~~
 
-In addition to all of those reasons, Github also has the benefit that while
-many of the options have similar features when you look at them in a feature
-matrix the Github version of each of those features tend to work better and be
-far more polished. This is hard to quantify objectively however it is a fairly
-common sentiment if you go around and ask people who are using these services
-often.
+GitHub is a very popular code hosting site and is increasingly becoming the
+primary place people look to contribute to a project. Enabling users to
+contribute through GitHub is enabling contributors to contribute using tooling
+that they are likely already familiar with and if they are not they are likely
+to be able to apply to another project.
 
-Finally, a reason to choose a web service at all over something that is
-self-hosted is to be able to more efficiently use volunteer time and donated
-resources. Every additional service hosted on the PSF infrastructure by the
-PSF infrastructure team further spreads out the amount of time that the
-volunteers on that team have to spend and uses some chunk of resources that
-could potentially be used for something where there is no free or affordable
-hosted solution available.
+GitHub Pull Requests have a fairly major advantage over the older "submit a
+patch to a bug tracker" model. It allows developers to work completely within
+their VCS using standard VCS tooling so it does not require creating a patch
+file and figuring out what the right location is to upload it to. This lowers
+the barrier for sending a change to be reviewed.
 
-One concern that people do have with using a hosted service is that there is a
-lack of control and that at some point in the future the service may no longer
-be suitable. It is the opinion of this PEP that Github does not currently and
-has not in the past engaged in any attempts to lock people into their platform
-and that if at some point in the future Github is no longer suitable for one
-reason or another, then at that point we can look at migrating away from Github
-onto a different solution. In other words, we'll cross that bridge if and when
-we come to it.
+On the reviewing side, GitHub Pull Requests are far easier to review, they have
+nice syntax highlighted diffs which can operate in either unified or side by
+side views. They allow expanding the context on a diff up to and including the
+entire file. Finally they allow commenting inline and on the pull request as
+a whole and they present that in a nice unified way which will also hide
+comments which no longer apply. Github also provides a "rendered diff" view
+which enables easily viewing a diff of rendered markup (such as rst) instead
+of needing to review the diff of the raw markup.
+
+The Pull Request work flow also makes it trivial to enable the ability to
+pre-test a change before actually merging it. Any particular pull request can
+have any number of different types of "commit statuses" applied to it, marking
+the commit (and thus the pull request) as either in a pending, successful,
+errored, or failure state. This makes it easy to see inline if the pull request
+is passing all of the tests, if the contributor has signed a CLA, etc.
+
+Actually merging a Github Pull Request is quite simple, a core reviewer simply
+needs to press the "Merge" button once the status of all the checks on the
+Pull Request are green for successful.
+
+GitHub also has a good workflow for submitting pull requests to a project
+completely through their web interface. This would enable the Python
+documentation to have "Edit on GitHub" buttons on every page and people who
+discover things like typos, inaccuracies, or just want to make improvements to
+the docs they are currently writing can simply hit that button and get an in
+browser editor that will let them make changes and submit a pull request all
+from the comfort of their browser.
+
+
+Phabricator
+~~~~~~~~~~~
+
+In addition to GitHub Pull Requests this PEP also proposes setting up a
+Phabricator instance and pointing it at the GitHub hosted repositories. This
+will allow utilizing the Phabricator review applications of Differential and
+Audit.
+
+Differential functions similarly to GitHub pull requests except that they
+require installing the ``arc`` command line tool to upload patches to
+Phabricator.
+
+Whether to enable Phabricator for any particular repository can be chosen on
+a case by case basis, this PEP only proposes that it must be enabled for the
+CPython repository, however for smaller repositories such as the PEP repository
+it may not be worth the effort.
+
+
+Criticism
+=========
+
+X is not written in Python
+--------------------------
+
+One feature that the current tooling (Mercurial, Rietveld) has is that the
+primary language for all of the pieces are written in Python. It is this PEPs
+belief that we should focus on the *best* tools for the job and not the *best*
+tools that happen to be written in Python. Volunteer time is a precious
+resource to any open source project and we can best respect and utilize that
+time by focusing on the benefits and downsides of the tools themselves rather
+than what language their authors happened to write them in.
+
+One concern is the ability to modify tools to work for us, however one of
+the Goals here is to *not* modify software to work for us and instead adapt
+ourselves to a more standard workflow. This standardization pays off in the
+ability to re-use tools out of the box freeing up developer time to actually
+work on Python itself as well as enabling knowledge sharing between projects.
+
+However if we do need to modify the tooling, Git itself is largely written in
+C the same as CPython itself is. It can also have commands written for it using
+any language, including Python. Phabricator is written in PHP which is a fairly
+common language in the web world and fairly easy to pick up. GitHub itself is
+largely written in Ruby but given that it's not Open Source there is no ability
+to modify it so it's implementation language is completely meaningless.
+
+
+GitHub is not Free/Open Source
+------------------------------
+
+GitHub is a big part of this proposal and someone who tends more to ideology
+rather than practicality may be opposed to this PEP on that grounds alone. It
+is this PEPs belief that while using entirely Free/Open Source software is an
+attractive idea and a noble goal, that valuing the time of the contributors by
+giving them good tooling that is well maintained and that they either already
+know or if they learn it they can apply to other projects is a more important
+concern than treating whether something is Free/Open Source is a hard
+requirement.
+
+However, history has shown us that sometimes benevolent proprietary companies
+can stop being benevolent. This is hedged against in a few ways:
+
+* We are not utilizing the GitHub Issue Tracker, both because it is not
+  powerful enough for CPython but also because for the primary CPython
+  repository the ability to take our issues and put them somewhere else if we
+  ever need to leave GitHub relies on GitHub continuing to allow API access.
+
+* We are utilizing the GitHub Pull Request workflow, however all of those
+  changes live inside of Git. So a mirror of the GitHub repositories can easily
+  contain all of those Pull Requests. We would potentially lose any comments if
+  GitHub suddenly turned "evil", but the changes themselves would still exist.
+
+* We are utilizing the GitHub repository hosting feature, however since this is
+  just git moving away from GitHub is as simple as pushing the repository to
+  a different location. Data portability for the repository itself is extremely
+  high.
+
+* We are also utilizing Phabricator to provide an alternative for people who
+  do not wish to use GitHub. This also acts as a fallback option which will
+  already be in place if we ever need to stop using GitHub.
+
+Relying on GitHub comes with a number of benefits beyond just the benefits of
+the platform itself. Since it is a commercially backed venture it has a full
+time staff responsible for maintaining its services. This includes making sure
+they stay up, making sure they stay patched for various security
+vulnerabilities, and further improving the software and infrastructure as time
+goes on.
+
+
+Mercurial is better than Git
+----------------------------
+
+Whether Mercurial or Git is better on a technical level is a highly subjective
+opinion. This PEP does not state whether the mechanics of Git or Mercurial is
+better and instead focuses on the network effect that is available for either
+option. Since this PEP proposes switching to Git this leaves the people who
+prefer Mercurial out, however those users can easily continue to work with
+Mercurial by using the hg-git [#hg-git]_ extension for Mercurial which will
+let it work with a repository which is Git on the serverside.
+
+
+CPython Workflow is too Complicated
+-----------------------------------
+
+One sentiment that came out of previous discussions was that the multi branch
+model of CPython was too complicated for Github Pull Requests. It is the belief
+of this PEP that statement is not accurate.
+
+Currently any particular change requires manually creating a patch for 2.7 and
+3.x which won't change at all in this regards.
+
+If someone submits a fix for the current stable branch (currently 3.4) the
+GitHub Pull Request workflow can be used to create, in the browser, a Pull
+Request to merge the current stable branch into the master branch (assuming
+there is no merge conflicts). If there is a merge conflict that would need to
+be handled locally. This provides an improvement over the current situation
+where the merge must always happen locally.
+
+Finally if someone submits a fix for the current development branch currently
+then this has to be manually applied to the stable branch if it desired to
+include it there as well. This must also happen locally as well in the new
+workflow, however for minor changes it could easily be accomplished in the
+GitHub web editor.
+
+Looking at this, I do not believe that *any* system can hide the complexities
+involved in maintaining several long running branches. The only thing that the
+tooling can do is make it as easy as possible to submit changes.
 
 
 Example: Scientific Python
---------------------------
+==========================
 
 One of the key ideas behind the move to both git and Github is that a feature
 of a DVCS, the repository hosting, and the workflow used is the social network
@@ -190,66 +339,11 @@
 workflow less reusable with other projects.
 
 
-Migration
-=========
-
-Through the use of hg-git [#hg-git]_ we can easily convert a Mercurial
-repository to a Git repository by simply pushing the Mercurial repository to
-the Git repository. People who wish to continue to use Mercurial locally can
-then use hg-git going into the future using the new Github URL. However they
-will need to re-clone their repositories as using Git as the server seems to
-trigger a one time change of the changeset ids.
-
-As none of the selected repositories have any tags, branches, or bookmarks
-other than the ``default`` branch the migration will simply map the ``default``
-branch in Mercurial to the ``master`` branch in git.
-
-In addition, since none of the selected projects have any great need of a
-complex bug tracker, they will also migrate their issue handling to using the
-GitHub issues.
-
-In addition to the migration of the repository hosting itself there are a
-number of locations for each particular repository which will require updating.
-The bulk of these will simply be changing commands from the hg equivalent to
-the git equivalent.
-
-In particular this will include:
-
-* Updating www.python.org to generate PEPs using a git clone and link to
-  Github.
-* Updating docs.python.org to pull from Github instead of hg.python.org for the
-  devguide.
-* Enabling the ability to send an email to python-checkins at python.org for each
-  push.
-* Enabling the ability to send an IRC message to #python-dev on Freenode for
-  each push.
-* Migrate any issues for these projects to their respective bug tracker on
-  Github.
-* Use hg-git to provide a read-only mirror on hg.python.org which will enable
-  read-only uses of the hg.python.org instances of the specified repositories
-  to remain the same.
-
-This will restore these repositories to similar functionality as they currently
-have. In addition to this the migration will also include enabling testing for
-each pull request using Travis CI [#travisci]_ where possible to ensure that
-a new PR does not break the ability to render the documentation or PEPs.
-
-
-User Access
-===========
-
-Moving to Github would involve adding an additional user account that will need
-to be managed, however it also offers finer grained control, allowing the
-ability to grant someone access to only one particular repository instead of
-the coarser grained ACLs available on hg.python.org.
-
-
 References
 ==========
 
 .. [#openhub-stats] `Open Hub Statistics <https://www.openhub.net/repositories/compare>`
-.. [#hg-git] `hg-git <https://hg-git.github.io/>`
-.. [#travisci] `Travis CI <https://travis-ci.org/>`
+.. [#hg-git] `Hg-Git mercurial plugin <https://hg-git.github.io/>`
 
 
 Copyright

-- 
Repository URL: https://hg.python.org/peps

From python-checkins at python.org  Sun Feb  1 18:06:35 2015
From: python-checkins at python.org (serhiy.storchaka)
Date: Sun, 01 Feb 2015 17:06:35 +0000
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Don=27t_seek_to_the_start_?=
 =?utf-8?q?of_the_file_when_open_ZipFile_with_the_=27w=27_mode?=
Message-ID: <20150201170146.34404.37043@psf.io>

https://hg.python.org/cpython/rev/4f96e9a8eee8
changeset:   94427:4f96e9a8eee8
user:        Serhiy Storchaka <storchaka at gmail.com>
date:        Sun Feb 01 19:01:10 2015 +0200
summary:
  Don't seek to the start of the file when open ZipFile with the 'w' mode
(regression introduced in issue #14099).

files:
  Lib/zipfile.py |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Lib/zipfile.py b/Lib/zipfile.py
--- a/Lib/zipfile.py
+++ b/Lib/zipfile.py
@@ -1002,7 +1002,7 @@
                 # set the modified flag so central directory gets written
                 # even if no files are added to the archive
                 self._didModify = True
-                self.start_dir = 0
+                self.start_dir = self.fp.tell()
             elif mode == 'a':
                 try:
                     # See if file is a zip file

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Sun Feb  1 19:21:07 2015
From: python-checkins at python.org (stefan.krah)
Date: Sun, 01 Feb 2015 18:21:07 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
	=?utf-8?b?KTogTWVyZ2UgZnJvbSAzLjQu?=
Message-ID: <20150201182100.96074.43470@psf.io>

https://hg.python.org/cpython/rev/9835a186573b
changeset:   94430:9835a186573b
parent:      94427:4f96e9a8eee8
parent:      94429:4d95956a748e
user:        Stefan Krah <skrah at bytereef.org>
date:        Sun Feb 01 19:19:49 2015 +0100
summary:
  Merge from 3.4.

files:
  Modules/_testcapimodule.c |  51 +++++++++++++++++++++++++++
  Objects/abstract.c        |   4 +-
  2 files changed, 53 insertions(+), 2 deletions(-)


diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c
--- a/Modules/_testcapimodule.c
+++ b/Modules/_testcapimodule.c
@@ -2468,6 +2468,56 @@
         return NULL;
     return PyMemoryView_FromBuffer(&info);
 }
+ 
+static PyObject *
+test_from_contiguous(PyObject* self, PyObject *noargs)
+{
+    int data[9] = {-1,-1,-1,-1,-1,-1,-1,-1,-1};
+    int init[5] = {0, 1, 2, 3, 4};
+    Py_ssize_t itemsize = sizeof(int);
+    Py_ssize_t shape = 5;
+    Py_ssize_t strides = 2 * itemsize;
+    Py_buffer view = {
+        data,
+        NULL,
+        5 * itemsize,
+        itemsize,
+        1,
+        1,
+        NULL,
+        &shape,
+        &strides,
+        NULL,
+        NULL
+    };
+    int *ptr;
+    int i;
+
+    PyBuffer_FromContiguous(&view, init, view.len, 'C');
+    ptr = view.buf;
+    for (i = 0; i < 5; i++) {
+        if (ptr[2*i] != i) {
+            PyErr_SetString(TestError,
+                "test_from_contiguous: incorrect result");
+            return NULL;
+        }
+    }
+
+    view.buf = &data[8];
+    view.strides[0] = -2 * itemsize;
+
+    PyBuffer_FromContiguous(&view, init, view.len, 'C');
+    ptr = view.buf;
+    for (i = 0; i < 5; i++) {
+        if (*(ptr-2*i) != i) {
+            PyErr_SetString(TestError,
+                "test_from_contiguous: incorrect result");
+            return NULL;
+        }
+    }
+
+    Py_RETURN_NONE;
+}
 
 /* Test that the fatal error from not having a current thread doesn't
    cause an infinite loop.  Run via Lib/test/test_capi.py */
@@ -3128,6 +3178,7 @@
     {"test_string_to_double", (PyCFunction)test_string_to_double, METH_NOARGS},
     {"test_unicode_compare_with_ascii", (PyCFunction)test_unicode_compare_with_ascii, METH_NOARGS},
     {"test_capsule", (PyCFunction)test_capsule, METH_NOARGS},
+    {"test_from_contiguous", (PyCFunction)test_from_contiguous, METH_NOARGS},
     {"getargs_tuple",           getargs_tuple,                   METH_VARARGS},
     {"getargs_keywords", (PyCFunction)getargs_keywords,
       METH_VARARGS|METH_KEYWORDS},
diff --git a/Objects/abstract.c b/Objects/abstract.c
--- a/Objects/abstract.c
+++ b/Objects/abstract.c
@@ -512,7 +512,7 @@
 
     /* Otherwise a more elaborate scheme is needed */
 
-    /* XXX(nnorwitz): need to check for overflow! */
+    /* view->ndim <= 64 */
     indices = (Py_ssize_t *)PyMem_Malloc(sizeof(Py_ssize_t)*(view->ndim));
     if (indices == NULL) {
         PyErr_NoMemory();
@@ -534,10 +534,10 @@
      */
     elements = len / view->itemsize;
     while (elements--) {
-        addone(view->ndim, indices, view->shape);
         ptr = PyBuffer_GetPointer(view, indices);
         memcpy(ptr, src, view->itemsize);
         src += view->itemsize;
+        addone(view->ndim, indices, view->shape);
     }
 
     PyMem_Free(indices);

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Sun Feb  1 19:21:07 2015
From: python-checkins at python.org (stefan.krah)
Date: Sun, 01 Feb 2015 18:21:07 +0000
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIzMzcw?=
 =?utf-8?q?=3A_Fix_off-by-one_error_for_non-contiguous_buffers=2E?=
Message-ID: <20150201182100.34382.45415@psf.io>

https://hg.python.org/cpython/rev/4d95956a748e
changeset:   94429:4d95956a748e
branch:      3.4
parent:      94425:35fa000406c4
user:        Stefan Krah <skrah at bytereef.org>
date:        Sun Feb 01 16:19:23 2015 +0100
summary:
  Issue #23370: Fix off-by-one error for non-contiguous buffers.

files:
  Modules/_testcapimodule.c |  51 +++++++++++++++++++++++++++
  Objects/abstract.c        |   4 +-
  2 files changed, 53 insertions(+), 2 deletions(-)


diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c
--- a/Modules/_testcapimodule.c
+++ b/Modules/_testcapimodule.c
@@ -2467,6 +2467,56 @@
         return NULL;
     return PyMemoryView_FromBuffer(&info);
 }
+ 
+static PyObject *
+test_from_contiguous(PyObject* self, PyObject *noargs)
+{
+    int data[9] = {-1,-1,-1,-1,-1,-1,-1,-1,-1};
+    int init[5] = {0, 1, 2, 3, 4};
+    Py_ssize_t itemsize = sizeof(int);
+    Py_ssize_t shape = 5;
+    Py_ssize_t strides = 2 * itemsize;
+    Py_buffer view = {
+        data,
+        NULL,
+        5 * itemsize,
+        itemsize,
+        1,
+        1,
+        NULL,
+        &shape,
+        &strides,
+        NULL,
+        NULL
+    };
+    int *ptr;
+    int i;
+
+    PyBuffer_FromContiguous(&view, init, view.len, 'C');
+    ptr = view.buf;
+    for (i = 0; i < 5; i++) {
+        if (ptr[2*i] != i) {
+            PyErr_SetString(TestError,
+                "test_from_contiguous: incorrect result");
+            return NULL;
+        }
+    }
+
+    view.buf = &data[8];
+    view.strides[0] = -2 * itemsize;
+
+    PyBuffer_FromContiguous(&view, init, view.len, 'C');
+    ptr = view.buf;
+    for (i = 0; i < 5; i++) {
+        if (*(ptr-2*i) != i) {
+            PyErr_SetString(TestError,
+                "test_from_contiguous: incorrect result");
+            return NULL;
+        }
+    }
+
+    Py_RETURN_NONE;
+}
 
 /* Test that the fatal error from not having a current thread doesn't
    cause an infinite loop.  Run via Lib/test/test_capi.py */
@@ -3031,6 +3081,7 @@
     {"test_string_to_double", (PyCFunction)test_string_to_double, METH_NOARGS},
     {"test_unicode_compare_with_ascii", (PyCFunction)test_unicode_compare_with_ascii, METH_NOARGS},
     {"test_capsule", (PyCFunction)test_capsule, METH_NOARGS},
+    {"test_from_contiguous", (PyCFunction)test_from_contiguous, METH_NOARGS},
     {"getargs_tuple",           getargs_tuple,                   METH_VARARGS},
     {"getargs_keywords", (PyCFunction)getargs_keywords,
       METH_VARARGS|METH_KEYWORDS},
diff --git a/Objects/abstract.c b/Objects/abstract.c
--- a/Objects/abstract.c
+++ b/Objects/abstract.c
@@ -488,7 +488,7 @@
 
     /* Otherwise a more elaborate scheme is needed */
 
-    /* XXX(nnorwitz): need to check for overflow! */
+    /* view->ndim <= 64 */
     indices = (Py_ssize_t *)PyMem_Malloc(sizeof(Py_ssize_t)*(view->ndim));
     if (indices == NULL) {
         PyErr_NoMemory();
@@ -510,10 +510,10 @@
      */
     elements = len / view->itemsize;
     while (elements--) {
-        addone(view->ndim, indices, view->shape);
         ptr = PyBuffer_GetPointer(view, indices);
         memcpy(ptr, src, view->itemsize);
         src += view->itemsize;
+        addone(view->ndim, indices, view->shape);
     }
 
     PyMem_Free(indices);

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Sun Feb  1 19:21:07 2015
From: python-checkins at python.org (stefan.krah)
Date: Sun, 01 Feb 2015 18:21:07 +0000
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzIzMzcw?=
 =?utf-8?q?=3A_Fix_off-by-one_error_for_non-contiguous_buffers=2E?=
Message-ID: <20150201182059.25845.22841@psf.io>

https://hg.python.org/cpython/rev/209776d49a9a
changeset:   94428:209776d49a9a
branch:      2.7
parent:      94424:7ee7d9fac852
user:        Stefan Krah <skrah at bytereef.org>
date:        Sun Feb 01 16:10:35 2015 +0100
summary:
  Issue #23370: Fix off-by-one error for non-contiguous buffers.

files:
  Modules/_testcapimodule.c |  53 +++++++++++++++++++++++++++
  Objects/abstract.c        |   4 +-
  2 files changed, 55 insertions(+), 2 deletions(-)


diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c
--- a/Modules/_testcapimodule.c
+++ b/Modules/_testcapimodule.c
@@ -424,6 +424,58 @@
     Py_RETURN_NONE;
 }
 
+static PyObject *
+test_from_contiguous(PyObject* self, PyObject *noargs)
+{
+    int data[9] = {-1,-1,-1,-1,-1,-1,-1,-1,-1};
+    int init[5] = {0, 1, 2, 3, 4};
+    Py_ssize_t itemsize = sizeof(int);
+    Py_ssize_t shape = 5;
+    Py_ssize_t strides = 2 * itemsize;
+    Py_buffer view = {
+        data,
+        NULL,
+        5 * itemsize,
+        itemsize,
+        1,
+        1,
+        NULL,
+        &shape,
+        &strides,
+        NULL,
+        {0, 0},
+        NULL
+    };
+    int *ptr;
+    int i;
+
+    PyBuffer_FromContiguous(&view, init, view.len, 'C');
+    ptr = view.buf;
+    for (i = 0; i < 5; i++) {
+        if (ptr[2*i] != i) {
+            PyErr_SetString(TestError,
+                "test_from_contiguous: incorrect result");
+            return NULL;
+        }
+    }
+
+    view.buf = &data[8];
+    view.strides[0] = -2 * itemsize;
+
+    PyBuffer_FromContiguous(&view, init, view.len, 'C');
+    ptr = view.buf;
+    for (i = 0; i < 5; i++) {
+        if (*(ptr-2*i) != i) {
+            PyErr_SetString(TestError,
+                "test_from_contiguous: incorrect result");
+            return NULL;
+        }
+    }
+
+    Py_RETURN_NONE;
+}
+
+
 /* Tests of PyLong_{As, From}{Unsigned,}Long(), and (#ifdef HAVE_LONG_LONG)
    PyLong_{As, From}{Unsigned,}LongLong().
 
@@ -1833,6 +1885,7 @@
     {"test_lazy_hash_inheritance",      (PyCFunction)test_lazy_hash_inheritance,METH_NOARGS},
     {"test_broken_memoryview",          (PyCFunction)test_broken_memoryview,METH_NOARGS},
     {"test_to_contiguous",      (PyCFunction)test_to_contiguous, METH_NOARGS},
+    {"test_from_contiguous",    (PyCFunction)test_from_contiguous, METH_NOARGS},
     {"test_long_api",           (PyCFunction)test_long_api,      METH_NOARGS},
     {"test_long_and_overflow", (PyCFunction)test_long_and_overflow,
      METH_NOARGS},
diff --git a/Objects/abstract.c b/Objects/abstract.c
--- a/Objects/abstract.c
+++ b/Objects/abstract.c
@@ -550,7 +550,7 @@
 
     /* Otherwise a more elaborate scheme is needed */
 
-    /* XXX(nnorwitz): need to check for overflow! */
+    /* view->ndim <= 64 */
     indices = (Py_ssize_t *)PyMem_Malloc(sizeof(Py_ssize_t)*(view->ndim));
     if (indices == NULL) {
         PyErr_NoMemory();
@@ -572,10 +572,10 @@
      */
     elements = len / view->itemsize;
     while (elements--) {
-        addone(view->ndim, indices, view->shape);
         ptr = PyBuffer_GetPointer(view, indices);
         memcpy(ptr, src, view->itemsize);
         src += view->itemsize;
+        addone(view->ndim, indices, view->shape);
     }
 
     PyMem_Free(indices);

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Sun Feb  1 19:51:07 2015
From: python-checkins at python.org (stefan.krah)
Date: Sun, 01 Feb 2015 18:51:07 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?q?=29=3A_Issue_=2323352=3A_Merge_from_3=2E4=2E?=
Message-ID: <20150201185055.25845.20792@psf.io>

https://hg.python.org/cpython/rev/5d097a74766f
changeset:   94433:5d097a74766f
parent:      94430:9835a186573b
parent:      94432:de5c8ee002bf
user:        Stefan Krah <skrah at bytereef.org>
date:        Sun Feb 01 19:42:45 2015 +0100
summary:
  Issue #23352: Merge from 3.4.

files:
  Doc/c-api/buffer.rst |  3 +++
  1 files changed, 3 insertions(+), 0 deletions(-)


diff --git a/Doc/c-api/buffer.rst b/Doc/c-api/buffer.rst
--- a/Doc/c-api/buffer.rst
+++ b/Doc/c-api/buffer.rst
@@ -198,6 +198,9 @@
       indicates that no de-referencing should occur (striding in a contiguous
       memory block).
 
+      If all suboffsets are negative (i.e. no de-referencing is needed, then
+      this field must be NULL (the default value). 
+
       This type of array representation is used by the Python Imaging Library
       (PIL). See `complex arrays`_ for further information how to access elements
       of such an array.

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Sun Feb  1 19:51:07 2015
From: python-checkins at python.org (stefan.krah)
Date: Sun, 01 Feb 2015 18:51:07 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Preserve_criti?=
	=?utf-8?q?cal_whitespace_in_Doc/*=2E?=
Message-ID: <20150201185055.39292.28896@psf.io>

https://hg.python.org/cpython/rev/17ab76dead0d
changeset:   94437:17ab76dead0d
branch:      2.7
parent:      94431:c4c1d68b6301
user:        Stefan Krah <skrah at bytereef.org>
date:        Sun Feb 01 19:49:38 2015 +0100
summary:
  Preserve critical whitespace in Doc/*.

files:
  Doc/c-api/buffer.rst |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Doc/c-api/buffer.rst b/Doc/c-api/buffer.rst
--- a/Doc/c-api/buffer.rst
+++ b/Doc/c-api/buffer.rst
@@ -99,7 +99,7 @@
       occur (striding in a contiguous memory block).
 
       If all suboffsets are negative (i.e. no de-referencing is needed, then
-      this field must be NULL (the default value). 
+      this field must be NULL (the default value).
 
       Here is a function that returns a pointer to the element in an N-D array
       pointed to by an N-dimensional index when there are both non-NULL strides

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Sun Feb  1 19:51:07 2015
From: python-checkins at python.org (stefan.krah)
Date: Sun, 01 Feb 2015 18:51:07 +0000
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Whitespace=2E?=
Message-ID: <20150201185055.106370.27932@psf.io>

https://hg.python.org/cpython/rev/5569ae9e2be2
changeset:   94434:5569ae9e2be2
user:        Stefan Krah <skrah at bytereef.org>
date:        Sun Feb 01 19:45:14 2015 +0100
summary:
  Whitespace.

files:
  Doc/c-api/buffer.rst |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Doc/c-api/buffer.rst b/Doc/c-api/buffer.rst
--- a/Doc/c-api/buffer.rst
+++ b/Doc/c-api/buffer.rst
@@ -199,7 +199,7 @@
       memory block).
 
       If all suboffsets are negative (i.e. no de-referencing is needed, then
-      this field must be NULL (the default value). 
+      this field must be NULL (the default value).
 
       This type of array representation is used by the Python Imaging Library
       (PIL). See `complex arrays`_ for further information how to access elements

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Sun Feb  1 19:51:07 2015
From: python-checkins at python.org (stefan.krah)
Date: Sun, 01 Feb 2015 18:51:07 +0000
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIzMzUy?=
 =?utf-8?q?=3A_Document_that_Py=5Fbuffer=2Esuboffsets_must_be_NULL_if_no_s?=
 =?utf-8?q?uboffsets?=
Message-ID: <20150201185054.39274.44459@psf.io>

https://hg.python.org/cpython/rev/de5c8ee002bf
changeset:   94432:de5c8ee002bf
branch:      3.4
parent:      94429:4d95956a748e
user:        Stefan Krah <skrah at bytereef.org>
date:        Sun Feb 01 19:42:12 2015 +0100
summary:
  Issue #23352: Document that Py_buffer.suboffsets must be NULL if no suboffsets
are required.

files:
  Doc/c-api/buffer.rst |  3 +++
  1 files changed, 3 insertions(+), 0 deletions(-)


diff --git a/Doc/c-api/buffer.rst b/Doc/c-api/buffer.rst
--- a/Doc/c-api/buffer.rst
+++ b/Doc/c-api/buffer.rst
@@ -198,6 +198,9 @@
       indicates that no de-referencing should occur (striding in a contiguous
       memory block).
 
+      If all suboffsets are negative (i.e. no de-referencing is needed, then
+      this field must be NULL (the default value). 
+
       This type of array representation is used by the Python Imaging Library
       (PIL). See `complex arrays`_ for further information how to access elements
       of such an array.

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Sun Feb  1 19:51:07 2015
From: python-checkins at python.org (stefan.krah)
Date: Sun, 01 Feb 2015 18:51:07 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Whitespace=2E?=
Message-ID: <20150201185055.106249.38178@psf.io>

https://hg.python.org/cpython/rev/24e3371cec2d
changeset:   94435:24e3371cec2d
branch:      3.4
parent:      94432:de5c8ee002bf
user:        Stefan Krah <skrah at bytereef.org>
date:        Sun Feb 01 19:46:31 2015 +0100
summary:
  Whitespace.

files:
  Doc/c-api/buffer.rst |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Doc/c-api/buffer.rst b/Doc/c-api/buffer.rst
--- a/Doc/c-api/buffer.rst
+++ b/Doc/c-api/buffer.rst
@@ -199,7 +199,7 @@
       memory block).
 
       If all suboffsets are negative (i.e. no de-referencing is needed, then
-      this field must be NULL (the default value). 
+      this field must be NULL (the default value).
 
       This type of array representation is used by the Python Imaging Library
       (PIL). See `complex arrays`_ for further information how to access elements

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Sun Feb  1 19:51:07 2015
From: python-checkins at python.org (stefan.krah)
Date: Sun, 01 Feb 2015 18:51:07 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?q?=29=3A_Merge_cosmetic_change=2E?=
Message-ID: <20150201185055.106379.6461@psf.io>

https://hg.python.org/cpython/rev/582aabcce2fd
changeset:   94436:582aabcce2fd
parent:      94434:5569ae9e2be2
parent:      94435:24e3371cec2d
user:        Stefan Krah <skrah at bytereef.org>
date:        Sun Feb 01 19:47:25 2015 +0100
summary:
  Merge cosmetic change.

files:



-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Sun Feb  1 19:51:07 2015
From: python-checkins at python.org (stefan.krah)
Date: Sun, 01 Feb 2015 18:51:07 +0000
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzIzMzUy?=
 =?utf-8?q?=3A_Document_that_Py=5Fbuffer=2Esuboffsets_must_be_NULL_if_no_s?=
 =?utf-8?q?uboffsets?=
Message-ID: <20150201185054.39296.30133@psf.io>

https://hg.python.org/cpython/rev/c4c1d68b6301
changeset:   94431:c4c1d68b6301
branch:      2.7
parent:      94428:209776d49a9a
user:        Stefan Krah <skrah at bytereef.org>
date:        Sun Feb 01 19:40:50 2015 +0100
summary:
  Issue #23352: Document that Py_buffer.suboffsets must be NULL if no suboffsets
are required.

files:
  Doc/c-api/buffer.rst |  5 ++++-
  1 files changed, 4 insertions(+), 1 deletions(-)


diff --git a/Doc/c-api/buffer.rst b/Doc/c-api/buffer.rst
--- a/Doc/c-api/buffer.rst
+++ b/Doc/c-api/buffer.rst
@@ -98,8 +98,11 @@
       suboffset value that it negative indicates that no de-referencing should
       occur (striding in a contiguous memory block).
 
+      If all suboffsets are negative (i.e. no de-referencing is needed, then
+      this field must be NULL (the default value). 
+
       Here is a function that returns a pointer to the element in an N-D array
-      pointed to by an N-dimesional index when there are both non-NULL strides
+      pointed to by an N-dimensional index when there are both non-NULL strides
       and suboffsets::
 
           void *get_item_pointer(int ndim, void *buf, Py_ssize_t *strides,

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Mon Feb  2 00:02:56 2015
From: python-checkins at python.org (benjamin.peterson)
Date: Sun, 01 Feb 2015 23:02:56 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
	=?utf-8?b?KTogbWVyZ2UgMy40?=
Message-ID: <20150201230228.39278.47884@psf.io>

https://hg.python.org/cpython/rev/cf4f7a572b79
changeset:   94443:cf4f7a572b79
parent:      94440:02aeca4974ac
parent:      94442:a8737f0bbb7a
user:        Benjamin Peterson <benjamin at python.org>
date:        Sun Feb 01 18:02:21 2015 -0500
summary:
  merge 3.4

files:
  Lib/test/test_json/test_encode_basestring_ascii.py |  1 -
  1 files changed, 0 insertions(+), 1 deletions(-)


diff --git a/Lib/test/test_json/test_encode_basestring_ascii.py b/Lib/test/test_json/test_encode_basestring_ascii.py
--- a/Lib/test/test_json/test_encode_basestring_ascii.py
+++ b/Lib/test/test_json/test_encode_basestring_ascii.py
@@ -45,4 +45,3 @@
         s = "\uffff"*((2**32)//6 + 1)
         with self.assertRaises(OverflowError):
             self.json.encoder.encode_basestring_ascii(s)
-

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Mon Feb  2 00:02:56 2015
From: python-checkins at python.org (benjamin.peterson)
Date: Sun, 01 Feb 2015 23:02:56 +0000
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4zIC0+IDMuNCk6?=
	=?utf-8?q?_merge_3=2E3?=
Message-ID: <20150201230228.25847.54178@psf.io>

https://hg.python.org/cpython/rev/a8737f0bbb7a
changeset:   94442:a8737f0bbb7a
branch:      3.4
parent:      94439:4f47509d7417
parent:      94441:1801b2571587
user:        Benjamin Peterson <benjamin at python.org>
date:        Sun Feb 01 18:02:15 2015 -0500
summary:
  merge 3.3

files:
  Lib/test/test_json/test_encode_basestring_ascii.py |  1 -
  1 files changed, 0 insertions(+), 1 deletions(-)


diff --git a/Lib/test/test_json/test_encode_basestring_ascii.py b/Lib/test/test_json/test_encode_basestring_ascii.py
--- a/Lib/test/test_json/test_encode_basestring_ascii.py
+++ b/Lib/test/test_json/test_encode_basestring_ascii.py
@@ -48,4 +48,3 @@
         s = "\uffff"*((2**32)//6 + 1)
         with self.assertRaises(OverflowError):
             self.json.encoder.encode_basestring_ascii(s)
-

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Mon Feb  2 00:02:56 2015
From: python-checkins at python.org (benjamin.peterson)
Date: Sun, 01 Feb 2015 23:02:56 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_fix_possible_o?=
 =?utf-8?q?verflow_in_encode=5Fbasestring=5Fascii_=28closes_=2323369=29?=
Message-ID: <20150201230227.25859.84821@psf.io>

https://hg.python.org/cpython/rev/8699b3085db3
changeset:   94438:8699b3085db3
branch:      3.3
parent:      94340:6caed177a028
user:        Benjamin Peterson <benjamin at python.org>
date:        Sun Feb 01 17:53:53 2015 -0500
summary:
  fix possible overflow in encode_basestring_ascii (closes #23369)

files:
  Lib/test/test_json/test_encode_basestring_ascii.py |   9 +++++-
  Misc/NEWS                                          |   6 ++++
  Modules/_json.c                                    |  15 +++++++--
  3 files changed, 25 insertions(+), 5 deletions(-)


diff --git a/Lib/test/test_json/test_encode_basestring_ascii.py b/Lib/test/test_json/test_encode_basestring_ascii.py
--- a/Lib/test/test_json/test_encode_basestring_ascii.py
+++ b/Lib/test/test_json/test_encode_basestring_ascii.py
@@ -1,5 +1,6 @@
 from collections import OrderedDict
 from test.test_json import PyTest, CTest
+from test.support import bigaddrspacetest
 
 
 CASES = [
@@ -41,4 +42,10 @@
 
 
 class TestPyEncodeBasestringAscii(TestEncodeBasestringAscii, PyTest): pass
-class TestCEncodeBasestringAscii(TestEncodeBasestringAscii, CTest): pass
+class TestCEncodeBasestringAscii(TestEncodeBasestringAscii, CTest):
+    @bigaddrspacetest
+    def test_overflow(self):
+        s = "\uffff"*((2**32)//6 + 1)
+        with self.assertRaises(OverflowError):
+            self.json.encoder.encode_basestring_ascii(s)
+
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -13,6 +13,12 @@
 - Issue #23055: Fixed a buffer overflow in PyUnicode_FromFormatV.  Analysis
   and fix by Guido Vranken.
 
+Library
+-------
+
+- Issue #23369: Fixed possible integer overflow in
+  _json.encode_basestring_ascii.
+
 
 What's New in Python 3.3.6?
 ===========================
diff --git a/Modules/_json.c b/Modules/_json.c
--- a/Modules/_json.c
+++ b/Modules/_json.c
@@ -216,17 +216,24 @@
     /* Compute the output size */
     for (i = 0, output_size = 2; i < input_chars; i++) {
         Py_UCS4 c = PyUnicode_READ(kind, input, i);
-        if (S_CHAR(c))
-            output_size++;
+        Py_ssize_t d;
+        if (S_CHAR(c)) {
+            d = 1;
+        }
         else {
             switch(c) {
             case '\\': case '"': case '\b': case '\f':
             case '\n': case '\r': case '\t':
-                output_size += 2; break;
+                d = 2; break;
             default:
-                output_size += c >= 0x10000 ? 12 : 6;
+                d = c >= 0x10000 ? 12 : 6;
             }
         }
+        if (output_size > PY_SSIZE_T_MAX - d) {
+            PyErr_SetString(PyExc_OverflowError, "string is too long to escape");
+            return NULL;
+        }
+        output_size += d;
     }
 
     rval = PyUnicode_New(output_size, 127);

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Mon Feb  2 00:02:56 2015
From: python-checkins at python.org (benjamin.peterson)
Date: Sun, 01 Feb 2015 23:02:56 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_remove_extra_w?=
	=?utf-8?q?s?=
Message-ID: <20150201230227.96098.99@psf.io>

https://hg.python.org/cpython/rev/1801b2571587
changeset:   94441:1801b2571587
branch:      3.3
parent:      94438:8699b3085db3
user:        Benjamin Peterson <benjamin at python.org>
date:        Sun Feb 01 18:02:09 2015 -0500
summary:
  remove extra ws

files:
  Lib/test/test_json/test_encode_basestring_ascii.py |  1 -
  1 files changed, 0 insertions(+), 1 deletions(-)


diff --git a/Lib/test/test_json/test_encode_basestring_ascii.py b/Lib/test/test_json/test_encode_basestring_ascii.py
--- a/Lib/test/test_json/test_encode_basestring_ascii.py
+++ b/Lib/test/test_json/test_encode_basestring_ascii.py
@@ -48,4 +48,3 @@
         s = "\uffff"*((2**32)//6 + 1)
         with self.assertRaises(OverflowError):
             self.json.encoder.encode_basestring_ascii(s)
-

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Mon Feb  2 00:02:56 2015
From: python-checkins at python.org (benjamin.peterson)
Date: Sun, 01 Feb 2015 23:02:56 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?b?KTogbWVyZ2UgMy40ICgjMjMzNjkp?=
Message-ID: <20150201230227.96094.94005@psf.io>

https://hg.python.org/cpython/rev/02aeca4974ac
changeset:   94440:02aeca4974ac
parent:      94436:582aabcce2fd
parent:      94439:4f47509d7417
user:        Benjamin Peterson <benjamin at python.org>
date:        Sun Feb 01 18:00:19 2015 -0500
summary:
  merge 3.4 (#23369)

files:
  Lib/test/test_json/test_encode_basestring_ascii.py |   9 +++++-
  Misc/NEWS                                          |   3 ++
  Modules/_json.c                                    |  15 +++++++--
  3 files changed, 22 insertions(+), 5 deletions(-)


diff --git a/Lib/test/test_json/test_encode_basestring_ascii.py b/Lib/test/test_json/test_encode_basestring_ascii.py
--- a/Lib/test/test_json/test_encode_basestring_ascii.py
+++ b/Lib/test/test_json/test_encode_basestring_ascii.py
@@ -1,5 +1,6 @@
 from collections import OrderedDict
 from test.test_json import PyTest, CTest
+from test.support import bigaddrspacetest
 
 
 CASES = [
@@ -38,4 +39,10 @@
 
 
 class TestPyEncodeBasestringAscii(TestEncodeBasestringAscii, PyTest): pass
-class TestCEncodeBasestringAscii(TestEncodeBasestringAscii, CTest): pass
+class TestCEncodeBasestringAscii(TestEncodeBasestringAscii, CTest):
+    @bigaddrspacetest
+    def test_overflow(self):
+        s = "\uffff"*((2**32)//6 + 1)
+        with self.assertRaises(OverflowError):
+            self.json.encoder.encode_basestring_ascii(s)
+
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -229,6 +229,9 @@
 - Issue #23326: Removed __ne__ implementations.  Since fixing default __ne__
   implementation in issue #21408 they are redundant.
 
+- Issue #23369: Fixed possible integer overflow in
+  _json.encode_basestring_ascii.
+
 - Issue #23353: Fix the exception handling of generators in
   PyEval_EvalFrameEx(). At entry, save or swap the exception state even if
   PyEval_EvalFrameEx() is called with throwflag=0. At exit, the exception state
diff --git a/Modules/_json.c b/Modules/_json.c
--- a/Modules/_json.c
+++ b/Modules/_json.c
@@ -182,17 +182,24 @@
     /* Compute the output size */
     for (i = 0, output_size = 2; i < input_chars; i++) {
         Py_UCS4 c = PyUnicode_READ(kind, input, i);
-        if (S_CHAR(c))
-            output_size++;
+        Py_ssize_t d;
+        if (S_CHAR(c)) {
+            d = 1;
+        }
         else {
             switch(c) {
             case '\\': case '"': case '\b': case '\f':
             case '\n': case '\r': case '\t':
-                output_size += 2; break;
+                d = 2; break;
             default:
-                output_size += c >= 0x10000 ? 12 : 6;
+                d = c >= 0x10000 ? 12 : 6;
             }
         }
+        if (output_size > PY_SSIZE_T_MAX - d) {
+            PyErr_SetString(PyExc_OverflowError, "string is too long to escape");
+            return NULL;
+        }
+        output_size += d;
     }
 
     rval = PyUnicode_New(output_size, 127);

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Mon Feb  2 00:02:56 2015
From: python-checkins at python.org (benjamin.peterson)
Date: Sun, 01 Feb 2015 23:02:56 +0000
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4zIC0+IDMuNCk6?=
 =?utf-8?q?_merge_3=2E3_=28=2323369=29?=
Message-ID: <20150201230227.39270.86230@psf.io>

https://hg.python.org/cpython/rev/4f47509d7417
changeset:   94439:4f47509d7417
branch:      3.4
parent:      94435:24e3371cec2d
parent:      94438:8699b3085db3
user:        Benjamin Peterson <benjamin at python.org>
date:        Sun Feb 01 17:59:49 2015 -0500
summary:
  merge 3.3 (#23369)

files:
  Lib/test/test_json/test_encode_basestring_ascii.py |   9 +++++-
  Misc/NEWS                                          |   3 ++
  Modules/_json.c                                    |  15 +++++++--
  3 files changed, 22 insertions(+), 5 deletions(-)


diff --git a/Lib/test/test_json/test_encode_basestring_ascii.py b/Lib/test/test_json/test_encode_basestring_ascii.py
--- a/Lib/test/test_json/test_encode_basestring_ascii.py
+++ b/Lib/test/test_json/test_encode_basestring_ascii.py
@@ -1,5 +1,6 @@
 from collections import OrderedDict
 from test.test_json import PyTest, CTest
+from test.support import bigaddrspacetest
 
 
 CASES = [
@@ -41,4 +42,10 @@
 
 
 class TestPyEncodeBasestringAscii(TestEncodeBasestringAscii, PyTest): pass
-class TestCEncodeBasestringAscii(TestEncodeBasestringAscii, CTest): pass
+class TestCEncodeBasestringAscii(TestEncodeBasestringAscii, CTest):
+    @bigaddrspacetest
+    def test_overflow(self):
+        s = "\uffff"*((2**32)//6 + 1)
+        with self.assertRaises(OverflowError):
+            self.json.encoder.encode_basestring_ascii(s)
+
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -50,6 +50,9 @@
 Library
 -------
 
+- Issue #23369: Fixed possible integer overflow in
+  _json.encode_basestring_ascii.
+
 - Issue #23353: Fix the exception handling of generators in
   PyEval_EvalFrameEx(). At entry, save or swap the exception state even if
   PyEval_EvalFrameEx() is called with throwflag=0. At exit, the exception state
diff --git a/Modules/_json.c b/Modules/_json.c
--- a/Modules/_json.c
+++ b/Modules/_json.c
@@ -182,17 +182,24 @@
     /* Compute the output size */
     for (i = 0, output_size = 2; i < input_chars; i++) {
         Py_UCS4 c = PyUnicode_READ(kind, input, i);
-        if (S_CHAR(c))
-            output_size++;
+        Py_ssize_t d;
+        if (S_CHAR(c)) {
+            d = 1;
+        }
         else {
             switch(c) {
             case '\\': case '"': case '\b': case '\f':
             case '\n': case '\r': case '\t':
-                output_size += 2; break;
+                d = 2; break;
             default:
-                output_size += c >= 0x10000 ? 12 : 6;
+                d = c >= 0x10000 ? 12 : 6;
             }
         }
+        if (output_size > PY_SSIZE_T_MAX - d) {
+            PyErr_SetString(PyExc_OverflowError, "string is too long to escape");
+            return NULL;
+        }
+        output_size += d;
     }
 
     rval = PyUnicode_New(output_size, 127);

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Mon Feb  2 02:18:36 2015
From: python-checkins at python.org (benjamin.peterson)
Date: Mon, 02 Feb 2015 01:18:36 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
	=?utf-8?b?KTogbWVyZ2UgMy40?=
Message-ID: <20150202011835.106317.94625@psf.io>

https://hg.python.org/cpython/rev/3231a13a0c82
changeset:   94447:3231a13a0c82
parent:      94443:cf4f7a572b79
parent:      94445:fb65ac5aaaac
user:        Benjamin Peterson <benjamin at python.org>
date:        Sun Feb 01 20:18:29 2015 -0500
summary:
  merge 3.4

files:
  Lib/site.py           |  2 +-
  Lib/test/test_site.py |  1 +
  2 files changed, 2 insertions(+), 1 deletions(-)


diff --git a/Lib/site.py b/Lib/site.py
--- a/Lib/site.py
+++ b/Lib/site.py
@@ -366,7 +366,7 @@
         dirs.extend([os.path.join(here, os.pardir), here, os.curdir])
     builtins.license = _sitebuiltins._Printer(
         "license",
-        "See http://www.python.org/psf/license/",
+        "See https://www.python.org/psf/license/",
         files, dirs)
 
 
diff --git a/Lib/test/test_site.py b/Lib/test/test_site.py
--- a/Lib/test/test_site.py
+++ b/Lib/test/test_site.py
@@ -410,6 +410,7 @@
                 self.fail("sitecustomize not imported automatically")
 
     @test.support.requires_resource('network')
+    @test.support.system_must_validate_cert
     @unittest.skipUnless(sys.version_info[3] == 'final',
                          'only for released versions')
     @unittest.skipUnless(hasattr(urllib.request, "HTTPSHandler"),

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Mon Feb  2 02:18:36 2015
From: python-checkins at python.org (benjamin.peterson)
Date: Mon, 02 Feb 2015 01:18:36 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_https_goodness?=
Message-ID: <20150202011835.39274.52073@psf.io>

https://hg.python.org/cpython/rev/fb65ac5aaaac
changeset:   94445:fb65ac5aaaac
branch:      3.4
user:        Benjamin Peterson <benjamin at python.org>
date:        Sun Feb 01 20:17:22 2015 -0500
summary:
  https goodness

files:
  Lib/site.py |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Lib/site.py b/Lib/site.py
--- a/Lib/site.py
+++ b/Lib/site.py
@@ -373,7 +373,7 @@
         dirs.extend([os.path.join(here, os.pardir), here, os.curdir])
     builtins.license = _sitebuiltins._Printer(
         "license",
-        "See http://www.python.org/psf/license/",
+        "See https://www.python.org/psf/license/",
         files, dirs)
 
 

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Mon Feb  2 02:18:36 2015
From: python-checkins at python.org (benjamin.peterson)
Date: Mon, 02 Feb 2015 01:18:36 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_https_goodness?=
Message-ID: <20150202011835.96070.24107@psf.io>

https://hg.python.org/cpython/rev/1cc37c52fed4
changeset:   94446:1cc37c52fed4
branch:      2.7
parent:      94437:17ab76dead0d
user:        Benjamin Peterson <benjamin at python.org>
date:        Sun Feb 01 20:17:22 2015 -0500
summary:
  https goodness

files:
  Lib/site.py |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Lib/site.py b/Lib/site.py
--- a/Lib/site.py
+++ b/Lib/site.py
@@ -436,7 +436,7 @@
     for supporting Python development.  See www.python.org for more information.""")
     here = os.path.dirname(os.__file__)
     __builtin__.license = _Printer(
-        "license", "See http://www.python.org/psf/license/",
+        "license", "See https://www.python.org/psf/license/",
         ["LICENSE.txt", "LICENSE"],
         [os.path.join(here, os.pardir), here, os.curdir])
 

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Mon Feb  2 02:18:36 2015
From: python-checkins at python.org (benjamin.peterson)
Date: Mon, 02 Feb 2015 01:18:36 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_fix_tests_on_s?=
 =?utf-8?q?ystems_that_can=27t_validate_python=2Eorg?=
Message-ID: <20150202011834.96084.25941@psf.io>

https://hg.python.org/cpython/rev/4bda963dcc11
changeset:   94444:4bda963dcc11
branch:      3.4
parent:      94442:a8737f0bbb7a
user:        Benjamin Peterson <benjamin at python.org>
date:        Sun Feb 01 20:16:59 2015 -0500
summary:
  fix tests on systems that can't validate python.org

files:
  Lib/test/test_site.py |  1 +
  1 files changed, 1 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_site.py b/Lib/test/test_site.py
--- a/Lib/test/test_site.py
+++ b/Lib/test/test_site.py
@@ -412,6 +412,7 @@
                 self.fail("sitecustomize not imported automatically")
 
     @test.support.requires_resource('network')
+    @test.support.system_must_validate_cert
     @unittest.skipUnless(sys.version_info[3] == 'final',
                          'only for released versions')
     @unittest.skipUnless(hasattr(urllib.request, "HTTPSHandler"),

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Mon Feb  2 03:05:12 2015
From: python-checkins at python.org (benjamin.peterson)
Date: Mon, 02 Feb 2015 02:05:12 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_detect_overflo?=
 =?utf-8?q?w_in_combinations_=28closes_=2323366=29?=
Message-ID: <20150202020454.106346.23397@psf.io>

https://hg.python.org/cpython/rev/fe203370c049
changeset:   94451:fe203370c049
branch:      2.7
parent:      94446:1cc37c52fed4
user:        Benjamin Peterson <benjamin at python.org>
date:        Sun Feb 01 20:59:00 2015 -0500
summary:
  detect overflow in combinations (closes #23366)

files:
  Lib/test/test_itertools.py |  5 +++++
  Misc/NEWS                  |  2 ++
  Modules/itertoolsmodule.c  |  4 ++++
  3 files changed, 11 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_itertools.py b/Lib/test/test_itertools.py
--- a/Lib/test/test_itertools.py
+++ b/Lib/test/test_itertools.py
@@ -137,6 +137,11 @@
                 self.assertEqual(result, list(combinations2(values, r))) # matches second pure python version
                 self.assertEqual(result, list(combinations3(values, r))) # matches second pure python version
 
+    @test_support.bigaddrspacetest
+    def test_combinations_overflow(self):
+        with self.assertRaises(OverflowError):
+            combinations("AA", 2**29)
+
     @test_support.impl_detail("tuple reuse is specific to CPython")
     def test_combinations_tuple_reuse(self):
         self.assertEqual(len(set(map(id, combinations('abcde', 3)))), 1)
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -18,6 +18,8 @@
 Library
 -------
 
+- Issue #23366: Fixed possible integer overflow in itertools.combinations.
+
 - Issue #23191: fnmatch functions that use caching are now threadsafe.
 
 - Issue #18518: timeit now rejects statements which can't be compiled outside
diff --git a/Modules/itertoolsmodule.c b/Modules/itertoolsmodule.c
--- a/Modules/itertoolsmodule.c
+++ b/Modules/itertoolsmodule.c
@@ -2093,6 +2093,10 @@
         goto error;
     }
 
+    if (r > PY_SSIZE_T_MAX/sizeof(Py_ssize_t)) {
+        PyErr_SetString(PyExc_OverflowError, "r is too big");
+        goto error;
+    }
     indices = PyMem_Malloc(r * sizeof(Py_ssize_t));
     if (indices == NULL) {
         PyErr_NoMemory();

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Mon Feb  2 03:05:12 2015
From: python-checkins at python.org (benjamin.peterson)
Date: Mon, 02 Feb 2015 02:05:12 +0000
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4zIC0+IDMuNCk6?=
 =?utf-8?q?_merge_3=2E3_=28=2323366=29?=
Message-ID: <20150202020454.25867.11193@psf.io>

https://hg.python.org/cpython/rev/2f73de7ffcf5
changeset:   94449:2f73de7ffcf5
branch:      3.4
parent:      94445:fb65ac5aaaac
parent:      94448:014886dae5c4
user:        Benjamin Peterson <benjamin at python.org>
date:        Sun Feb 01 21:00:15 2015 -0500
summary:
  merge 3.3 (#23366)

files:
  Lib/test/test_itertools.py |  5 +++++
  Misc/NEWS                  |  2 ++
  Modules/itertoolsmodule.c  |  4 ++++
  3 files changed, 11 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_itertools.py b/Lib/test/test_itertools.py
--- a/Lib/test/test_itertools.py
+++ b/Lib/test/test_itertools.py
@@ -264,6 +264,11 @@
                 for proto in range(pickle.HIGHEST_PROTOCOL + 1):
                     self.pickletest(proto, combinations(values, r))      # test pickling
 
+    @support.bigaddrspacetest
+    def test_combinations_overflow(self):
+        with self.assertRaises(OverflowError):
+            combinations("AA", 2**29)
+
         # Test implementation detail:  tuple re-use
     @support.impl_detail("tuple reuse is specific to CPython")
     def test_combinations_tuple_reuse(self):
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -50,6 +50,8 @@
 Library
 -------
 
+- Issue #23366: Fixed possible integer overflow in itertools.combinations.
+
 - Issue #23369: Fixed possible integer overflow in
   _json.encode_basestring_ascii.
 
diff --git a/Modules/itertoolsmodule.c b/Modules/itertoolsmodule.c
--- a/Modules/itertoolsmodule.c
+++ b/Modules/itertoolsmodule.c
@@ -2359,6 +2359,10 @@
         goto error;
     }
 
+    if (r > PY_SSIZE_T_MAX/sizeof(Py_ssize_t)) {
+        PyErr_SetString(PyExc_OverflowError, "r is too big");
+        goto error;
+    }
     indices = PyMem_Malloc(r * sizeof(Py_ssize_t));
     if (indices == NULL) {
         PyErr_NoMemory();

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Mon Feb  2 03:05:12 2015
From: python-checkins at python.org (benjamin.peterson)
Date: Mon, 02 Feb 2015 02:05:12 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?b?KTogbWVyZ2UgMy40ICgjMjMzNjYp?=
Message-ID: <20150202020454.39288.20159@psf.io>

https://hg.python.org/cpython/rev/886559229911
changeset:   94450:886559229911
parent:      94447:3231a13a0c82
parent:      94449:2f73de7ffcf5
user:        Benjamin Peterson <benjamin at python.org>
date:        Sun Feb 01 21:01:43 2015 -0500
summary:
  merge 3.4 (#23366)

files:
  Lib/test/test_itertools.py |  5 +++++
  Misc/NEWS                  |  2 ++
  Modules/itertoolsmodule.c  |  4 ++++
  3 files changed, 11 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_itertools.py b/Lib/test/test_itertools.py
--- a/Lib/test/test_itertools.py
+++ b/Lib/test/test_itertools.py
@@ -264,6 +264,11 @@
                 for proto in range(pickle.HIGHEST_PROTOCOL + 1):
                     self.pickletest(proto, combinations(values, r))      # test pickling
 
+    @support.bigaddrspacetest
+    def test_combinations_overflow(self):
+        with self.assertRaises(OverflowError):
+            combinations("AA", 2**29)
+
         # Test implementation detail:  tuple re-use
     @support.impl_detail("tuple reuse is specific to CPython")
     def test_combinations_tuple_reuse(self):
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -229,6 +229,8 @@
 - Issue #23326: Removed __ne__ implementations.  Since fixing default __ne__
   implementation in issue #21408 they are redundant.
 
+- Issue #23366: Fixed possible integer overflow in itertools.combinations.
+
 - Issue #23369: Fixed possible integer overflow in
   _json.encode_basestring_ascii.
 
diff --git a/Modules/itertoolsmodule.c b/Modules/itertoolsmodule.c
--- a/Modules/itertoolsmodule.c
+++ b/Modules/itertoolsmodule.c
@@ -2359,6 +2359,10 @@
         goto error;
     }
 
+    if (r > PY_SSIZE_T_MAX/sizeof(Py_ssize_t)) {
+        PyErr_SetString(PyExc_OverflowError, "r is too big");
+        goto error;
+    }
     indices = PyMem_Malloc(r * sizeof(Py_ssize_t));
     if (indices == NULL) {
         PyErr_NoMemory();

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Mon Feb  2 03:05:12 2015
From: python-checkins at python.org (benjamin.peterson)
Date: Mon, 02 Feb 2015 02:05:12 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_detect_overflo?=
 =?utf-8?q?w_in_combinations_=28closes_=2323366=29?=
Message-ID: <20150202020453.34396.20345@psf.io>

https://hg.python.org/cpython/rev/014886dae5c4
changeset:   94448:014886dae5c4
branch:      3.3
parent:      94441:1801b2571587
user:        Benjamin Peterson <benjamin at python.org>
date:        Sun Feb 01 20:59:00 2015 -0500
summary:
  detect overflow in combinations (closes #23366)

files:
  Lib/test/test_itertools.py |  5 +++++
  Misc/NEWS                  |  2 ++
  Modules/itertoolsmodule.c  |  4 ++++
  3 files changed, 11 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_itertools.py b/Lib/test/test_itertools.py
--- a/Lib/test/test_itertools.py
+++ b/Lib/test/test_itertools.py
@@ -258,6 +258,11 @@
 
                 self.pickletest(combinations(values, r))                 # test pickling
 
+    @support.bigaddrspacetest
+    def test_combinations_overflow(self):
+        with self.assertRaises(OverflowError):
+            combinations("AA", 2**29)
+
         # Test implementation detail:  tuple re-use
     @support.impl_detail("tuple reuse is specific to CPython")
     def test_combinations_tuple_reuse(self):
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -19,6 +19,8 @@
 - Issue #23369: Fixed possible integer overflow in
   _json.encode_basestring_ascii.
 
+- Issue #23366: Fixed possible integer overflow in itertools.combinations.
+
 
 What's New in Python 3.3.6?
 ===========================
diff --git a/Modules/itertoolsmodule.c b/Modules/itertoolsmodule.c
--- a/Modules/itertoolsmodule.c
+++ b/Modules/itertoolsmodule.c
@@ -2326,6 +2326,10 @@
         goto error;
     }
 
+    if (r > PY_SSIZE_T_MAX/sizeof(Py_ssize_t)) {
+        PyErr_SetString(PyExc_OverflowError, "r is too big");
+        goto error;
+    }
     indices = PyMem_Malloc(r * sizeof(Py_ssize_t));
     if (indices == NULL) {
         PyErr_NoMemory();

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Mon Feb  2 03:13:50 2015
From: python-checkins at python.org (benjamin.peterson)
Date: Mon, 02 Feb 2015 02:13:50 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_check_for_over?=
 =?utf-8?q?flow_in_combinations=5Fwith=5Freplacement_=28closes_=2323365=29?=
Message-ID: <20150202021350.34382.1937@psf.io>

https://hg.python.org/cpython/rev/93d445cd5f70
changeset:   94452:93d445cd5f70
branch:      3.3
parent:      94448:014886dae5c4
user:        Benjamin Peterson <benjamin at python.org>
date:        Sun Feb 01 21:10:47 2015 -0500
summary:
  check for overflow in combinations_with_replacement (closes #23365)

files:
  Lib/test/test_itertools.py |  6 +++++-
  Misc/NEWS                  |  3 +++
  Modules/itertoolsmodule.c  |  4 ++++
  3 files changed, 12 insertions(+), 1 deletions(-)


diff --git a/Lib/test/test_itertools.py b/Lib/test/test_itertools.py
--- a/Lib/test/test_itertools.py
+++ b/Lib/test/test_itertools.py
@@ -344,8 +344,12 @@
 
                 self.pickletest(cwr(values,r))                          # test pickling
 
+    @support.bigaddrspacetest
+    def test_combinations_with_replacement_overflow(self):
+        with self.assertRaises(OverflowError):
+            combinations_with_replacement("AA", 2**30)
+
         # Test implementation detail:  tuple re-use
-
     @support.impl_detail("tuple reuse is specific to CPython")
     def test_combinations_with_replacement_tuple_reuse(self):
         cwr = combinations_with_replacement
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -21,6 +21,9 @@
 
 - Issue #23366: Fixed possible integer overflow in itertools.combinations.
 
+- Issue #23365: Fixed possible integer overflow in
+  itertools.combinations_with_replacement.
+
 
 What's New in Python 3.3.6?
 ===========================
diff --git a/Modules/itertoolsmodule.c b/Modules/itertoolsmodule.c
--- a/Modules/itertoolsmodule.c
+++ b/Modules/itertoolsmodule.c
@@ -2659,6 +2659,10 @@
         goto error;
     }
 
+    if (r > PY_SSIZE_T_MAX/sizeof(Py_ssize_t)) {
+        PyErr_SetString(PyExc_OverflowError, "r is too big");
+        goto error;
+    }
     indices = PyMem_Malloc(r * sizeof(Py_ssize_t));
     if (indices == NULL) {
         PyErr_NoMemory();

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Mon Feb  2 03:13:51 2015
From: python-checkins at python.org (benjamin.peterson)
Date: Mon, 02 Feb 2015 02:13:51 +0000
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4zIC0+IDMuNCk6?=
 =?utf-8?q?_merge_3=2E3_=28=2323365=29?=
Message-ID: <20150202021350.39276.15842@psf.io>

https://hg.python.org/cpython/rev/2e7a02e4cf2c
changeset:   94453:2e7a02e4cf2c
branch:      3.4
parent:      94449:2f73de7ffcf5
parent:      94452:93d445cd5f70
user:        Benjamin Peterson <benjamin at python.org>
date:        Sun Feb 01 21:11:39 2015 -0500
summary:
  merge 3.3 (#23365)

files:
  Lib/test/test_itertools.py |  6 +++++-
  Misc/NEWS                  |  2 ++
  Modules/itertoolsmodule.c  |  4 ++++
  3 files changed, 11 insertions(+), 1 deletions(-)


diff --git a/Lib/test/test_itertools.py b/Lib/test/test_itertools.py
--- a/Lib/test/test_itertools.py
+++ b/Lib/test/test_itertools.py
@@ -351,8 +351,12 @@
                 for proto in range(pickle.HIGHEST_PROTOCOL + 1):
                     self.pickletest(proto, cwr(values,r))               # test pickling
 
+    @support.bigaddrspacetest
+    def test_combinations_with_replacement_overflow(self):
+        with self.assertRaises(OverflowError):
+            combinations_with_replacement("AA", 2**30)
+
         # Test implementation detail:  tuple re-use
-
     @support.impl_detail("tuple reuse is specific to CPython")
     def test_combinations_with_replacement_tuple_reuse(self):
         cwr = combinations_with_replacement
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -52,6 +52,8 @@
 
 - Issue #23366: Fixed possible integer overflow in itertools.combinations.
 
+- Issue #23366: Fixed possible integer overflow in itertools.combinations.
+
 - Issue #23369: Fixed possible integer overflow in
   _json.encode_basestring_ascii.
 
diff --git a/Modules/itertoolsmodule.c b/Modules/itertoolsmodule.c
--- a/Modules/itertoolsmodule.c
+++ b/Modules/itertoolsmodule.c
@@ -2704,6 +2704,10 @@
         goto error;
     }
 
+    if (r > PY_SSIZE_T_MAX/sizeof(Py_ssize_t)) {
+        PyErr_SetString(PyExc_OverflowError, "r is too big");
+        goto error;
+    }
     indices = PyMem_Malloc(r * sizeof(Py_ssize_t));
     if (indices == NULL) {
         PyErr_NoMemory();

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Mon Feb  2 03:13:51 2015
From: python-checkins at python.org (benjamin.peterson)
Date: Mon, 02 Feb 2015 02:13:51 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?b?KTogbWVyZ2UgMy40ICgjMjMzNjUp?=
Message-ID: <20150202021350.96098.81716@psf.io>

https://hg.python.org/cpython/rev/4d875a690c01
changeset:   94454:4d875a690c01
parent:      94450:886559229911
parent:      94453:2e7a02e4cf2c
user:        Benjamin Peterson <benjamin at python.org>
date:        Sun Feb 01 21:11:54 2015 -0500
summary:
  merge 3.4 (#23365)

files:
  Lib/test/test_itertools.py |  6 +++++-
  Misc/NEWS                  |  2 ++
  Modules/itertoolsmodule.c  |  4 ++++
  3 files changed, 11 insertions(+), 1 deletions(-)


diff --git a/Lib/test/test_itertools.py b/Lib/test/test_itertools.py
--- a/Lib/test/test_itertools.py
+++ b/Lib/test/test_itertools.py
@@ -351,8 +351,12 @@
                 for proto in range(pickle.HIGHEST_PROTOCOL + 1):
                     self.pickletest(proto, cwr(values,r))               # test pickling
 
+    @support.bigaddrspacetest
+    def test_combinations_with_replacement_overflow(self):
+        with self.assertRaises(OverflowError):
+            combinations_with_replacement("AA", 2**30)
+
         # Test implementation detail:  tuple re-use
-
     @support.impl_detail("tuple reuse is specific to CPython")
     def test_combinations_with_replacement_tuple_reuse(self):
         cwr = combinations_with_replacement
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -231,6 +231,8 @@
 
 - Issue #23366: Fixed possible integer overflow in itertools.combinations.
 
+- Issue #23366: Fixed possible integer overflow in itertools.combinations.
+
 - Issue #23369: Fixed possible integer overflow in
   _json.encode_basestring_ascii.
 
diff --git a/Modules/itertoolsmodule.c b/Modules/itertoolsmodule.c
--- a/Modules/itertoolsmodule.c
+++ b/Modules/itertoolsmodule.c
@@ -2704,6 +2704,10 @@
         goto error;
     }
 
+    if (r > PY_SSIZE_T_MAX/sizeof(Py_ssize_t)) {
+        PyErr_SetString(PyExc_OverflowError, "r is too big");
+        goto error;
+    }
     indices = PyMem_Malloc(r * sizeof(Py_ssize_t));
     if (indices == NULL) {
         PyErr_NoMemory();

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Mon Feb  2 03:13:51 2015
From: python-checkins at python.org (benjamin.peterson)
Date: Mon, 02 Feb 2015 02:13:51 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_check_for_over?=
 =?utf-8?q?flow_in_combinations=5Fwith=5Freplacement_=28closes_=2323365=29?=
Message-ID: <20150202021351.106293.11136@psf.io>

https://hg.python.org/cpython/rev/366018a91457
changeset:   94455:366018a91457
branch:      2.7
parent:      94451:fe203370c049
user:        Benjamin Peterson <benjamin at python.org>
date:        Sun Feb 01 21:10:47 2015 -0500
summary:
  check for overflow in combinations_with_replacement (closes #23365)

files:
  Lib/test/test_itertools.py |  5 +++++
  Misc/NEWS                  |  3 +++
  Modules/itertoolsmodule.c  |  4 ++++
  3 files changed, 12 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_itertools.py b/Lib/test/test_itertools.py
--- a/Lib/test/test_itertools.py
+++ b/Lib/test/test_itertools.py
@@ -213,6 +213,11 @@
                 self.assertEqual(result, list(cwr1(values, r)))         # matches first pure python version
                 self.assertEqual(result, list(cwr2(values, r)))         # matches second pure python version
 
+    @test_support.bigaddrspacetest
+    def test_combinations_with_replacement_overflow(self):
+        with self.assertRaises(OverflowError):
+            combinations_with_replacement("AA", 2**30)
+
     @test_support.impl_detail("tuple reuse is specific to CPython")
     def test_combinations_with_replacement_tuple_reuse(self):
         cwr = combinations_with_replacement
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -18,6 +18,9 @@
 Library
 -------
 
+- Issue #23365: Fixed possible integer overflow in
+  itertools.combinations_with_replacement.
+
 - Issue #23366: Fixed possible integer overflow in itertools.combinations.
 
 - Issue #23191: fnmatch functions that use caching are now threadsafe.
diff --git a/Modules/itertoolsmodule.c b/Modules/itertoolsmodule.c
--- a/Modules/itertoolsmodule.c
+++ b/Modules/itertoolsmodule.c
@@ -2346,6 +2346,10 @@
         goto error;
     }
 
+    if (r > PY_SSIZE_T_MAX/sizeof(Py_ssize_t)) {
+        PyErr_SetString(PyExc_OverflowError, "r is too big");
+        goto error;
+    }
     indices = PyMem_Malloc(r * sizeof(Py_ssize_t));
     if (indices == NULL) {
         PyErr_NoMemory();

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Mon Feb  2 03:39:20 2015
From: python-checkins at python.org (benjamin.peterson)
Date: Mon, 02 Feb 2015 02:39:20 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_check_for_over?=
 =?utf-8?q?flows_in_permutations=28=29_and_product=28=29_=28closes_=232336?=
 =?utf-8?q?3=2C_closes?=
Message-ID: <20150202023854.106357.36009@psf.io>

https://hg.python.org/cpython/rev/7133582b6769
changeset:   94456:7133582b6769
branch:      3.3
parent:      94452:93d445cd5f70
user:        Benjamin Peterson <benjamin at python.org>
date:        Sun Feb 01 21:34:07 2015 -0500
summary:
  check for overflows in permutations() and product() (closes #23363, closes #23364)

files:
  Lib/test/test_itertools.py |  12 ++++++++++++
  Misc/NEWS                  |   4 ++++
  Modules/itertoolsmodule.c  |  18 ++++++++++++++++--
  3 files changed, 32 insertions(+), 2 deletions(-)


diff --git a/Lib/test/test_itertools.py b/Lib/test/test_itertools.py
--- a/Lib/test/test_itertools.py
+++ b/Lib/test/test_itertools.py
@@ -418,6 +418,13 @@
 
                 self.pickletest(permutations(values, r))                # test pickling
 
+    @support.bigaddrspacetest
+    def test_permutations_overflow(self):
+        with self.assertRaises(OverflowError):
+            permutations("A", 2**30)
+        with self.assertRaises(OverflowError):
+            permutations("A", 2, 2**30)
+
     @support.impl_detail("tuple resuse is CPython specific")
     def test_permutations_tuple_reuse(self):
         self.assertEqual(len(set(map(id, permutations('abcde', 3)))), 1)
@@ -930,6 +937,11 @@
             args = map(iter, args)
             self.assertEqual(len(list(product(*args))), expected_len)
 
+    @support.bigaddrspacetest
+    def test_product_overflow(self):
+        with self.assertRaises(OverflowError):
+            product(["a"]*(2**16), repeat=2**16)
+
     @support.impl_detail("tuple reuse is specific to CPython")
     def test_product_tuple_reuse(self):
         self.assertEqual(len(set(map(id, product('abc', 'def')))), 1)
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -16,6 +16,10 @@
 Library
 -------
 
+- Issue #23363: Fix possible overflow in itertools.permutations.
+
+- Issue #23364: Fix possible overflow in itertools.product.
+
 - Issue #23369: Fixed possible integer overflow in
   _json.encode_basestring_ascii.
 
diff --git a/Modules/itertoolsmodule.c b/Modules/itertoolsmodule.c
--- a/Modules/itertoolsmodule.c
+++ b/Modules/itertoolsmodule.c
@@ -1998,8 +1998,17 @@
         }
     }
 
-    assert(PyTuple_Check(args));
-    nargs = (repeat == 0) ? 0 : PyTuple_GET_SIZE(args);
+    assert(PyTuple_CheckExact(args));
+    if (repeat == 0) {
+        nargs = 0;
+    } else {
+        nargs = PyTuple_GET_SIZE(args);
+        if (repeat > PY_SSIZE_T_MAX/sizeof(Py_ssize_t) ||
+            nargs > PY_SSIZE_T_MAX/(repeat * sizeof(Py_ssize_t))) {
+            PyErr_SetString(PyExc_OverflowError, "repeat argument too large");
+            return NULL;
+        }
+    }
     npools = nargs * repeat;
 
     indices = PyMem_Malloc(npools * sizeof(Py_ssize_t));
@@ -2992,6 +3001,11 @@
         goto error;
     }
 
+    if (n > PY_SSIZE_T_MAX/sizeof(Py_ssize_t) ||
+        r > PY_SSIZE_T_MAX/sizeof(Py_ssize_t)) {
+        PyErr_SetString(PyExc_OverflowError, "parameters too large");
+        goto error;
+    }
     indices = PyMem_Malloc(n * sizeof(Py_ssize_t));
     cycles = PyMem_Malloc(r * sizeof(Py_ssize_t));
     if (indices == NULL || cycles == NULL) {

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Mon Feb  2 03:39:20 2015
From: python-checkins at python.org (benjamin.peterson)
Date: Mon, 02 Feb 2015 02:39:20 +0000
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4zIC0+IDMuNCk6?=
 =?utf-8?b?IG1lcmdlIDMuMyAoIzIzMzY0LCAjMjMzNjMp?=
Message-ID: <20150202023854.34408.32956@psf.io>

https://hg.python.org/cpython/rev/9ae055c3db32
changeset:   94457:9ae055c3db32
branch:      3.4
parent:      94453:2e7a02e4cf2c
parent:      94456:7133582b6769
user:        Benjamin Peterson <benjamin at python.org>
date:        Sun Feb 01 21:35:34 2015 -0500
summary:
  merge 3.3 (#23364, #23363)

files:
  Lib/test/test_itertools.py |  12 ++++++++++++
  Misc/NEWS                  |   4 ++++
  Modules/itertoolsmodule.c  |  18 ++++++++++++++++--
  3 files changed, 32 insertions(+), 2 deletions(-)


diff --git a/Lib/test/test_itertools.py b/Lib/test/test_itertools.py
--- a/Lib/test/test_itertools.py
+++ b/Lib/test/test_itertools.py
@@ -426,6 +426,13 @@
                 for proto in range(pickle.HIGHEST_PROTOCOL + 1):
                     self.pickletest(proto, permutations(values, r))     # test pickling
 
+    @support.bigaddrspacetest
+    def test_permutations_overflow(self):
+        with self.assertRaises(OverflowError):
+            permutations("A", 2**30)
+        with self.assertRaises(OverflowError):
+            permutations("A", 2, 2**30)
+
     @support.impl_detail("tuple reuse is specific to CPython")
     def test_permutations_tuple_reuse(self):
         self.assertEqual(len(set(map(id, permutations('abcde', 3)))), 1)
@@ -955,6 +962,11 @@
             args = map(iter, args)
             self.assertEqual(len(list(product(*args))), expected_len)
 
+    @support.bigaddrspacetest
+    def test_product_overflow(self):
+        with self.assertRaises(OverflowError):
+            product(["a"]*(2**16), repeat=2**16)
+
     @support.impl_detail("tuple reuse is specific to CPython")
     def test_product_tuple_reuse(self):
         self.assertEqual(len(set(map(id, product('abc', 'def')))), 1)
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -50,6 +50,10 @@
 Library
 -------
 
+- Issue #23363: Fix possible overflow in itertools.permutations.
+
+- Issue #23364: Fix possible overflow in itertools.product.
+
 - Issue #23366: Fixed possible integer overflow in itertools.combinations.
 
 - Issue #23366: Fixed possible integer overflow in itertools.combinations.
diff --git a/Modules/itertoolsmodule.c b/Modules/itertoolsmodule.c
--- a/Modules/itertoolsmodule.c
+++ b/Modules/itertoolsmodule.c
@@ -2017,8 +2017,17 @@
         }
     }
 
-    assert(PyTuple_Check(args));
-    nargs = (repeat == 0) ? 0 : PyTuple_GET_SIZE(args);
+    assert(PyTuple_CheckExact(args));
+    if (repeat == 0) {
+        nargs = 0;
+    } else {
+        nargs = PyTuple_GET_SIZE(args);
+        if (repeat > PY_SSIZE_T_MAX/sizeof(Py_ssize_t) ||
+            nargs > PY_SSIZE_T_MAX/(repeat * sizeof(Py_ssize_t))) {
+            PyErr_SetString(PyExc_OverflowError, "repeat argument too large");
+            return NULL;
+        }
+    }
     npools = nargs * repeat;
 
     indices = PyMem_Malloc(npools * sizeof(Py_ssize_t));
@@ -3049,6 +3058,11 @@
         goto error;
     }
 
+    if (n > PY_SSIZE_T_MAX/sizeof(Py_ssize_t) ||
+        r > PY_SSIZE_T_MAX/sizeof(Py_ssize_t)) {
+        PyErr_SetString(PyExc_OverflowError, "parameters too large");
+        goto error;
+    }
     indices = PyMem_Malloc(n * sizeof(Py_ssize_t));
     cycles = PyMem_Malloc(r * sizeof(Py_ssize_t));
     if (indices == NULL || cycles == NULL) {

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Mon Feb  2 03:39:20 2015
From: python-checkins at python.org (benjamin.peterson)
Date: Mon, 02 Feb 2015 02:39:20 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_check_for_over?=
 =?utf-8?q?flows_in_permutations=28=29_and_product=28=29_=28closes_=232336?=
 =?utf-8?q?3=2C_closes?=
Message-ID: <20150202023854.96098.35857@psf.io>

https://hg.python.org/cpython/rev/acc2c3479f2e
changeset:   94459:acc2c3479f2e
branch:      2.7
parent:      94455:366018a91457
user:        Benjamin Peterson <benjamin at python.org>
date:        Sun Feb 01 21:34:07 2015 -0500
summary:
  check for overflows in permutations() and product() (closes #23363, closes #23364)

files:
  Lib/test/test_itertools.py |  12 ++++++++++++
  Misc/NEWS                  |   4 ++++
  Modules/itertoolsmodule.c  |  18 ++++++++++++++++--
  3 files changed, 32 insertions(+), 2 deletions(-)


diff --git a/Lib/test/test_itertools.py b/Lib/test/test_itertools.py
--- a/Lib/test/test_itertools.py
+++ b/Lib/test/test_itertools.py
@@ -284,6 +284,13 @@
                     self.assertEqual(result, list(permutations(values, None))) # test r as None
                     self.assertEqual(result, list(permutations(values)))       # test default r
 
+    @test_support.bigaddrspacetest
+    def test_permutations_overflow(self):
+        with self.assertRaises(OverflowError):
+            permutations("A", 2**30)
+        with self.assertRaises(OverflowError):
+            permutations("A", 2, 2**30)
+
     @test_support.impl_detail("tuple reuse is specific to CPython")
     def test_permutations_tuple_reuse(self):
         self.assertEqual(len(set(map(id, permutations('abcde', 3)))), 1)
@@ -702,6 +709,11 @@
             args = map(iter, args)
             self.assertEqual(len(list(product(*args))), expected_len)
 
+    @test_support.bigaddrspacetest
+    def test_product_overflow(self):
+        with self.assertRaises(OverflowError):
+            product(["a"]*(2**16), repeat=2**16)
+
     @test_support.impl_detail("tuple reuse is specific to CPython")
     def test_product_tuple_reuse(self):
         self.assertEqual(len(set(map(id, product('abc', 'def')))), 1)
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -18,6 +18,10 @@
 Library
 -------
 
+- Issue #23363: Fix possible overflow in itertools.permutations.
+
+- Issue #23364: Fix possible overflow in itertools.product.
+
 - Issue #23365: Fixed possible integer overflow in
   itertools.combinations_with_replacement.
 
diff --git a/Modules/itertoolsmodule.c b/Modules/itertoolsmodule.c
--- a/Modules/itertoolsmodule.c
+++ b/Modules/itertoolsmodule.c
@@ -1842,8 +1842,17 @@
         }
     }
 
-    assert(PyTuple_Check(args));
-    nargs = (repeat == 0) ? 0 : PyTuple_GET_SIZE(args);
+    assert(PyTuple_CheckExact(args));
+    if (repeat == 0) {
+        nargs = 0;
+    } else {
+        nargs = PyTuple_GET_SIZE(args);
+        if (repeat > PY_SSIZE_T_MAX/sizeof(Py_ssize_t) ||
+            nargs > PY_SSIZE_T_MAX/(repeat * sizeof(Py_ssize_t))) {
+            PyErr_SetString(PyExc_OverflowError, "repeat argument too large");
+            return NULL;
+        }
+    }
     npools = nargs * repeat;
 
     indices = PyMem_Malloc(npools * sizeof(Py_ssize_t));
@@ -2603,6 +2612,11 @@
         goto error;
     }
 
+    if (n > PY_SSIZE_T_MAX/sizeof(Py_ssize_t) ||
+        r > PY_SSIZE_T_MAX/sizeof(Py_ssize_t)) {
+        PyErr_SetString(PyExc_OverflowError, "parameters too large");
+        goto error;
+    }
     indices = PyMem_Malloc(n * sizeof(Py_ssize_t));
     cycles = PyMem_Malloc(r * sizeof(Py_ssize_t));
     if (indices == NULL || cycles == NULL) {

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Mon Feb  2 03:39:20 2015
From: python-checkins at python.org (benjamin.peterson)
Date: Mon, 02 Feb 2015 02:39:20 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?b?KTogbWVyZ2UgMy40ICgjMjMzNjQsICMyMzM2Myk=?=
Message-ID: <20150202023854.96072.86609@psf.io>

https://hg.python.org/cpython/rev/31dc5a40d2ab
changeset:   94458:31dc5a40d2ab
parent:      94454:4d875a690c01
parent:      94457:9ae055c3db32
user:        Benjamin Peterson <benjamin at python.org>
date:        Sun Feb 01 21:36:01 2015 -0500
summary:
  merge 3.4 (#23364, #23363)

files:
  Lib/test/test_itertools.py |  12 ++++++++++++
  Misc/NEWS                  |   4 ++++
  Modules/itertoolsmodule.c  |  18 ++++++++++++++++--
  3 files changed, 32 insertions(+), 2 deletions(-)


diff --git a/Lib/test/test_itertools.py b/Lib/test/test_itertools.py
--- a/Lib/test/test_itertools.py
+++ b/Lib/test/test_itertools.py
@@ -426,6 +426,13 @@
                 for proto in range(pickle.HIGHEST_PROTOCOL + 1):
                     self.pickletest(proto, permutations(values, r))     # test pickling
 
+    @support.bigaddrspacetest
+    def test_permutations_overflow(self):
+        with self.assertRaises(OverflowError):
+            permutations("A", 2**30)
+        with self.assertRaises(OverflowError):
+            permutations("A", 2, 2**30)
+
     @support.impl_detail("tuple reuse is specific to CPython")
     def test_permutations_tuple_reuse(self):
         self.assertEqual(len(set(map(id, permutations('abcde', 3)))), 1)
@@ -955,6 +962,11 @@
             args = map(iter, args)
             self.assertEqual(len(list(product(*args))), expected_len)
 
+    @support.bigaddrspacetest
+    def test_product_overflow(self):
+        with self.assertRaises(OverflowError):
+            product(["a"]*(2**16), repeat=2**16)
+
     @support.impl_detail("tuple reuse is specific to CPython")
     def test_product_tuple_reuse(self):
         self.assertEqual(len(set(map(id, product('abc', 'def')))), 1)
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -229,6 +229,10 @@
 - Issue #23326: Removed __ne__ implementations.  Since fixing default __ne__
   implementation in issue #21408 they are redundant.
 
+- Issue #23363: Fix possible overflow in itertools.permutations.
+
+- Issue #23364: Fix possible overflow in itertools.product.
+
 - Issue #23366: Fixed possible integer overflow in itertools.combinations.
 
 - Issue #23366: Fixed possible integer overflow in itertools.combinations.
diff --git a/Modules/itertoolsmodule.c b/Modules/itertoolsmodule.c
--- a/Modules/itertoolsmodule.c
+++ b/Modules/itertoolsmodule.c
@@ -2017,8 +2017,17 @@
         }
     }
 
-    assert(PyTuple_Check(args));
-    nargs = (repeat == 0) ? 0 : PyTuple_GET_SIZE(args);
+    assert(PyTuple_CheckExact(args));
+    if (repeat == 0) {
+        nargs = 0;
+    } else {
+        nargs = PyTuple_GET_SIZE(args);
+        if (repeat > PY_SSIZE_T_MAX/sizeof(Py_ssize_t) ||
+            nargs > PY_SSIZE_T_MAX/(repeat * sizeof(Py_ssize_t))) {
+            PyErr_SetString(PyExc_OverflowError, "repeat argument too large");
+            return NULL;
+        }
+    }
     npools = nargs * repeat;
 
     indices = PyMem_Malloc(npools * sizeof(Py_ssize_t));
@@ -3049,6 +3058,11 @@
         goto error;
     }
 
+    if (n > PY_SSIZE_T_MAX/sizeof(Py_ssize_t) ||
+        r > PY_SSIZE_T_MAX/sizeof(Py_ssize_t)) {
+        PyErr_SetString(PyExc_OverflowError, "parameters too large");
+        goto error;
+    }
     indices = PyMem_Malloc(n * sizeof(Py_ssize_t));
     cycles = PyMem_Malloc(r * sizeof(Py_ssize_t));
     if (indices == NULL || cycles == NULL) {

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Mon Feb  2 07:53:47 2015
From: python-checkins at python.org (raymond.hettinger)
Date: Mon, 02 Feb 2015 06:53:47 +0000
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Optimization_guides_sugges?=
 =?utf-8?q?t_copying_memory_in_an_ascending_direction_when?=
Message-ID: <20150202065347.96080.89203@psf.io>

https://hg.python.org/cpython/rev/673191aa1724
changeset:   94460:673191aa1724
parent:      94458:31dc5a40d2ab
user:        Raymond Hettinger <python at rcn.com>
date:        Sun Feb 01 22:53:41 2015 -0800
summary:
  Optimization guides suggest copying memory in an ascending direction when possible.

files:
  Modules/_collectionsmodule.c |  6 +++---
  1 files changed, 3 insertions(+), 3 deletions(-)


diff --git a/Modules/_collectionsmodule.c b/Modules/_collectionsmodule.c
--- a/Modules/_collectionsmodule.c
+++ b/Modules/_collectionsmodule.c
@@ -534,13 +534,13 @@
             if (m > leftindex)
                 m = leftindex;
             assert (m > 0 && m <= len);
-            src = &rightblock->data[rightindex];
-            dest = &leftblock->data[leftindex - 1];
             rightindex -= m;
             leftindex -= m;
+            src = &rightblock->data[rightindex + 1];
+            dest = &leftblock->data[leftindex];
             n -= m;
             do {
-                *(dest--) = *(src--);
+                *(dest++) = *(src++);
             } while (--m);
         }
         if (rightindex == -1) {

-- 
Repository URL: https://hg.python.org/cpython

From solipsis at pitrou.net  Mon Feb  2 09:04:49 2015
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Mon, 02 Feb 2015 09:04:49 +0100
Subject: [Python-checkins] Daily reference leaks (4d875a690c01): sum=1
Message-ID: <E1YIC01-0005NO-KQ@ssl.pitrou.net>

results for 4d875a690c01 on branch "default"
--------------------------------------------

test_collections leaked [-2, 0, 0] references, sum=-2
test_functools leaked [0, 0, 3] memory blocks, sum=3


Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflog2SFUL1', '-x']

From python-checkins at python.org  Mon Feb  2 14:33:54 2015
From: python-checkins at python.org (berker.peksag)
Date: Mon, 02 Feb 2015 13:33:54 +0000
Subject: [Python-checkins] =?utf-8?q?peps=3A_PEP_448=3A_Update_from_Joshua?=
	=?utf-8?q?_Landau=2E?=
Message-ID: <20150202133148.96076.87734@psf.io>

https://hg.python.org/peps/rev/04cfd59d12fb
changeset:   5687:04cfd59d12fb
user:        Berker Peksag <berker.peksag at gmail.com>
date:        Mon Feb 02 15:31:33 2015 +0200
summary:
  PEP 448: Update from Joshua Landau.

files:
  pep-0448.txt |  41 ++++++++++++++++++++++-----------------
  1 files changed, 23 insertions(+), 18 deletions(-)


diff --git a/pep-0448.txt b/pep-0448.txt
--- a/pep-0448.txt
+++ b/pep-0448.txt
@@ -16,11 +16,11 @@
 ========
 
 This PEP proposes extended usages of the ``*`` iterable unpacking
-operator and ``**`` dictionary unpacking operator
+operator and ``**`` dictionary unpacking operators
 to allow unpacking in more positions, an arbitrary number of
-times, and in additional circumstances.  Specifically
-in function calls, in comprehensions and generator expressions,
-and in displays.
+times, and in additional circumstances.  Specifically,
+in function calls, in comprehensions and generator expressions, and
+in displays.
 
 Function calls are proposed to support an arbitrary number of
 unpackings rather than just one::
@@ -42,8 +42,7 @@
     >>> {'x': 1, **{'y': 2}}
     {'x': 1, 'y': 2}
 
-In sets and dictionaries, this provides a concise overriding
-notation::
+In dictionaries, later values will always override earlier ones::
 
     >>> {'x': 1, **{'x': 2}}
     {'x': 2}
@@ -124,7 +123,9 @@
 
 The addition of unpacking to comprehensions is a logical extension.
 It's usage will primarily be a neat replacement for ``[i for j in
-2D_list for i in j]``, as the more readable ``[*l for l in 2D_list]``.
+list_of_lists for i in j]``, as the more readable
+``[*l for l in list_of_lists]``. The iterable version,
+``(*l for l in list_of_lists)``, replaces ``itertools.chain.from_iterable``.
 Other uses are possible, but expected to occur rarely.
 
 
@@ -143,7 +144,7 @@
 Currently, if an argument is given multiple times ? such as a
 positional argument given both positionally and by keyword ? a
 ``TypeError`` is raised.  This remains true for duplicate arguments
-provided through multiple keyword argument unpackings,
+provided through multiple ``**`` unpackings,
 e.g. ``f(**{'x': 2}, **{'x': 3})``.
 
 A function looks like this::
@@ -176,22 +177,26 @@
 
     {**locals(), "override": None}
 
-However, ``f(*x for x in it)`` and ``f(**x for x in it)`` continue
-to raise SyntaxError.
+Unbracketed comprehensions in function calls, such as ``f(x for x in it)``,
+are already valid. These could be extended to::
+
+    f(*x for x in it) == f((*x for x in it))
+    f(**x for x in it) == f({**x for x in it})
+
+However, this is likely to be confusing and is not included in this
+PEP. These will throw ``SyntaxError`` and comprehensions with explicit
+brackets should be used instead.
 
 
 Disadvantages
 =============
 
-The allowable orders for arguments in a function call is more
-complicated than before.
-The simplest explanation for the rules may be "positional arguments
-come first and keyword arguments follow, but iterable unpackings are
-allowed after keyword arguments" or "iterable arguments precede
-all keyword arguments and keyword argument unpackings, and iterable
-argument unpackings precede all keyword argument unpackings".
+The allowable orders for arguments in a function call are more
+complicated than before. The simplest explanation for the rules
+may be "positional arguments precede keyword arguments and ``**``
+unpacking; ``*`` unpacking precedes ``**`` unpacking".
 
-While ``*elements, = iterable`` causes ``elements`` to be a list,
+Whilst ``*elements, = iterable`` causes ``elements`` to be a list,
 ``elements = *iterable,`` causes ``elements`` to be a tuple.  The
 reason for this may confuse people unfamiliar with the construct.
 

-- 
Repository URL: https://hg.python.org/peps

From python-checkins at python.org  Mon Feb  2 14:37:41 2015
From: python-checkins at python.org (berker.peksag)
Date: Mon, 02 Feb 2015 13:37:41 +0000
Subject: [Python-checkins] =?utf-8?q?peps=3A_Update_PEP_editors_list=2E?=
Message-ID: <20150202133705.96098.93407@psf.io>

https://hg.python.org/peps/rev/803169da11e3
changeset:   5688:803169da11e3
user:        Berker Peksag <berker.peksag at gmail.com>
date:        Mon Feb 02 15:37:11 2015 +0200
summary:
  Update PEP editors list.

files:
  pep-0001.txt |  2 ++
  1 files changed, 2 insertions(+), 0 deletions(-)


diff --git a/pep-0001.txt b/pep-0001.txt
--- a/pep-0001.txt
+++ b/pep-0001.txt
@@ -82,11 +82,13 @@
 changing their status).  See `PEP Editor Responsibilities & Workflow`_ for
 details.  The current editors are:
 
+* Chris Angelico
 * Anthony Baxter
 * Georg Brandl
 * Brett Cannon
 * David Goodger
 * Jesse Noller
+* Berker Peksag
 * Guido van Rossum
 * Barry Warsaw
 

-- 
Repository URL: https://hg.python.org/peps

From python-checkins at python.org  Mon Feb  2 16:53:42 2015
From: python-checkins at python.org (benjamin.peterson)
Date: Mon, 02 Feb 2015 15:53:42 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_revert_lineno_?=
 =?utf-8?q?and_col=5Foffset_changes_from_=2316795_=28closes_=2321295=29?=
Message-ID: <20150202155303.96090.24843@psf.io>

https://hg.python.org/cpython/rev/7d1c32ddc432
changeset:   94461:7d1c32ddc432
branch:      3.4
parent:      94457:9ae055c3db32
user:        Benjamin Peterson <benjamin at python.org>
date:        Mon Feb 02 10:51:20 2015 -0500
summary:
  revert lineno and col_offset changes from #16795 (closes #21295)

files:
  Lib/test/test_ast.py |    45 +-
  Misc/NEWS            |     3 +
  Python/ast.c         |    35 +-
  Python/importlib.h   |  7812 +++++++++++++++---------------
  4 files changed, 3936 insertions(+), 3959 deletions(-)


diff --git a/Lib/test/test_ast.py b/Lib/test/test_ast.py
--- a/Lib/test/test_ast.py
+++ b/Lib/test/test_ast.py
@@ -180,36 +180,20 @@
 
 class AST_Tests(unittest.TestCase):
 
-    def _assertTrueorder(self, ast_node, parent_pos, reverse_check = False):
-        def should_reverse_check(parent, child):
-            # In some situations, the children of nodes occur before
-            # their parents, for example in a.b.c, a occurs before b
-            # but a is a child of b.
-            if isinstance(parent, ast.Call):
-                if parent.func == child:
-                    return True
-            if isinstance(parent, (ast.Attribute, ast.Subscript)):
-                return True
-            return False
-
+    def _assertTrueorder(self, ast_node, parent_pos):
         if not isinstance(ast_node, ast.AST) or ast_node._fields is None:
             return
         if isinstance(ast_node, (ast.expr, ast.stmt, ast.excepthandler)):
             node_pos = (ast_node.lineno, ast_node.col_offset)
-            if reverse_check:
-                self.assertTrue(node_pos <= parent_pos)
-            else:
-                self.assertTrue(node_pos >= parent_pos)
+            self.assertTrue(node_pos >= parent_pos)
             parent_pos = (ast_node.lineno, ast_node.col_offset)
         for name in ast_node._fields:
             value = getattr(ast_node, name)
             if isinstance(value, list):
                 for child in value:
-                    self._assertTrueorder(child, parent_pos,
-                                         should_reverse_check(ast_node, child))
+                    self._assertTrueorder(child, parent_pos)
             elif value is not None:
-                self._assertTrueorder(value, parent_pos,
-                                      should_reverse_check(ast_node, value))
+                self._assertTrueorder(value, parent_pos)
 
     def test_AST_objects(self):
         x = ast.AST()
@@ -278,9 +262,8 @@
 
     def test_arguments(self):
         x = ast.arguments()
-        self.assertEqual(x._fields, ('args', 'vararg',
-                                      'kwonlyargs', 'kw_defaults',
-                                      'kwarg', 'defaults'))
+        self.assertEqual(x._fields, ('args', 'vararg', 'kwonlyargs',
+                                      'kw_defaults', 'kwarg', 'defaults'))
 
         with self.assertRaises(AttributeError):
             x.vararg
@@ -455,7 +438,7 @@
             "lineno=1, col_offset=0), args=[Name(id='eggs', ctx=Load(), "
             "lineno=1, col_offset=5), Str(s='and cheese', lineno=1, "
             "col_offset=11)], keywords=[], starargs=None, kwargs=None, "
-            "lineno=1, col_offset=4), lineno=1, col_offset=0)])"
+            "lineno=1, col_offset=0), lineno=1, col_offset=0)])"
         )
 
     def test_copy_location(self):
@@ -476,7 +459,7 @@
             "Module(body=[Expr(value=Call(func=Name(id='write', ctx=Load(), "
             "lineno=1, col_offset=0), args=[Str(s='spam', lineno=1, "
             "col_offset=6)], keywords=[], starargs=None, kwargs=None, "
-            "lineno=1, col_offset=5), lineno=1, col_offset=0), "
+            "lineno=1, col_offset=0), lineno=1, col_offset=0), "
             "Expr(value=Call(func=Name(id='spam', ctx=Load(), lineno=1, "
             "col_offset=0), args=[Str(s='eggs', lineno=1, col_offset=0)], "
             "keywords=[], starargs=None, kwargs=None, lineno=1, "
@@ -973,7 +956,7 @@
 ('Module', [('FunctionDef', (1, 0), 'f', ('arguments', [('arg', (1, 6), 'a', None)], None, [], [], None, [('Num', (1, 8), 0)]), [('Pass', (1, 12))], [], None)]),
 ('Module', [('FunctionDef', (1, 0), 'f', ('arguments', [], ('arg', (1, 7), 'args', None), [], [], None, []), [('Pass', (1, 14))], [], None)]),
 ('Module', [('FunctionDef', (1, 0), 'f', ('arguments', [], None, [], [], ('arg', (1, 8), 'kwargs', None), []), [('Pass', (1, 17))], [], None)]),
-    ('Module', [('FunctionDef', (1, 0), 'f', ('arguments', [('arg', (1, 6), 'a', None), ('arg', (1, 9), 'b', None), ('arg', (1, 14), 'c', None), ('arg', (1, 22), 'd', None), ('arg', (1, 28), 'e', None)], ('arg', (1, 35), 'args', None), [('arg', (1, 41), 'f', None)], [('Num', (1, 43), 42)], ('arg', (1, 49), 'kwargs', None), [('Num', (1, 11), 1), ('NameConstant', (1, 16), None), ('List', (1, 24), [], ('Load',)), ('Dict', (1, 30), [], [])]), [('Pass', (1, 58))], [], None)]),
+('Module', [('FunctionDef', (1, 0), 'f', ('arguments', [('arg', (1, 6), 'a', None), ('arg', (1, 9), 'b', None), ('arg', (1, 14), 'c', None), ('arg', (1, 22), 'd', None), ('arg', (1, 28), 'e', None)], ('arg', (1, 35), 'args', None), [('arg', (1, 41), 'f', None)], [('Num', (1, 43), 42)], ('arg', (1, 49), 'kwargs', None), [('Num', (1, 11), 1), ('NameConstant', (1, 16), None), ('List', (1, 24), [], ('Load',)), ('Dict', (1, 30), [], [])]), [('Pass', (1, 58))], [], None)]),
 ('Module', [('ClassDef', (1, 0), 'C', [], [], None, None, [('Pass', (1, 8))], [])]),
 ('Module', [('ClassDef', (1, 0), 'C', [('Name', (1, 8), 'object', ('Load',))], [], None, None, [('Pass', (1, 17))], [])]),
 ('Module', [('FunctionDef', (1, 0), 'f', ('arguments', [], None, [], [], None, []), [('Return', (1, 8), ('Num', (1, 15), 1))], [], None)]),
@@ -985,7 +968,7 @@
 ('Module', [('If', (1, 0), ('Name', (1, 3), 'v', ('Load',)), [('Pass', (1, 5))], [])]),
 ('Module', [('With', (1, 0), [('withitem', ('Name', (1, 5), 'x', ('Load',)), ('Name', (1, 10), 'y', ('Store',)))], [('Pass', (1, 13))])]),
 ('Module', [('With', (1, 0), [('withitem', ('Name', (1, 5), 'x', ('Load',)), ('Name', (1, 10), 'y', ('Store',))), ('withitem', ('Name', (1, 13), 'z', ('Load',)), ('Name', (1, 18), 'q', ('Store',)))], [('Pass', (1, 21))])]),
-('Module', [('Raise', (1, 0), ('Call', (1, 15), ('Name', (1, 6), 'Exception', ('Load',)), [('Str', (1, 16), 'string')], [], None, None), None)]),
+('Module', [('Raise', (1, 0), ('Call', (1, 6), ('Name', (1, 6), 'Exception', ('Load',)), [('Str', (1, 16), 'string')], [], None, None), None)]),
 ('Module', [('Try', (1, 0), [('Pass', (2, 2))], [('ExceptHandler', (3, 0), ('Name', (3, 7), 'Exception', ('Load',)), None, [('Pass', (4, 2))])], [], [])]),
 ('Module', [('Try', (1, 0), [('Pass', (2, 2))], [], [], [('Pass', (4, 2))])]),
 ('Module', [('Assert', (1, 0), ('Name', (1, 7), 'v', ('Load',)), None)]),
@@ -1022,17 +1005,17 @@
 ('Expression', ('ListComp', (1, 1), ('Name', (1, 1), 'a', ('Load',)), [('comprehension', ('Name', (1, 7), 'b', ('Store',)), ('Name', (1, 12), 'c', ('Load',)), [('Name', (1, 17), 'd', ('Load',))])])),
 ('Expression', ('GeneratorExp', (1, 1), ('Name', (1, 1), 'a', ('Load',)), [('comprehension', ('Name', (1, 7), 'b', ('Store',)), ('Name', (1, 12), 'c', ('Load',)), [('Name', (1, 17), 'd', ('Load',))])])),
 ('Expression', ('Compare', (1, 0), ('Num', (1, 0), 1), [('Lt',), ('Lt',)], [('Num', (1, 4), 2), ('Num', (1, 8), 3)])),
-('Expression', ('Call', (1, 1), ('Name', (1, 0), 'f', ('Load',)), [('Num', (1, 2), 1), ('Num', (1, 4), 2)], [('keyword', 'c', ('Num', (1, 8), 3))], ('Name', (1, 11), 'd', ('Load',)), ('Name', (1, 15), 'e', ('Load',)))),
+('Expression', ('Call', (1, 0), ('Name', (1, 0), 'f', ('Load',)), [('Num', (1, 2), 1), ('Num', (1, 4), 2)], [('keyword', 'c', ('Num', (1, 8), 3))], ('Name', (1, 11), 'd', ('Load',)), ('Name', (1, 15), 'e', ('Load',)))),
 ('Expression', ('Num', (1, 0), 10)),
 ('Expression', ('Str', (1, 0), 'string')),
-('Expression', ('Attribute', (1, 2), ('Name', (1, 0), 'a', ('Load',)), 'b', ('Load',))),
-('Expression', ('Subscript', (1, 2), ('Name', (1, 0), 'a', ('Load',)), ('Slice', ('Name', (1, 2), 'b', ('Load',)), ('Name', (1, 4), 'c', ('Load',)), None), ('Load',))),
+('Expression', ('Attribute', (1, 0), ('Name', (1, 0), 'a', ('Load',)), 'b', ('Load',))),
+('Expression', ('Subscript', (1, 0), ('Name', (1, 0), 'a', ('Load',)), ('Slice', ('Name', (1, 2), 'b', ('Load',)), ('Name', (1, 4), 'c', ('Load',)), None), ('Load',))),
 ('Expression', ('Name', (1, 0), 'v', ('Load',))),
 ('Expression', ('List', (1, 0), [('Num', (1, 1), 1), ('Num', (1, 3), 2), ('Num', (1, 5), 3)], ('Load',))),
 ('Expression', ('List', (1, 0), [], ('Load',))),
 ('Expression', ('Tuple', (1, 0), [('Num', (1, 0), 1), ('Num', (1, 2), 2), ('Num', (1, 4), 3)], ('Load',))),
 ('Expression', ('Tuple', (1, 1), [('Num', (1, 1), 1), ('Num', (1, 3), 2), ('Num', (1, 5), 3)], ('Load',))),
 ('Expression', ('Tuple', (1, 0), [], ('Load',))),
-('Expression', ('Call', (1, 7), ('Attribute', (1, 6), ('Attribute', (1, 4), ('Attribute', (1, 2), ('Name', (1, 0), 'a', ('Load',)), 'b', ('Load',)), 'c', ('Load',)), 'd', ('Load',)), [('Subscript', (1, 12), ('Attribute', (1, 10), ('Name', (1, 8), 'a', ('Load',)), 'b', ('Load',)), ('Slice', ('Num', (1, 12), 1), ('Num', (1, 14), 2), None), ('Load',))], [], None, None)),
+('Expression', ('Call', (1, 0), ('Attribute', (1, 0), ('Attribute', (1, 0), ('Attribute', (1, 0), ('Name', (1, 0), 'a', ('Load',)), 'b', ('Load',)), 'c', ('Load',)), 'd', ('Load',)), [('Subscript', (1, 8), ('Attribute', (1, 8), ('Name', (1, 8), 'a', ('Load',)), 'b', ('Load',)), ('Slice', ('Num', (1, 12), 1), ('Num', (1, 14), 2), None), ('Load',))], [], None, None)),
 ]
 main()
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -11,6 +11,9 @@
 Core and Builtins
 -----------------
 
+- Issue #21295: Revert some changes (issue #16795) to AST line numbers and
+  column offsets that constituted a regression.
+
 - Issue #21408: The default __ne__() now returns NotImplemented if __eq__()
   returned NotImplemented.  Original patch by Martin Panter.
 
diff --git a/Python/ast.c b/Python/ast.c
--- a/Python/ast.c
+++ b/Python/ast.c
@@ -1123,7 +1123,7 @@
     identifier name;
     expr_ty annotation = NULL;
     node *ch;
-    arg_ty tmp;
+    arg_ty ret;
 
     assert(TYPE(n) == tfpdef || TYPE(n) == vfpdef);
     ch = CHILD(n, 0);
@@ -1139,13 +1139,12 @@
             return NULL;
     }
 
-    tmp = arg(name, annotation, c->c_arena);
-    if (!tmp)
+    ret = arg(name, annotation, c->c_arena);
+    if (!ret)
         return NULL;
-
-    tmp->lineno = LINENO(n);
-    tmp->col_offset = n->n_col_offset;
-    return tmp;
+    ret->lineno = LINENO(n);
+    ret->col_offset = n->n_col_offset;
+    return ret;
 }
 
 /* returns -1 if failed to handle keyword only arguments
@@ -2103,22 +2102,15 @@
         if (NCH(n) == 2)
             return Call(left_expr, NULL, NULL, NULL, NULL, LINENO(n),
                         n->n_col_offset, c->c_arena);
-        else {
-            expr_ty tmp = ast_for_call(c, CHILD(n, 1), left_expr);
-            if (!tmp)
-                return NULL;
-
-            tmp->lineno = LINENO(n);
-            tmp->col_offset = n->n_col_offset;
-            return tmp;
-        }
+        else
+            return ast_for_call(c, CHILD(n, 1), left_expr);
     }
-    else if (TYPE(CHILD(n, 0)) == DOT ) {
+    else if (TYPE(CHILD(n, 0)) == DOT) {
         PyObject *attr_id = NEW_IDENTIFIER(CHILD(n, 1));
         if (!attr_id)
             return NULL;
         return Attribute(left_expr, attr_id, Load,
-                         LINENO(CHILD(n, 1)), CHILD(n, 1)->n_col_offset, c->c_arena);
+                         LINENO(n), n->n_col_offset, c->c_arena);
     }
     else {
         REQ(CHILD(n, 0), LSQB);
@@ -2219,16 +2211,15 @@
         tmp = ast_for_trailer(c, ch, e);
         if (!tmp)
             return NULL;
+        tmp->lineno = e->lineno;
+        tmp->col_offset = e->col_offset;
         e = tmp;
     }
     if (TYPE(CHILD(n, NCH(n) - 1)) == factor) {
         expr_ty f = ast_for_expr(c, CHILD(n, NCH(n) - 1));
         if (!f)
             return NULL;
-        tmp = BinOp(e, Pow, f, LINENO(n), n->n_col_offset, c->c_arena);
-        if (!tmp)
-            return NULL;
-        e = tmp;
+        e = BinOp(e, Pow, f, LINENO(n), n->n_col_offset, c->c_arena);
     }
     return e;
 }
diff --git a/Python/importlib.h b/Python/importlib.h
--- a/Python/importlib.h
+++ b/Python/importlib.h
[stripped]

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Mon Feb  2 16:53:41 2015
From: python-checkins at python.org (benjamin.peterson)
Date: Mon, 02 Feb 2015 15:53:41 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?b?KTogbWVyZ2UgMy40ICgjMjEyOTUp?=
Message-ID: <20150202155303.34398.88559@psf.io>

https://hg.python.org/cpython/rev/8ab6b404248c
changeset:   94462:8ab6b404248c
parent:      94460:673191aa1724
parent:      94461:7d1c32ddc432
user:        Benjamin Peterson <benjamin at python.org>
date:        Mon Feb 02 10:52:56 2015 -0500
summary:
  merge 3.4 (#21295)

files:
  Lib/test/test_ast.py |   45 +-
  Misc/NEWS            |    3 +
  Python/ast.c         |   35 +-
  Python/importlib.h   |  698 +++++++++++++++---------------
  4 files changed, 379 insertions(+), 402 deletions(-)


diff --git a/Lib/test/test_ast.py b/Lib/test/test_ast.py
--- a/Lib/test/test_ast.py
+++ b/Lib/test/test_ast.py
@@ -180,36 +180,20 @@
 
 class AST_Tests(unittest.TestCase):
 
-    def _assertTrueorder(self, ast_node, parent_pos, reverse_check = False):
-        def should_reverse_check(parent, child):
-            # In some situations, the children of nodes occur before
-            # their parents, for example in a.b.c, a occurs before b
-            # but a is a child of b.
-            if isinstance(parent, ast.Call):
-                if parent.func == child:
-                    return True
-            if isinstance(parent, (ast.Attribute, ast.Subscript)):
-                return True
-            return False
-
+    def _assertTrueorder(self, ast_node, parent_pos):
         if not isinstance(ast_node, ast.AST) or ast_node._fields is None:
             return
         if isinstance(ast_node, (ast.expr, ast.stmt, ast.excepthandler)):
             node_pos = (ast_node.lineno, ast_node.col_offset)
-            if reverse_check:
-                self.assertTrue(node_pos <= parent_pos)
-            else:
-                self.assertTrue(node_pos >= parent_pos)
+            self.assertTrue(node_pos >= parent_pos)
             parent_pos = (ast_node.lineno, ast_node.col_offset)
         for name in ast_node._fields:
             value = getattr(ast_node, name)
             if isinstance(value, list):
                 for child in value:
-                    self._assertTrueorder(child, parent_pos,
-                                         should_reverse_check(ast_node, child))
+                    self._assertTrueorder(child, parent_pos)
             elif value is not None:
-                self._assertTrueorder(value, parent_pos,
-                                      should_reverse_check(ast_node, value))
+                self._assertTrueorder(value, parent_pos)
 
     def test_AST_objects(self):
         x = ast.AST()
@@ -278,9 +262,8 @@
 
     def test_arguments(self):
         x = ast.arguments()
-        self.assertEqual(x._fields, ('args', 'vararg',
-                                      'kwonlyargs', 'kw_defaults',
-                                      'kwarg', 'defaults'))
+        self.assertEqual(x._fields, ('args', 'vararg', 'kwonlyargs',
+                                      'kw_defaults', 'kwarg', 'defaults'))
 
         with self.assertRaises(AttributeError):
             x.vararg
@@ -455,7 +438,7 @@
             "lineno=1, col_offset=0), args=[Name(id='eggs', ctx=Load(), "
             "lineno=1, col_offset=5), Str(s='and cheese', lineno=1, "
             "col_offset=11)], keywords=[], starargs=None, kwargs=None, "
-            "lineno=1, col_offset=4), lineno=1, col_offset=0)])"
+            "lineno=1, col_offset=0), lineno=1, col_offset=0)])"
         )
 
     def test_copy_location(self):
@@ -476,7 +459,7 @@
             "Module(body=[Expr(value=Call(func=Name(id='write', ctx=Load(), "
             "lineno=1, col_offset=0), args=[Str(s='spam', lineno=1, "
             "col_offset=6)], keywords=[], starargs=None, kwargs=None, "
-            "lineno=1, col_offset=5), lineno=1, col_offset=0), "
+            "lineno=1, col_offset=0), lineno=1, col_offset=0), "
             "Expr(value=Call(func=Name(id='spam', ctx=Load(), lineno=1, "
             "col_offset=0), args=[Str(s='eggs', lineno=1, col_offset=0)], "
             "keywords=[], starargs=None, kwargs=None, lineno=1, "
@@ -973,7 +956,7 @@
 ('Module', [('FunctionDef', (1, 0), 'f', ('arguments', [('arg', (1, 6), 'a', None)], None, [], [], None, [('Num', (1, 8), 0)]), [('Pass', (1, 12))], [], None)]),
 ('Module', [('FunctionDef', (1, 0), 'f', ('arguments', [], ('arg', (1, 7), 'args', None), [], [], None, []), [('Pass', (1, 14))], [], None)]),
 ('Module', [('FunctionDef', (1, 0), 'f', ('arguments', [], None, [], [], ('arg', (1, 8), 'kwargs', None), []), [('Pass', (1, 17))], [], None)]),
-    ('Module', [('FunctionDef', (1, 0), 'f', ('arguments', [('arg', (1, 6), 'a', None), ('arg', (1, 9), 'b', None), ('arg', (1, 14), 'c', None), ('arg', (1, 22), 'd', None), ('arg', (1, 28), 'e', None)], ('arg', (1, 35), 'args', None), [('arg', (1, 41), 'f', None)], [('Num', (1, 43), 42)], ('arg', (1, 49), 'kwargs', None), [('Num', (1, 11), 1), ('NameConstant', (1, 16), None), ('List', (1, 24), [], ('Load',)), ('Dict', (1, 30), [], [])]), [('Pass', (1, 58))], [], None)]),
+('Module', [('FunctionDef', (1, 0), 'f', ('arguments', [('arg', (1, 6), 'a', None), ('arg', (1, 9), 'b', None), ('arg', (1, 14), 'c', None), ('arg', (1, 22), 'd', None), ('arg', (1, 28), 'e', None)], ('arg', (1, 35), 'args', None), [('arg', (1, 41), 'f', None)], [('Num', (1, 43), 42)], ('arg', (1, 49), 'kwargs', None), [('Num', (1, 11), 1), ('NameConstant', (1, 16), None), ('List', (1, 24), [], ('Load',)), ('Dict', (1, 30), [], [])]), [('Pass', (1, 58))], [], None)]),
 ('Module', [('ClassDef', (1, 0), 'C', [], [], None, None, [('Pass', (1, 8))], [])]),
 ('Module', [('ClassDef', (1, 0), 'C', [('Name', (1, 8), 'object', ('Load',))], [], None, None, [('Pass', (1, 17))], [])]),
 ('Module', [('FunctionDef', (1, 0), 'f', ('arguments', [], None, [], [], None, []), [('Return', (1, 8), ('Num', (1, 15), 1))], [], None)]),
@@ -985,7 +968,7 @@
 ('Module', [('If', (1, 0), ('Name', (1, 3), 'v', ('Load',)), [('Pass', (1, 5))], [])]),
 ('Module', [('With', (1, 0), [('withitem', ('Name', (1, 5), 'x', ('Load',)), ('Name', (1, 10), 'y', ('Store',)))], [('Pass', (1, 13))])]),
 ('Module', [('With', (1, 0), [('withitem', ('Name', (1, 5), 'x', ('Load',)), ('Name', (1, 10), 'y', ('Store',))), ('withitem', ('Name', (1, 13), 'z', ('Load',)), ('Name', (1, 18), 'q', ('Store',)))], [('Pass', (1, 21))])]),
-('Module', [('Raise', (1, 0), ('Call', (1, 15), ('Name', (1, 6), 'Exception', ('Load',)), [('Str', (1, 16), 'string')], [], None, None), None)]),
+('Module', [('Raise', (1, 0), ('Call', (1, 6), ('Name', (1, 6), 'Exception', ('Load',)), [('Str', (1, 16), 'string')], [], None, None), None)]),
 ('Module', [('Try', (1, 0), [('Pass', (2, 2))], [('ExceptHandler', (3, 0), ('Name', (3, 7), 'Exception', ('Load',)), None, [('Pass', (4, 2))])], [], [])]),
 ('Module', [('Try', (1, 0), [('Pass', (2, 2))], [], [], [('Pass', (4, 2))])]),
 ('Module', [('Assert', (1, 0), ('Name', (1, 7), 'v', ('Load',)), None)]),
@@ -1022,17 +1005,17 @@
 ('Expression', ('ListComp', (1, 1), ('Name', (1, 1), 'a', ('Load',)), [('comprehension', ('Name', (1, 7), 'b', ('Store',)), ('Name', (1, 12), 'c', ('Load',)), [('Name', (1, 17), 'd', ('Load',))])])),
 ('Expression', ('GeneratorExp', (1, 1), ('Name', (1, 1), 'a', ('Load',)), [('comprehension', ('Name', (1, 7), 'b', ('Store',)), ('Name', (1, 12), 'c', ('Load',)), [('Name', (1, 17), 'd', ('Load',))])])),
 ('Expression', ('Compare', (1, 0), ('Num', (1, 0), 1), [('Lt',), ('Lt',)], [('Num', (1, 4), 2), ('Num', (1, 8), 3)])),
-('Expression', ('Call', (1, 1), ('Name', (1, 0), 'f', ('Load',)), [('Num', (1, 2), 1), ('Num', (1, 4), 2)], [('keyword', 'c', ('Num', (1, 8), 3))], ('Name', (1, 11), 'd', ('Load',)), ('Name', (1, 15), 'e', ('Load',)))),
+('Expression', ('Call', (1, 0), ('Name', (1, 0), 'f', ('Load',)), [('Num', (1, 2), 1), ('Num', (1, 4), 2)], [('keyword', 'c', ('Num', (1, 8), 3))], ('Name', (1, 11), 'd', ('Load',)), ('Name', (1, 15), 'e', ('Load',)))),
 ('Expression', ('Num', (1, 0), 10)),
 ('Expression', ('Str', (1, 0), 'string')),
-('Expression', ('Attribute', (1, 2), ('Name', (1, 0), 'a', ('Load',)), 'b', ('Load',))),
-('Expression', ('Subscript', (1, 2), ('Name', (1, 0), 'a', ('Load',)), ('Slice', ('Name', (1, 2), 'b', ('Load',)), ('Name', (1, 4), 'c', ('Load',)), None), ('Load',))),
+('Expression', ('Attribute', (1, 0), ('Name', (1, 0), 'a', ('Load',)), 'b', ('Load',))),
+('Expression', ('Subscript', (1, 0), ('Name', (1, 0), 'a', ('Load',)), ('Slice', ('Name', (1, 2), 'b', ('Load',)), ('Name', (1, 4), 'c', ('Load',)), None), ('Load',))),
 ('Expression', ('Name', (1, 0), 'v', ('Load',))),
 ('Expression', ('List', (1, 0), [('Num', (1, 1), 1), ('Num', (1, 3), 2), ('Num', (1, 5), 3)], ('Load',))),
 ('Expression', ('List', (1, 0), [], ('Load',))),
 ('Expression', ('Tuple', (1, 0), [('Num', (1, 0), 1), ('Num', (1, 2), 2), ('Num', (1, 4), 3)], ('Load',))),
 ('Expression', ('Tuple', (1, 1), [('Num', (1, 1), 1), ('Num', (1, 3), 2), ('Num', (1, 5), 3)], ('Load',))),
 ('Expression', ('Tuple', (1, 0), [], ('Load',))),
-('Expression', ('Call', (1, 7), ('Attribute', (1, 6), ('Attribute', (1, 4), ('Attribute', (1, 2), ('Name', (1, 0), 'a', ('Load',)), 'b', ('Load',)), 'c', ('Load',)), 'd', ('Load',)), [('Subscript', (1, 12), ('Attribute', (1, 10), ('Name', (1, 8), 'a', ('Load',)), 'b', ('Load',)), ('Slice', ('Num', (1, 12), 1), ('Num', (1, 14), 2), None), ('Load',))], [], None, None)),
+('Expression', ('Call', (1, 0), ('Attribute', (1, 0), ('Attribute', (1, 0), ('Attribute', (1, 0), ('Name', (1, 0), 'a', ('Load',)), 'b', ('Load',)), 'c', ('Load',)), 'd', ('Load',)), [('Subscript', (1, 8), ('Attribute', (1, 8), ('Name', (1, 8), 'a', ('Load',)), 'b', ('Load',)), ('Slice', ('Num', (1, 12), 1), ('Num', (1, 14), 2), None), ('Load',))], [], None, None)),
 ]
 main()
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -10,6 +10,9 @@
 Core and Builtins
 -----------------
 
+- Issue #21295: Revert some changes (issue #16795) to AST line numbers and
+  column offsets that constituted a regression.
+
 - Issue #22986: Allow changing an object's __class__ between a dynamic type and
   static type in some cases.
 
diff --git a/Python/ast.c b/Python/ast.c
--- a/Python/ast.c
+++ b/Python/ast.c
@@ -1127,7 +1127,7 @@
     identifier name;
     expr_ty annotation = NULL;
     node *ch;
-    arg_ty tmp;
+    arg_ty ret;
 
     assert(TYPE(n) == tfpdef || TYPE(n) == vfpdef);
     ch = CHILD(n, 0);
@@ -1143,13 +1143,12 @@
             return NULL;
     }
 
-    tmp = arg(name, annotation, c->c_arena);
-    if (!tmp)
+    ret = arg(name, annotation, c->c_arena);
+    if (!ret)
         return NULL;
-
-    tmp->lineno = LINENO(n);
-    tmp->col_offset = n->n_col_offset;
-    return tmp;
+    ret->lineno = LINENO(n);
+    ret->col_offset = n->n_col_offset;
+    return ret;
 }
 
 /* returns -1 if failed to handle keyword only arguments
@@ -2107,22 +2106,15 @@
         if (NCH(n) == 2)
             return Call(left_expr, NULL, NULL, NULL, NULL, LINENO(n),
                         n->n_col_offset, c->c_arena);
-        else {
-            expr_ty tmp = ast_for_call(c, CHILD(n, 1), left_expr);
-            if (!tmp)
-                return NULL;
-
-            tmp->lineno = LINENO(n);
-            tmp->col_offset = n->n_col_offset;
-            return tmp;
-        }
+        else
+            return ast_for_call(c, CHILD(n, 1), left_expr);
     }
-    else if (TYPE(CHILD(n, 0)) == DOT ) {
+    else if (TYPE(CHILD(n, 0)) == DOT) {
         PyObject *attr_id = NEW_IDENTIFIER(CHILD(n, 1));
         if (!attr_id)
             return NULL;
         return Attribute(left_expr, attr_id, Load,
-                         LINENO(CHILD(n, 1)), CHILD(n, 1)->n_col_offset, c->c_arena);
+                         LINENO(n), n->n_col_offset, c->c_arena);
     }
     else {
         REQ(CHILD(n, 0), LSQB);
@@ -2223,16 +2215,15 @@
         tmp = ast_for_trailer(c, ch, e);
         if (!tmp)
             return NULL;
+        tmp->lineno = e->lineno;
+        tmp->col_offset = e->col_offset;
         e = tmp;
     }
     if (TYPE(CHILD(n, NCH(n) - 1)) == factor) {
         expr_ty f = ast_for_expr(c, CHILD(n, NCH(n) - 1));
         if (!f)
             return NULL;
-        tmp = BinOp(e, Pow, f, LINENO(n), n->n_col_offset, c->c_arena);
-        if (!tmp)
-            return NULL;
-        e = tmp;
+        e = BinOp(e, Pow, f, LINENO(n), n->n_col_offset, c->c_arena);
     }
     return e;
 }
diff --git a/Python/importlib.h b/Python/importlib.h
--- a/Python/importlib.h
+++ b/Python/importlib.h
[stripped]

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Mon Feb  2 17:35:27 2015
From: python-checkins at python.org (raymond.hettinger)
Date: Mon, 02 Feb 2015 16:35:27 +0000
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_23359=3A_Tighten_inn?=
 =?utf-8?q?er_search_loop_for_sets_=28don=27t_and-mask_every_entry?=
Message-ID: <20150202163507.25859.398@psf.io>

https://hg.python.org/cpython/rev/0b3bc51341aa
changeset:   94463:0b3bc51341aa
user:        Raymond Hettinger <python at rcn.com>
date:        Mon Feb 02 08:35:00 2015 -0800
summary:
  Issue 23359: Tighten inner search loop for sets (don't and-mask every entry lookup).

files:
  Objects/setobject.c |  77 ++++++++++++++++++++++----------
  1 files changed, 53 insertions(+), 24 deletions(-)


diff --git a/Objects/setobject.c b/Objects/setobject.c
--- a/Objects/setobject.c
+++ b/Objects/setobject.c
@@ -88,31 +88,60 @@
         if (entry->key == dummy && freeslot == NULL)
             freeslot = entry;
 
-        for (j = 1 ; j <= LINEAR_PROBES ; j++) {
-            entry = &table[(i + j) & mask];
-            if (entry->key == NULL)
-                goto found_null;
-            if (entry->hash == hash) {
-                PyObject *startkey = entry->key;
-                assert(startkey != dummy);
-                if (startkey == key)
-                    return entry;
-                if (PyUnicode_CheckExact(startkey)
-                    && PyUnicode_CheckExact(key)
-                    && unicode_eq(startkey, key))
-                    return entry;
-                Py_INCREF(startkey);
-                cmp = PyObject_RichCompareBool(startkey, key, Py_EQ);
-                Py_DECREF(startkey);
-                if (cmp < 0)
-                    return NULL;
-                if (table != so->table || entry->key != startkey)
-                    return set_lookkey(so, key, hash);
-                if (cmp > 0)
-                    return entry;
+        if (i + LINEAR_PROBES <= mask) {
+            for (j = 1 ; j <= LINEAR_PROBES ; j++) {
+                entry++;
+                if (entry->key == NULL)
+                    goto found_null;
+                if (entry->hash == hash) {
+                    PyObject *startkey = entry->key;
+                    assert(startkey != dummy);
+                    if (startkey == key)
+                        return entry;
+                    if (PyUnicode_CheckExact(startkey)
+                        && PyUnicode_CheckExact(key)
+                        && unicode_eq(startkey, key))
+                        return entry;
+                    Py_INCREF(startkey);
+                    cmp = PyObject_RichCompareBool(startkey, key, Py_EQ);
+                    Py_DECREF(startkey);
+                    if (cmp < 0)
+                        return NULL;
+                    if (table != so->table || entry->key != startkey)
+                        return set_lookkey(so, key, hash);
+                    if (cmp > 0)
+                        return entry;
+                }
+                if (entry->key == dummy && freeslot == NULL)
+                    freeslot = entry;
             }
-            if (entry->key == dummy && freeslot == NULL)
-                freeslot = entry;
+        } else {
+            for (j = 1 ; j <= LINEAR_PROBES ; j++) {
+                entry = &table[(i + j) & mask];
+                if (entry->key == NULL)
+                    goto found_null;
+                if (entry->hash == hash) {
+                    PyObject *startkey = entry->key;
+                    assert(startkey != dummy);
+                    if (startkey == key)
+                        return entry;
+                    if (PyUnicode_CheckExact(startkey)
+                        && PyUnicode_CheckExact(key)
+                        && unicode_eq(startkey, key))
+                        return entry;
+                    Py_INCREF(startkey);
+                    cmp = PyObject_RichCompareBool(startkey, key, Py_EQ);
+                    Py_DECREF(startkey);
+                    if (cmp < 0)
+                        return NULL;
+                    if (table != so->table || entry->key != startkey)
+                        return set_lookkey(so, key, hash);
+                    if (cmp > 0)
+                        return entry;
+                }
+                if (entry->key == dummy && freeslot == NULL)
+                    freeslot = entry;
+            }
         }
 
         perturb >>= PERTURB_SHIFT;

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Mon Feb  2 18:38:10 2015
From: python-checkins at python.org (victor.stinner)
Date: Mon, 02 Feb 2015 17:38:10 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio=29?=
Message-ID: <20150202173756.39270.61855@psf.io>

https://hg.python.org/cpython/rev/42b376c8cf60
changeset:   94465:42b376c8cf60
parent:      94463:0b3bc51341aa
parent:      94464:2cd6621a9fbc
user:        Victor Stinner <victor.stinner at gmail.com>
date:        Mon Feb 02 18:36:59 2015 +0100
summary:
  Merge 3.4 (asyncio)

files:
  Lib/asyncio/test_utils.py     |   4 ++++
  Lib/asyncio/unix_events.py    |  12 ++++++++++--
  Lib/asyncio/windows_events.py |  11 +++++++++--
  3 files changed, 23 insertions(+), 4 deletions(-)


diff --git a/Lib/asyncio/test_utils.py b/Lib/asyncio/test_utils.py
--- a/Lib/asyncio/test_utils.py
+++ b/Lib/asyncio/test_utils.py
@@ -416,6 +416,10 @@
     def tearDown(self):
         events.set_event_loop(None)
 
+        # Detect CPython bug #23353: ensure that yield/yield-from is not used
+        # in an except block of a generator
+        self.assertEqual(sys.exc_info(), (None, None, None))
+
 
 @contextlib.contextmanager
 def disable_logger():
diff --git a/Lib/asyncio/unix_events.py b/Lib/asyncio/unix_events.py
--- a/Lib/asyncio/unix_events.py
+++ b/Lib/asyncio/unix_events.py
@@ -186,10 +186,18 @@
                                       self._child_watcher_callback, transp)
             try:
                 yield from waiter
-            except:
+            except Exception as exc:
+                # Workaround CPython bug #23353: using yield/yield-from in an
+                # except block of a generator doesn't clear properly
+                # sys.exc_info()
+                err = exc
+            else:
+                err = None
+
+            if err is not None:
                 transp.close()
                 yield from transp._wait()
-                raise
+                raise err
 
         return transp
 
diff --git a/Lib/asyncio/windows_events.py b/Lib/asyncio/windows_events.py
--- a/Lib/asyncio/windows_events.py
+++ b/Lib/asyncio/windows_events.py
@@ -373,10 +373,17 @@
                                              **kwargs)
         try:
             yield from waiter
-        except:
+        except Exception as exc:
+            # Workaround CPython bug #23353: using yield/yield-from in an
+            # except block of a generator doesn't clear properly sys.exc_info()
+            err = exc
+        else:
+            err = None
+
+        if err is not None:
             transp.close()
             yield from transp._wait()
-            raise
+            raise err
 
         return transp
 

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Mon Feb  2 18:38:13 2015
From: python-checkins at python.org (victor.stinner)
Date: Mon, 02 Feb 2015 17:38:13 +0000
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIzMzUz?=
 =?utf-8?q?=2C_asyncio=3A_Workaround_CPython_bug_=2323353?=
Message-ID: <20150202173755.25859.54709@psf.io>

https://hg.python.org/cpython/rev/2cd6621a9fbc
changeset:   94464:2cd6621a9fbc
branch:      3.4
parent:      94461:7d1c32ddc432
user:        Victor Stinner <victor.stinner at gmail.com>
date:        Mon Feb 02 18:36:31 2015 +0100
summary:
  Issue #23353, asyncio: Workaround CPython bug #23353

Don't use yield/yield-from in an except block of a generator. Store the
exception and handle it outside the except block.

files:
  Lib/asyncio/test_utils.py     |   4 ++++
  Lib/asyncio/unix_events.py    |  12 ++++++++++--
  Lib/asyncio/windows_events.py |  11 +++++++++--
  3 files changed, 23 insertions(+), 4 deletions(-)


diff --git a/Lib/asyncio/test_utils.py b/Lib/asyncio/test_utils.py
--- a/Lib/asyncio/test_utils.py
+++ b/Lib/asyncio/test_utils.py
@@ -416,6 +416,10 @@
     def tearDown(self):
         events.set_event_loop(None)
 
+        # Detect CPython bug #23353: ensure that yield/yield-from is not used
+        # in an except block of a generator
+        self.assertEqual(sys.exc_info(), (None, None, None))
+
 
 @contextlib.contextmanager
 def disable_logger():
diff --git a/Lib/asyncio/unix_events.py b/Lib/asyncio/unix_events.py
--- a/Lib/asyncio/unix_events.py
+++ b/Lib/asyncio/unix_events.py
@@ -186,10 +186,18 @@
                                       self._child_watcher_callback, transp)
             try:
                 yield from waiter
-            except:
+            except Exception as exc:
+                # Workaround CPython bug #23353: using yield/yield-from in an
+                # except block of a generator doesn't clear properly
+                # sys.exc_info()
+                err = exc
+            else:
+                err = None
+
+            if err is not None:
                 transp.close()
                 yield from transp._wait()
-                raise
+                raise err
 
         return transp
 
diff --git a/Lib/asyncio/windows_events.py b/Lib/asyncio/windows_events.py
--- a/Lib/asyncio/windows_events.py
+++ b/Lib/asyncio/windows_events.py
@@ -373,10 +373,17 @@
                                              **kwargs)
         try:
             yield from waiter
-        except:
+        except Exception as exc:
+            # Workaround CPython bug #23353: using yield/yield-from in an
+            # except block of a generator doesn't clear properly sys.exc_info()
+            err = exc
+        else:
+            err = None
+
+        if err is not None:
             transp.close()
             yield from transp._wait()
-            raise
+            raise err
 
         return transp
 

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Mon Feb  2 20:06:38 2015
From: python-checkins at python.org (benjamin.peterson)
Date: Mon, 02 Feb 2015 19:06:38 +0000
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogX2NsZWFyX3R5cGVf?=
	=?utf-8?q?cache_is_cpython-only?=
Message-ID: <20150202190633.106327.12022@psf.io>

https://hg.python.org/cpython/rev/70966f32793d
changeset:   94467:70966f32793d
branch:      3.4
parent:      94464:2cd6621a9fbc
user:        Benjamin Peterson <benjamin at python.org>
date:        Mon Feb 02 14:06:11 2015 -0500
summary:
  _clear_type_cache is cpython-only

files:
  Lib/test/test_sys.py |  1 +
  1 files changed, 1 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_sys.py b/Lib/test/test_sys.py
--- a/Lib/test/test_sys.py
+++ b/Lib/test/test_sys.py
@@ -539,6 +539,7 @@
         test.support.get_attribute(sys, "getwindowsversion")
         self.assert_raise_on_new_sys_type(sys.getwindowsversion())
 
+    @test.test_support.cpython_only
     def test_clear_type_cache(self):
         sys._clear_type_cache()
 

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Mon Feb  2 20:06:38 2015
From: python-checkins at python.org (benjamin.peterson)
Date: Mon, 02 Feb 2015 19:06:38 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
	=?utf-8?b?KTogbWVyZ2UgMy40?=
Message-ID: <20150202190633.106264.71652@psf.io>

https://hg.python.org/cpython/rev/c670249c7956
changeset:   94468:c670249c7956
parent:      94465:42b376c8cf60
parent:      94467:70966f32793d
user:        Benjamin Peterson <benjamin at python.org>
date:        Mon Feb 02 14:06:29 2015 -0500
summary:
  merge 3.4

files:
  Lib/test/test_sys.py |  1 +
  1 files changed, 1 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_sys.py b/Lib/test/test_sys.py
--- a/Lib/test/test_sys.py
+++ b/Lib/test/test_sys.py
@@ -539,6 +539,7 @@
         test.support.get_attribute(sys, "getwindowsversion")
         self.assert_raise_on_new_sys_type(sys.getwindowsversion())
 
+    @test.test_support.cpython_only
     def test_clear_type_cache(self):
         sys._clear_type_cache()
 

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Mon Feb  2 20:06:38 2015
From: python-checkins at python.org (benjamin.peterson)
Date: Mon, 02 Feb 2015 19:06:38 +0000
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogX2NsZWFyX3R5cGVf?=
	=?utf-8?q?cache_is_cpython-only?=
Message-ID: <20150202190633.34388.58215@psf.io>

https://hg.python.org/cpython/rev/83b75b016c77
changeset:   94466:83b75b016c77
branch:      2.7
parent:      94459:acc2c3479f2e
user:        Benjamin Peterson <benjamin at python.org>
date:        Mon Feb 02 14:06:11 2015 -0500
summary:
  _clear_type_cache is cpython-only

files:
  Lib/test/test_sys.py |  1 +
  1 files changed, 1 insertions(+), 0 deletions(-)


diff --git a/Lib/test/test_sys.py b/Lib/test/test_sys.py
--- a/Lib/test/test_sys.py
+++ b/Lib/test/test_sys.py
@@ -425,6 +425,7 @@
             self.assertEqual(type(getattr(sys.flags, attr)), int, attr)
         self.assertTrue(repr(sys.flags))
 
+    @test.test_support.cpython_only
     def test_clear_type_cache(self):
         sys._clear_type_cache()
 

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Mon Feb  2 20:22:29 2015
From: python-checkins at python.org (benjamin.peterson)
Date: Mon, 02 Feb 2015 19:22:29 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_adjust_for_py3?=
	=?utf-8?q?k_module_renaming?=
Message-ID: <20150202192225.34390.27827@psf.io>

https://hg.python.org/cpython/rev/dc699f150fd3
changeset:   94469:dc699f150fd3
branch:      3.4
parent:      94467:70966f32793d
user:        Benjamin Peterson <benjamin at python.org>
date:        Mon Feb 02 14:22:13 2015 -0500
summary:
  adjust for py3k module renaming

files:
  Lib/test/test_sys.py |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Lib/test/test_sys.py b/Lib/test/test_sys.py
--- a/Lib/test/test_sys.py
+++ b/Lib/test/test_sys.py
@@ -539,7 +539,7 @@
         test.support.get_attribute(sys, "getwindowsversion")
         self.assert_raise_on_new_sys_type(sys.getwindowsversion())
 
-    @test.test_support.cpython_only
+    @test.support.cpython_only
     def test_clear_type_cache(self):
         sys._clear_type_cache()
 

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Mon Feb  2 20:22:29 2015
From: python-checkins at python.org (benjamin.peterson)
Date: Mon, 02 Feb 2015 19:22:29 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
	=?utf-8?b?KTogbWVyZ2UgMy40?=
Message-ID: <20150202192225.25867.49368@psf.io>

https://hg.python.org/cpython/rev/298f56ee74f4
changeset:   94470:298f56ee74f4
parent:      94468:c670249c7956
parent:      94469:dc699f150fd3
user:        Benjamin Peterson <benjamin at python.org>
date:        Mon Feb 02 14:22:19 2015 -0500
summary:
  merge 3.4

files:
  Lib/test/test_sys.py |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Lib/test/test_sys.py b/Lib/test/test_sys.py
--- a/Lib/test/test_sys.py
+++ b/Lib/test/test_sys.py
@@ -539,7 +539,7 @@
         test.support.get_attribute(sys, "getwindowsversion")
         self.assert_raise_on_new_sys_type(sys.getwindowsversion())
 
-    @test.test_support.cpython_only
+    @test.support.cpython_only
     def test_clear_type_cache(self):
         sys._clear_type_cache()
 

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Mon Feb  2 21:39:28 2015
From: python-checkins at python.org (antoine.pitrou)
Date: Mon, 02 Feb 2015 20:39:28 +0000
Subject: [Python-checkins] =?utf-8?q?peps=3A_Update_PEP_with_latest_change?=
	=?utf-8?q?s=2E?=
Message-ID: <20150202203858.96068.23038@psf.io>

https://hg.python.org/peps/rev/9cdb1b30f643
changeset:   5689:9cdb1b30f643
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Mon Feb 02 21:38:52 2015 +0100
summary:
  Update PEP with latest changes.

files:
  pep-0475.txt |  7 +++++++
  1 files changed, 7 insertions(+), 0 deletions(-)


diff --git a/pep-0475.txt b/pep-0475.txt
--- a/pep-0475.txt
+++ b/pep-0475.txt
@@ -167,6 +167,7 @@
 Example of standard library functions that need to be modified to comply
 with this PEP:
 
+* ``open()``, ``os.open()``
 * ``os.read()``, ``io.FileIO.read()``, ``io.FileIO.readinto()``
 * ``os.write()``, ``io.FileIO.write()``
 * ``os.waitpid()``
@@ -187,6 +188,12 @@
 (note: the ``selector`` module already retries on ``InterruptedError``, but it
 doesn't recompute the timeout yet)
 
+``os.close`` and ``close()`` methods are a special case: they will ignore
+EINTR instead of retrying.  The reason is complex but involves behaviour
+under Linux and the fact that `the file descriptor may really be closed
+<http://lwn.net/Articles/576478/>`_ even if EINTR is returned.
+
+
 InterruptedError handling
 -------------------------
 

-- 
Repository URL: https://hg.python.org/peps

From python-checkins at python.org  Mon Feb  2 21:47:18 2015
From: python-checkins at python.org (antoine.pitrou)
Date: Mon, 02 Feb 2015 20:47:18 +0000
Subject: [Python-checkins] =?utf-8?q?peps=3A_Mark_accepted?=
Message-ID: <20150202204701.106379.91104@psf.io>

https://hg.python.org/peps/rev/7edc0eddb68f
changeset:   5690:7edc0eddb68f
user:        Antoine Pitrou <solipsis at pitrou.net>
date:        Mon Feb 02 21:46:57 2015 +0100
summary:
  Mark accepted

files:
  pep-0475.txt |  3 ++-
  1 files changed, 2 insertions(+), 1 deletions(-)


diff --git a/pep-0475.txt b/pep-0475.txt
--- a/pep-0475.txt
+++ b/pep-0475.txt
@@ -4,11 +4,12 @@
 Last-Modified: $Date$
 Author: Charles-Fran?ois Natali <cf.natali at gmail.com>, Victor Stinner <victor.stinner at gmail.com>
 BDFL-Delegate: Antoine Pitrou <solipsis at pitrou.net>
-Status: Draft
+Status: Accepted
 Type: Standards Track
 Content-Type: text/x-rst
 Created: 29-July-2014
 Python-Version: 3.5
+Resolution: https://mail.python.org/pipermail/python-dev/2015-February/138018.html
 
 
 Abstract

-- 
Repository URL: https://hg.python.org/peps

From python-checkins at python.org  Mon Feb  2 23:47:39 2015
From: python-checkins at python.org (benjamin.peterson)
Date: Mon, 02 Feb 2015 22:47:39 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E3=29=3A_reduce_memory_?=
 =?utf-8?q?usage_of_test_=28closes_=2323369=29?=
Message-ID: <20150202224736.106317.76992@psf.io>

https://hg.python.org/cpython/rev/5c730d30ffbc
changeset:   94471:5c730d30ffbc
branch:      3.3
parent:      94456:7133582b6769
user:        Benjamin Peterson <benjamin at python.org>
date:        Mon Feb 02 17:47:07 2015 -0500
summary:
  reduce memory usage of test (closes #23369)

files:
  Lib/test/test_json/test_encode_basestring_ascii.py |  3 ++-
  1 files changed, 2 insertions(+), 1 deletions(-)


diff --git a/Lib/test/test_json/test_encode_basestring_ascii.py b/Lib/test/test_json/test_encode_basestring_ascii.py
--- a/Lib/test/test_json/test_encode_basestring_ascii.py
+++ b/Lib/test/test_json/test_encode_basestring_ascii.py
@@ -45,6 +45,7 @@
 class TestCEncodeBasestringAscii(TestEncodeBasestringAscii, CTest):
     @bigaddrspacetest
     def test_overflow(self):
-        s = "\uffff"*((2**32)//6 + 1)
+        size = (2**32)//6 + 1
+        s = "\x00"*size
         with self.assertRaises(OverflowError):
             self.json.encoder.encode_basestring_ascii(s)

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Mon Feb  2 23:47:39 2015
From: python-checkins at python.org (benjamin.peterson)
Date: Mon, 02 Feb 2015 22:47:39 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
	=?utf-8?b?KTogbWVyZ2UgMy40?=
Message-ID: <20150202224736.96094.55799@psf.io>

https://hg.python.org/cpython/rev/651aa21433ba
changeset:   94473:651aa21433ba
parent:      94470:298f56ee74f4
parent:      94472:b883bb8bd744
user:        Benjamin Peterson <benjamin at python.org>
date:        Mon Feb 02 17:47:31 2015 -0500
summary:
  merge 3.4

files:
  Lib/test/test_json/test_encode_basestring_ascii.py |  3 ++-
  1 files changed, 2 insertions(+), 1 deletions(-)


diff --git a/Lib/test/test_json/test_encode_basestring_ascii.py b/Lib/test/test_json/test_encode_basestring_ascii.py
--- a/Lib/test/test_json/test_encode_basestring_ascii.py
+++ b/Lib/test/test_json/test_encode_basestring_ascii.py
@@ -42,6 +42,7 @@
 class TestCEncodeBasestringAscii(TestEncodeBasestringAscii, CTest):
     @bigaddrspacetest
     def test_overflow(self):
-        s = "\uffff"*((2**32)//6 + 1)
+        size = (2**32)//6 + 1
+        s = "\x00"*size
         with self.assertRaises(OverflowError):
             self.json.encoder.encode_basestring_ascii(s)

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Mon Feb  2 23:47:39 2015
From: python-checkins at python.org (benjamin.peterson)
Date: Mon, 02 Feb 2015 22:47:39 +0000
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4zIC0+IDMuNCk6?=
	=?utf-8?q?_merge_3=2E3?=
Message-ID: <20150202224736.39292.30713@psf.io>

https://hg.python.org/cpython/rev/b883bb8bd744
changeset:   94472:b883bb8bd744
branch:      3.4
parent:      94469:dc699f150fd3
parent:      94471:5c730d30ffbc
user:        Benjamin Peterson <benjamin at python.org>
date:        Mon Feb 02 17:47:26 2015 -0500
summary:
  merge 3.3

files:
  Lib/test/test_json/test_encode_basestring_ascii.py |  3 ++-
  1 files changed, 2 insertions(+), 1 deletions(-)


diff --git a/Lib/test/test_json/test_encode_basestring_ascii.py b/Lib/test/test_json/test_encode_basestring_ascii.py
--- a/Lib/test/test_json/test_encode_basestring_ascii.py
+++ b/Lib/test/test_json/test_encode_basestring_ascii.py
@@ -45,6 +45,7 @@
 class TestCEncodeBasestringAscii(TestEncodeBasestringAscii, CTest):
     @bigaddrspacetest
     def test_overflow(self):
-        s = "\uffff"*((2**32)//6 + 1)
+        size = (2**32)//6 + 1
+        s = "\x00"*size
         with self.assertRaises(OverflowError):
             self.json.encoder.encode_basestring_ascii(s)

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Tue Feb  3 01:05:57 2015
From: python-checkins at python.org (serhiy.storchaka)
Date: Tue, 03 Feb 2015 00:05:57 +0000
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIzMDk5?=
 =?utf-8?q?=3A_Closing_io=2EBytesIO_with_exported_buffer_is_rejected_now_t?=
 =?utf-8?q?o?=
Message-ID: <20150203000516.106346.11737@psf.io>

https://hg.python.org/cpython/rev/e62d54128bd3
changeset:   94480:e62d54128bd3
branch:      3.4
parent:      94477:98c720c3e061
user:        Serhiy Storchaka <storchaka at gmail.com>
date:        Tue Feb 03 02:00:18 2015 +0200
summary:
  Issue #23099: Closing io.BytesIO with exported buffer is rejected now to
prevent corrupting exported buffer.

files:
  Doc/library/io.rst        |  13 +++++++------
  Lib/_pyio.py              |   6 ++++++
  Lib/test/test_memoryio.py |   7 ++++++-
  Misc/NEWS                 |   3 +++
  Modules/_io/bytesio.c     |   1 +
  5 files changed, 23 insertions(+), 7 deletions(-)


diff --git a/Doc/library/io.rst b/Doc/library/io.rst
--- a/Doc/library/io.rst
+++ b/Doc/library/io.rst
@@ -563,7 +563,8 @@
 .. class:: BytesIO([initial_bytes])
 
    A stream implementation using an in-memory bytes buffer.  It inherits
-   :class:`BufferedIOBase`.
+   :class:`BufferedIOBase`.  The buffer is discarded when the
+   :meth:`~IOBase.close` method is called.
 
    The argument *initial_bytes* contains optional initial :class:`bytes` data.
 
@@ -584,7 +585,7 @@
 
       .. note::
          As long as the view exists, the :class:`BytesIO` object cannot be
-         resized.
+         resized or closed.
 
       .. versionadded:: 3.2
 
@@ -592,6 +593,7 @@
 
       Return :class:`bytes` containing the entire contents of the buffer.
 
+
    .. method:: read1()
 
       In :class:`BytesIO`, this is the same as :meth:`read`.
@@ -858,7 +860,8 @@
 
 .. class:: StringIO(initial_value='', newline='\\n')
 
-   An in-memory stream for text I/O.
+   An in-memory stream for text I/O.  The text buffer is discarded when the
+   :meth:`~IOBase.close` method is called.
 
    The initial value of the buffer (an empty string by default) can be set by
    providing *initial_value*.  The *newline* argument works like that of
@@ -870,9 +873,7 @@
 
    .. method:: getvalue()
 
-      Return a ``str`` containing the entire contents of the buffer at any
-      time before the :class:`StringIO` object's :meth:`close` method is
-      called.
+      Return a ``str`` containing the entire contents of the buffer.
 
    Example usage::
 
diff --git a/Lib/_pyio.py b/Lib/_pyio.py
--- a/Lib/_pyio.py
+++ b/Lib/_pyio.py
@@ -833,8 +833,14 @@
     def getbuffer(self):
         """Return a readable and writable view of the buffer.
         """
+        if self.closed:
+            raise ValueError("getbuffer on closed file")
         return memoryview(self._buffer)
 
+    def close(self):
+        self._buffer.clear()
+        super().close()
+
     def read(self, size=None):
         if self.closed:
             raise ValueError("read from closed file")
diff --git a/Lib/test/test_memoryio.py b/Lib/test/test_memoryio.py
--- a/Lib/test/test_memoryio.py
+++ b/Lib/test/test_memoryio.py
@@ -398,14 +398,19 @@
         # raises a BufferError.
         self.assertRaises(BufferError, memio.write, b'x' * 100)
         self.assertRaises(BufferError, memio.truncate)
+        self.assertRaises(BufferError, memio.close)
+        self.assertFalse(memio.closed)
         # Mutating the buffer updates the BytesIO
         buf[3:6] = b"abc"
         self.assertEqual(bytes(buf), b"123abc7890")
         self.assertEqual(memio.getvalue(), b"123abc7890")
-        # After the buffer gets released, we can resize the BytesIO again
+        # After the buffer gets released, we can resize and close the BytesIO
+        # again
         del buf
         support.gc_collect()
         memio.truncate()
+        memio.close()
+        self.assertRaises(ValueError, memio.getbuffer)
 
 
 class PyBytesIOTest(MemoryTestMixin, MemorySeekTestMixin,
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -56,6 +56,9 @@
 Library
 -------
 
+- Issue #23099: Closing io.BytesIO with exported buffer is rejected now to
+  prevent corrupting exported buffer.
+
 - Issue #23363: Fix possible overflow in itertools.permutations.
 
 - Issue #23364: Fix possible overflow in itertools.product.
diff --git a/Modules/_io/bytesio.c b/Modules/_io/bytesio.c
--- a/Modules/_io/bytesio.c
+++ b/Modules/_io/bytesio.c
@@ -657,6 +657,7 @@
 static PyObject *
 bytesio_close(bytesio *self)
 {
+    CHECK_EXPORTS(self);
     if (self->buf != NULL) {
         PyMem_Free(self->buf);
         self->buf = NULL;

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Tue Feb  3 01:05:57 2015
From: python-checkins at python.org (serhiy.storchaka)
Date: Tue, 03 Feb 2015 00:05:57 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?b?KTogSXNzdWVzICMyMzM2MywgIzIzMzY0LCAjMjMzNjUsICMyMzM2NjogRml4?=
 =?utf-8?q?ed_itertools_overflow_tests=2E?=
Message-ID: <20150203000516.39272.69905@psf.io>

https://hg.python.org/cpython/rev/4cb316fe6bf2
changeset:   94479:4cb316fe6bf2
parent:      94478:0024537a4687
parent:      94477:98c720c3e061
user:        Serhiy Storchaka <storchaka at gmail.com>
date:        Tue Feb 03 01:50:31 2015 +0200
summary:
  Issues #23363, #23364, #23365, #23366: Fixed itertools overflow tests.
Used PyMem_New to check overflow.

files:
  Lib/test/test_itertools.py |  12 +++++-------
  Modules/itertoolsmodule.c  |  26 ++++++--------------------
  2 files changed, 11 insertions(+), 27 deletions(-)


diff --git a/Lib/test/test_itertools.py b/Lib/test/test_itertools.py
--- a/Lib/test/test_itertools.py
+++ b/Lib/test/test_itertools.py
@@ -266,7 +266,7 @@
 
     @support.bigaddrspacetest
     def test_combinations_overflow(self):
-        with self.assertRaises(OverflowError):
+        with self.assertRaises((OverflowError, MemoryError)):
             combinations("AA", 2**29)
 
         # Test implementation detail:  tuple re-use
@@ -353,7 +353,7 @@
 
     @support.bigaddrspacetest
     def test_combinations_with_replacement_overflow(self):
-        with self.assertRaises(OverflowError):
+        with self.assertRaises((OverflowError, MemoryError)):
             combinations_with_replacement("AA", 2**30)
 
         # Test implementation detail:  tuple re-use
@@ -428,10 +428,8 @@
 
     @support.bigaddrspacetest
     def test_permutations_overflow(self):
-        with self.assertRaises(OverflowError):
+        with self.assertRaises((OverflowError, MemoryError)):
             permutations("A", 2**30)
-        with self.assertRaises(OverflowError):
-            permutations("A", 2, 2**30)
 
     @support.impl_detail("tuple reuse is specific to CPython")
     def test_permutations_tuple_reuse(self):
@@ -964,8 +962,8 @@
 
     @support.bigaddrspacetest
     def test_product_overflow(self):
-        with self.assertRaises(OverflowError):
-            product(["a"]*(2**16), repeat=2**16)
+        with self.assertRaises((OverflowError, MemoryError)):
+            product(*(['ab']*2**5), repeat=2**25)
 
     @support.impl_detail("tuple reuse is specific to CPython")
     def test_product_tuple_reuse(self):
diff --git a/Modules/itertoolsmodule.c b/Modules/itertoolsmodule.c
--- a/Modules/itertoolsmodule.c
+++ b/Modules/itertoolsmodule.c
@@ -2022,15 +2022,14 @@
         nargs = 0;
     } else {
         nargs = PyTuple_GET_SIZE(args);
-        if (repeat > PY_SSIZE_T_MAX/sizeof(Py_ssize_t) ||
-            nargs > PY_SSIZE_T_MAX/(repeat * sizeof(Py_ssize_t))) {
+        if ((size_t)nargs > PY_SSIZE_T_MAX/sizeof(Py_ssize_t)/repeat) {
             PyErr_SetString(PyExc_OverflowError, "repeat argument too large");
             return NULL;
         }
     }
     npools = nargs * repeat;
 
-    indices = PyMem_Malloc(npools * sizeof(Py_ssize_t));
+    indices = PyMem_New(Py_ssize_t, npools);
     if (indices == NULL) {
         PyErr_NoMemory();
         goto error;
@@ -2368,11 +2367,7 @@
         goto error;
     }
 
-    if (r > PY_SSIZE_T_MAX/sizeof(Py_ssize_t)) {
-        PyErr_SetString(PyExc_OverflowError, "r is too big");
-        goto error;
-    }
-    indices = PyMem_Malloc(r * sizeof(Py_ssize_t));
+    indices = PyMem_New(Py_ssize_t, r);
     if (indices == NULL) {
         PyErr_NoMemory();
         goto error;
@@ -2713,11 +2708,7 @@
         goto error;
     }
 
-    if (r > PY_SSIZE_T_MAX/sizeof(Py_ssize_t)) {
-        PyErr_SetString(PyExc_OverflowError, "r is too big");
-        goto error;
-    }
-    indices = PyMem_Malloc(r * sizeof(Py_ssize_t));
+    indices = PyMem_New(Py_ssize_t, r);
     if (indices == NULL) {
         PyErr_NoMemory();
         goto error;
@@ -3058,13 +3049,8 @@
         goto error;
     }
 
-    if (n > PY_SSIZE_T_MAX/sizeof(Py_ssize_t) ||
-        r > PY_SSIZE_T_MAX/sizeof(Py_ssize_t)) {
-        PyErr_SetString(PyExc_OverflowError, "parameters too large");
-        goto error;
-    }
-    indices = PyMem_Malloc(n * sizeof(Py_ssize_t));
-    cycles = PyMem_Malloc(r * sizeof(Py_ssize_t));
+    indices = PyMem_New(Py_ssize_t, n);
+    cycles = PyMem_New(Py_ssize_t, r);
     if (indices == NULL || cycles == NULL) {
         PyErr_NoMemory();
         goto error;

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Tue Feb  3 01:05:58 2015
From: python-checkins at python.org (serhiy.storchaka)
Date: Tue, 03 Feb 2015 00:05:58 +0000
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2322896=3A_Fixed_us?=
 =?utf-8?q?ing_=5Fgetbuffer=28=29_in_recently_added_=5FPyBytes=5FFormat=28?=
 =?utf-8?b?KS4=?=
Message-ID: <20150203000515.39296.45267@psf.io>

https://hg.python.org/cpython/rev/0024537a4687
changeset:   94478:0024537a4687
parent:      94475:2e684ce772de
user:        Serhiy Storchaka <storchaka at gmail.com>
date:        Tue Feb 03 01:49:18 2015 +0200
summary:
  Issue #22896: Fixed using _getbuffer() in recently added _PyBytes_Format().

files:
  Objects/bytesobject.c |  5 ++---
  1 files changed, 2 insertions(+), 3 deletions(-)


diff --git a/Objects/bytesobject.c b/Objects/bytesobject.c
--- a/Objects/bytesobject.c
+++ b/Objects/bytesobject.c
@@ -622,14 +622,13 @@
             int isnumok;
             PyObject *v = NULL;
             PyObject *temp = NULL;
-            Py_buffer buf;
+            Py_buffer buf = {NULL, NULL};
             char *pbuf;
             int sign;
             Py_ssize_t len;
             char formatbuf[FORMATBUFLEN];
                  /* For format{int,char}() */
 
-            buf.obj = NULL;
             fmt++;
             if (*fmt == '(') {
                 char *keystart;
@@ -790,7 +789,7 @@
                     Py_DECREF(temp);
                     goto error;
                 }
-                if (_getbuffer(repr, &buf) < 0) {
+                if (PyObject_GetBuffer(repr, &buf, PyBUF_SIMPLE) != 0) {
                     temp = format_obj(repr);
                     if (temp == NULL) {
                         Py_DECREF(repr);

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Tue Feb  3 01:05:57 2015
From: python-checkins at python.org (serhiy.storchaka)
Date: Tue, 03 Feb 2015 00:05:57 +0000
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAobWVyZ2UgMy4zIC0+IDMuNCk6?=
 =?utf-8?q?_Issues_=2323363=2C_=2323364=2C_=2323365=2C_=2323366=3A_Fixed_i?=
 =?utf-8?q?tertools_overflow_tests=2E?=
Message-ID: <20150203000515.96094.80555@psf.io>

https://hg.python.org/cpython/rev/98c720c3e061
changeset:   94477:98c720c3e061
branch:      3.4
parent:      94474:1da9630e9b7f
parent:      94476:356ed025dbae
user:        Serhiy Storchaka <storchaka at gmail.com>
date:        Tue Feb 03 01:35:10 2015 +0200
summary:
  Issues #23363, #23364, #23365, #23366: Fixed itertools overflow tests.
Used PyMem_New to check overflow.

files:
  Lib/test/test_itertools.py |  12 +++++-------
  Modules/itertoolsmodule.c  |  26 ++++++--------------------
  2 files changed, 11 insertions(+), 27 deletions(-)


diff --git a/Lib/test/test_itertools.py b/Lib/test/test_itertools.py
--- a/Lib/test/test_itertools.py
+++ b/Lib/test/test_itertools.py
@@ -266,7 +266,7 @@
 
     @support.bigaddrspacetest
     def test_combinations_overflow(self):
-        with self.assertRaises(OverflowError):
+        with self.assertRaises((OverflowError, MemoryError)):
             combinations("AA", 2**29)
 
         # Test implementation detail:  tuple re-use
@@ -353,7 +353,7 @@
 
     @support.bigaddrspacetest
     def test_combinations_with_replacement_overflow(self):
-        with self.assertRaises(OverflowError):
+        with self.assertRaises((OverflowError, MemoryError)):
             combinations_with_replacement("AA", 2**30)
 
         # Test implementation detail:  tuple re-use
@@ -428,10 +428,8 @@
 
     @support.bigaddrspacetest
     def test_permutations_overflow(self):
-        with self.assertRaises(OverflowError):
+        with self.assertRaises((OverflowError, MemoryError)):
             permutations("A", 2**30)
-        with self.assertRaises(OverflowError):
-            permutations("A", 2, 2**30)
 
     @support.impl_detail("tuple reuse is specific to CPython")
     def test_permutations_tuple_reuse(self):
@@ -964,8 +962,8 @@
 
     @support.bigaddrspacetest
     def test_product_overflow(self):
-        with self.assertRaises(OverflowError):
-            product(["a"]*(2**16), repeat=2**16)
+        with self.assertRaises((OverflowError, MemoryError)):
+            product(*(['ab']*2**5), repeat=2**25)
 
     @support.impl_detail("tuple reuse is specific to CPython")
     def test_product_tuple_reuse(self):
diff --git a/Modules/itertoolsmodule.c b/Modules/itertoolsmodule.c
--- a/Modules/itertoolsmodule.c
+++ b/Modules/itertoolsmodule.c
@@ -2022,15 +2022,14 @@
         nargs = 0;
     } else {
         nargs = PyTuple_GET_SIZE(args);
-        if (repeat > PY_SSIZE_T_MAX/sizeof(Py_ssize_t) ||
-            nargs > PY_SSIZE_T_MAX/(repeat * sizeof(Py_ssize_t))) {
+        if ((size_t)nargs > PY_SSIZE_T_MAX/sizeof(Py_ssize_t)/repeat) {
             PyErr_SetString(PyExc_OverflowError, "repeat argument too large");
             return NULL;
         }
     }
     npools = nargs * repeat;
 
-    indices = PyMem_Malloc(npools * sizeof(Py_ssize_t));
+    indices = PyMem_New(Py_ssize_t, npools);
     if (indices == NULL) {
         PyErr_NoMemory();
         goto error;
@@ -2368,11 +2367,7 @@
         goto error;
     }
 
-    if (r > PY_SSIZE_T_MAX/sizeof(Py_ssize_t)) {
-        PyErr_SetString(PyExc_OverflowError, "r is too big");
-        goto error;
-    }
-    indices = PyMem_Malloc(r * sizeof(Py_ssize_t));
+    indices = PyMem_New(Py_ssize_t, r);
     if (indices == NULL) {
         PyErr_NoMemory();
         goto error;
@@ -2713,11 +2708,7 @@
         goto error;
     }
 
-    if (r > PY_SSIZE_T_MAX/sizeof(Py_ssize_t)) {
-        PyErr_SetString(PyExc_OverflowError, "r is too big");
-        goto error;
-    }
-    indices = PyMem_Malloc(r * sizeof(Py_ssize_t));
+    indices = PyMem_New(Py_ssize_t, r);
     if (indices == NULL) {
         PyErr_NoMemory();
         goto error;
@@ -3058,13 +3049,8 @@
         goto error;
     }
 
-    if (n > PY_SSIZE_T_MAX/sizeof(Py_ssize_t) ||
-        r > PY_SSIZE_T_MAX/sizeof(Py_ssize_t)) {
-        PyErr_SetString(PyExc_OverflowError, "parameters too large");
-        goto error;
-    }
-    indices = PyMem_Malloc(n * sizeof(Py_ssize_t));
-    cycles = PyMem_Malloc(r * sizeof(Py_ssize_t));
+    indices = PyMem_New(Py_ssize_t, n);
+    cycles = PyMem_New(Py_ssize_t, r);
     if (indices == NULL || cycles == NULL) {
         PyErr_NoMemory();
         goto error;

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Tue Feb  3 01:05:57 2015
From: python-checkins at python.org (serhiy.storchaka)
Date: Tue, 03 Feb 2015 00:05:57 +0000
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy4zKTogSXNzdWVzICMyMzM2?=
 =?utf-8?q?3=2C_=2323364=2C_=2323365=2C_=2323366=3A_Fixed_itertools_overfl?=
 =?utf-8?q?ow_tests=2E?=
Message-ID: <20150203000515.106264.31815@psf.io>

https://hg.python.org/cpython/rev/356ed025dbae
changeset:   94476:356ed025dbae
branch:      3.3
parent:      94471:5c730d30ffbc
user:        Serhiy Storchaka <storchaka at gmail.com>
date:        Tue Feb 03 01:34:09 2015 +0200
summary:
  Issues #23363, #23364, #23365, #23366: Fixed itertools overflow tests.
Used PyMem_New to check overflow.

files:
  Lib/test/test_itertools.py |  12 +++++-------
  Modules/itertoolsmodule.c  |  26 ++++++--------------------
  2 files changed, 11 insertions(+), 27 deletions(-)


diff --git a/Lib/test/test_itertools.py b/Lib/test/test_itertools.py
--- a/Lib/test/test_itertools.py
+++ b/Lib/test/test_itertools.py
@@ -260,7 +260,7 @@
 
     @support.bigaddrspacetest
     def test_combinations_overflow(self):
-        with self.assertRaises(OverflowError):
+        with self.assertRaises((OverflowError, MemoryError)):
             combinations("AA", 2**29)
 
         # Test implementation detail:  tuple re-use
@@ -346,7 +346,7 @@
 
     @support.bigaddrspacetest
     def test_combinations_with_replacement_overflow(self):
-        with self.assertRaises(OverflowError):
+        with self.assertRaises((OverflowError, MemoryError)):
             combinations_with_replacement("AA", 2**30)
 
         # Test implementation detail:  tuple re-use
@@ -420,10 +420,8 @@
 
     @support.bigaddrspacetest
     def test_permutations_overflow(self):
-        with self.assertRaises(OverflowError):
+        with self.assertRaises((OverflowError, MemoryError)):
             permutations("A", 2**30)
-        with self.assertRaises(OverflowError):
-            permutations("A", 2, 2**30)
 
     @support.impl_detail("tuple resuse is CPython specific")
     def test_permutations_tuple_reuse(self):
@@ -939,8 +937,8 @@
 
     @support.bigaddrspacetest
     def test_product_overflow(self):
-        with self.assertRaises(OverflowError):
-            product(["a"]*(2**16), repeat=2**16)
+        with self.assertRaises((OverflowError, MemoryError)):
+            product(*(['ab']*2**5), repeat=2**25)
 
     @support.impl_detail("tuple reuse is specific to CPython")
     def test_product_tuple_reuse(self):
diff --git a/Modules/itertoolsmodule.c b/Modules/itertoolsmodule.c
--- a/Modules/itertoolsmodule.c
+++ b/Modules/itertoolsmodule.c
@@ -2003,15 +2003,14 @@
         nargs = 0;
     } else {
         nargs = PyTuple_GET_SIZE(args);
-        if (repeat > PY_SSIZE_T_MAX/sizeof(Py_ssize_t) ||
-            nargs > PY_SSIZE_T_MAX/(repeat * sizeof(Py_ssize_t))) {
+        if ((size_t)nargs > PY_SSIZE_T_MAX/sizeof(Py_ssize_t)/repeat) {
             PyErr_SetString(PyExc_OverflowError, "repeat argument too large");
             return NULL;
         }
     }
     npools = nargs * repeat;
 
-    indices = PyMem_Malloc(npools * sizeof(Py_ssize_t));
+    indices = PyMem_New(Py_ssize_t, npools);
     if (indices == NULL) {
         PyErr_NoMemory();
         goto error;
@@ -2335,11 +2334,7 @@
         goto error;
     }
 
-    if (r > PY_SSIZE_T_MAX/sizeof(Py_ssize_t)) {
-        PyErr_SetString(PyExc_OverflowError, "r is too big");
-        goto error;
-    }
-    indices = PyMem_Malloc(r * sizeof(Py_ssize_t));
+    indices = PyMem_New(Py_ssize_t, r);
     if (indices == NULL) {
         PyErr_NoMemory();
         goto error;
@@ -2668,11 +2663,7 @@
         goto error;
     }
 
-    if (r > PY_SSIZE_T_MAX/sizeof(Py_ssize_t)) {
-        PyErr_SetString(PyExc_OverflowError, "r is too big");
-        goto error;
-    }
-    indices = PyMem_Malloc(r * sizeof(Py_ssize_t));
+    indices = PyMem_New(Py_ssize_t, r);
     if (indices == NULL) {
         PyErr_NoMemory();
         goto error;
@@ -3001,13 +2992,8 @@
         goto error;
     }
 
-    if (n > PY_SSIZE_T_MAX/sizeof(Py_ssize_t) ||
-        r > PY_SSIZE_T_MAX/sizeof(Py_ssize_t)) {
-        PyErr_SetString(PyExc_OverflowError, "parameters too large");
-        goto error;
-    }
-    indices = PyMem_Malloc(n * sizeof(Py_ssize_t));
-    cycles = PyMem_Malloc(r * sizeof(Py_ssize_t));
+    indices = PyMem_New(Py_ssize_t, n);
+    cycles = PyMem_New(Py_ssize_t, r);
     if (indices == NULL || cycles == NULL) {
         PyErr_NoMemory();
         goto error;

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Tue Feb  3 01:05:57 2015
From: python-checkins at python.org (serhiy.storchaka)
Date: Tue, 03 Feb 2015 00:05:57 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?q?=29=3A_Issue_=2322896=3A_Avoid_to_use_PyObject=5FAsCharBuffer?=
 =?utf-8?b?KCksIFB5T2JqZWN0X0FzUmVhZEJ1ZmZlcigp?=
Message-ID: <20150203000515.96070.13019@psf.io>

https://hg.python.org/cpython/rev/2e684ce772de
changeset:   94475:2e684ce772de
parent:      94473:651aa21433ba
parent:      94474:1da9630e9b7f
user:        Serhiy Storchaka <storchaka at gmail.com>
date:        Tue Feb 03 01:25:42 2015 +0200
summary:
  Issue #22896: Avoid to use PyObject_AsCharBuffer(), PyObject_AsReadBuffer()
and PyObject_AsWriteBuffer().

files:
  Include/bytes_methods.h            |    2 +-
  Lib/ctypes/test/test_frombuffer.py |   50 ++-
  Misc/NEWS                          |    3 +
  Modules/_codecsmodule.c            |   24 +-
  Modules/_ctypes/_ctypes.c          |   59 ++-
  Modules/_io/bytesio.c              |   10 +-
  Modules/_sqlite/connection.c       |   11 +-
  Modules/_sqlite/statement.c        |   13 +-
  Modules/_struct.c                  |   23 +-
  Objects/abstract.c                 |   82 +---
  Objects/bytearrayobject.c          |  146 ++++-----
  Objects/bytes_methods.c            |   46 +--
  Objects/bytesobject.c              |  272 +++++++++-------
  Objects/complexobject.c            |    9 +-
  Objects/exceptions.c               |   32 +-
  Objects/floatobject.c              |    8 +-
  Objects/stringlib/join.h           |    9 +-
  Python/bltinmodule.c               |   38 +-
  18 files changed, 431 insertions(+), 406 deletions(-)


diff --git a/Include/bytes_methods.h b/Include/bytes_methods.h
--- a/Include/bytes_methods.h
+++ b/Include/bytes_methods.h
@@ -22,7 +22,7 @@
 extern void _Py_bytes_swapcase(char *result, char *s, Py_ssize_t len);
 
 /* The maketrans() static method. */
-extern PyObject* _Py_bytes_maketrans(PyObject *frm, PyObject *to);
+extern PyObject* _Py_bytes_maketrans(Py_buffer *frm, Py_buffer *to);
 
 /* Shared __doc__ strings. */
 extern const char _Py_isspace__doc__[];
diff --git a/Lib/ctypes/test/test_frombuffer.py b/Lib/ctypes/test/test_frombuffer.py
--- a/Lib/ctypes/test/test_frombuffer.py
+++ b/Lib/ctypes/test/test_frombuffer.py
@@ -10,7 +10,7 @@
         self._init_called = True
 
 class Test(unittest.TestCase):
-    def test_fom_buffer(self):
+    def test_from_buffer(self):
         a = array.array("i", range(16))
         x = (c_int * 16).from_buffer(a)
 
@@ -23,25 +23,37 @@
         a[0], a[-1] = 200, -200
         self.assertEqual(x[:], a.tolist())
 
-        self.assertIn(a, x._objects.values())
+        self.assertRaises(BufferError, a.append, 100)
+        self.assertRaises(BufferError, a.pop)
 
-        self.assertRaises(ValueError,
-                          c_int.from_buffer, a, -1)
+        del x; del y; gc.collect(); gc.collect(); gc.collect()
+        a.append(100)
+        a.pop()
+        x = (c_int * 16).from_buffer(a)
+
+        self.assertIn(a, [obj.obj if isinstance(obj, memoryview) else obj
+                          for obj in x._objects.values()])
 
         expected = x[:]
         del a; gc.collect(); gc.collect(); gc.collect()
         self.assertEqual(x[:], expected)
 
-        self.assertRaises(TypeError,
-                          (c_char * 16).from_buffer, "a" * 16)
+        with self.assertRaises(TypeError):
+            (c_char * 16).from_buffer(b"a" * 16)
+        with self.assertRaises(TypeError):
+            (c_char * 16).from_buffer("a" * 16)
 
-    def test_fom_buffer_with_offset(self):
+    def test_from_buffer_with_offset(self):
         a = array.array("i", range(16))
         x = (c_int * 15).from_buffer(a, sizeof(c_int))
 
         self.assertEqual(x[:], a.tolist()[1:])
-        self.assertRaises(ValueError, lambda: (c_int * 16).from_buffer(a, sizeof(c_int)))
-        self.assertRaises(ValueError, lambda: (c_int * 1).from_buffer(a, 16 * sizeof(c_int)))
+        with self.assertRaises(ValueError):
+            c_int.from_buffer(a, -1)
+        with self.assertRaises(ValueError):
+            (c_int * 16).from_buffer(a, sizeof(c_int))
+        with self.assertRaises(ValueError):
+            (c_int * 1).from_buffer(a, 16 * sizeof(c_int))
 
     def test_from_buffer_copy(self):
         a = array.array("i", range(16))
@@ -56,26 +68,30 @@
         a[0], a[-1] = 200, -200
         self.assertEqual(x[:], list(range(16)))
 
+        a.append(100)
+        self.assertEqual(x[:], list(range(16)))
+
         self.assertEqual(x._objects, None)
 
-        self.assertRaises(ValueError,
-                          c_int.from_buffer, a, -1)
-
         del a; gc.collect(); gc.collect(); gc.collect()
         self.assertEqual(x[:], list(range(16)))
 
         x = (c_char * 16).from_buffer_copy(b"a" * 16)
         self.assertEqual(x[:], b"a" * 16)
+        with self.assertRaises(TypeError):
+            (c_char * 16).from_buffer_copy("a" * 16)
 
-    def test_fom_buffer_copy_with_offset(self):
+    def test_from_buffer_copy_with_offset(self):
         a = array.array("i", range(16))
         x = (c_int * 15).from_buffer_copy(a, sizeof(c_int))
 
         self.assertEqual(x[:], a.tolist()[1:])
-        self.assertRaises(ValueError,
-                          (c_int * 16).from_buffer_copy, a, sizeof(c_int))
-        self.assertRaises(ValueError,
-                          (c_int * 1).from_buffer_copy, a, 16 * sizeof(c_int))
+        with self.assertRaises(ValueError):
+            c_int.from_buffer_copy(a, -1)
+        with self.assertRaises(ValueError):
+            (c_int * 16).from_buffer_copy(a, sizeof(c_int))
+        with self.assertRaises(ValueError):
+            (c_int * 1).from_buffer_copy(a, 16 * sizeof(c_int))
 
 if __name__ == '__main__':
     unittest.main()
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -10,6 +10,9 @@
 Core and Builtins
 -----------------
 
+- Issue #22896: Avoid using PyObject_AsCharBuffer(), PyObject_AsReadBuffer()
+  and PyObject_AsWriteBuffer().
+
 - Issue #21295: Revert some changes (issue #16795) to AST line numbers and
   column offsets that constituted a regression.
 
diff --git a/Modules/_codecsmodule.c b/Modules/_codecsmodule.c
--- a/Modules/_codecsmodule.c
+++ b/Modules/_codecsmodule.c
@@ -288,8 +288,6 @@
 {
     PyObject *obj;
     const char *errors = NULL;
-    const char *data;
-    Py_ssize_t size;
 
     if (!PyArg_ParseTuple(args, "O|z:unicode_internal_decode",
                           &obj, &errors))
@@ -302,11 +300,16 @@
         return codec_tuple(obj, PyUnicode_GET_LENGTH(obj));
     }
     else {
-        if (PyObject_AsReadBuffer(obj, (const void **)&data, &size))
+        Py_buffer view;
+        PyObject *result;
+        if (PyObject_GetBuffer(obj, &view, PyBUF_SIMPLE) != 0)
             return NULL;
 
-        return codec_tuple(_PyUnicode_DecodeUnicodeInternal(data, size, errors),
-                           size);
+        result = codec_tuple(
+                _PyUnicode_DecodeUnicodeInternal(view.buf, view.len, errors),
+                view.len);
+        PyBuffer_Release(&view);
+        return result;
     }
 }
 
@@ -731,8 +734,6 @@
 {
     PyObject *obj;
     const char *errors = NULL;
-    const char *data;
-    Py_ssize_t len, size;
 
     if (PyErr_WarnEx(PyExc_DeprecationWarning,
                      "unicode_internal codec has been deprecated",
@@ -745,6 +746,7 @@
 
     if (PyUnicode_Check(obj)) {
         Py_UNICODE *u;
+        Py_ssize_t len, size;
 
         if (PyUnicode_READY(obj) < 0)
             return NULL;
@@ -759,9 +761,13 @@
                            PyUnicode_GET_LENGTH(obj));
     }
     else {
-        if (PyObject_AsReadBuffer(obj, (const void **)&data, &size))
+        Py_buffer view;
+        PyObject *result;
+        if (PyObject_GetBuffer(obj, &view, PyBUF_SIMPLE) != 0)
             return NULL;
-        return codec_tuple(PyBytes_FromStringAndSize(data, size), size);
+        result = codec_tuple(PyBytes_FromStringAndSize(view.buf, view.len), view.len);
+        PyBuffer_Release(&view);
+        return result;
     }
 }
 
diff --git a/Modules/_ctypes/_ctypes.c b/Modules/_ctypes/_ctypes.c
--- a/Modules/_ctypes/_ctypes.c
+++ b/Modules/_ctypes/_ctypes.c
@@ -463,39 +463,45 @@
 static PyObject *
 CDataType_from_buffer(PyObject *type, PyObject *args)
 {
-    void *buffer;
-    Py_ssize_t buffer_len;
+    Py_buffer buffer;
     Py_ssize_t offset = 0;
-    PyObject *obj, *result;
+    PyObject *result, *mv;
     StgDictObject *dict = PyType_stgdict(type);
     assert (dict);
 
-    if (!PyArg_ParseTuple(args, "O|n:from_buffer", &obj, &offset))
-        return NULL;
-
-    if (-1 == PyObject_AsWriteBuffer(obj, &buffer, &buffer_len))
+    if (!PyArg_ParseTuple(args, "w*|n:from_buffer", &buffer, &offset))
         return NULL;
 
     if (offset < 0) {
         PyErr_SetString(PyExc_ValueError,
                         "offset cannot be negative");
+        PyBuffer_Release(&buffer);
         return NULL;
     }
-    if (dict->size > buffer_len - offset) {
+    if (dict->size > buffer.len - offset) {
         PyErr_Format(PyExc_ValueError,
                      "Buffer size too small (%zd instead of at least %zd bytes)",
-                     buffer_len, dict->size + offset);
+                     buffer.len, dict->size + offset);
+        PyBuffer_Release(&buffer);
         return NULL;
     }
 
-    result = PyCData_AtAddress(type, (char *)buffer + offset);
-    if (result == NULL)
+    result = PyCData_AtAddress(type, (char *)buffer.buf + offset);
+    if (result == NULL) {
+        PyBuffer_Release(&buffer);
         return NULL;
-
-    Py_INCREF(obj);
-    if (-1 == KeepRef((CDataObject *)result, -1, obj)) {
+    }
+
+    mv = PyMemoryView_FromBuffer(&buffer);
+    if (mv == NULL) {
+        PyBuffer_Release(&buffer);
         return NULL;
     }
+    /* Hack the memoryview so that it will release the buffer. */
+    ((PyMemoryViewObject *)mv)->mbuf->master.obj = buffer.obj;
+    ((PyMemoryViewObject *)mv)->view.obj = buffer.obj;
+    if (-1 == KeepRef((CDataObject *)result, -1, mv))
+        result = NULL;
     return result;
 }
 
@@ -508,37 +514,36 @@
 static PyObject *
 CDataType_from_buffer_copy(PyObject *type, PyObject *args)
 {
-    const void *buffer;
-    Py_ssize_t buffer_len;
+    Py_buffer buffer;
     Py_ssize_t offset = 0;
-    PyObject *obj, *result;
+    PyObject *result;
     StgDictObject *dict = PyType_stgdict(type);
     assert (dict);
 
-    if (!PyArg_ParseTuple(args, "O|n:from_buffer", &obj, &offset))
-        return NULL;
-
-    if (-1 == PyObject_AsReadBuffer(obj, (const void**)&buffer, &buffer_len))
+    if (!PyArg_ParseTuple(args, "y*|n:from_buffer", &buffer, &offset))
         return NULL;
 
     if (offset < 0) {
         PyErr_SetString(PyExc_ValueError,
                         "offset cannot be negative");
+        PyBuffer_Release(&buffer);
         return NULL;
     }
 
-    if (dict->size > buffer_len - offset) {
+    if (dict->size > buffer.len - offset) {
         PyErr_Format(PyExc_ValueError,
                      "Buffer size too small (%zd instead of at least %zd bytes)",
-                     buffer_len, dict->size + offset);
+                     buffer.len, dict->size + offset);
+        PyBuffer_Release(&buffer);
         return NULL;
     }
 
     result = GenericPyCData_new((PyTypeObject *)type, NULL, NULL);
-    if (result == NULL)
-        return NULL;
-    memcpy(((CDataObject *)result)->b_ptr,
-           (char *)buffer+offset, dict->size);
+    if (result != NULL) {
+        memcpy(((CDataObject *)result)->b_ptr,
+               (char *)buffer.buf + offset, dict->size);
+    }
+    PyBuffer_Release(&buffer);
     return result;
 }
 
diff --git a/Modules/_io/bytesio.c b/Modules/_io/bytesio.c
--- a/Modules/_io/bytesio.c
+++ b/Modules/_io/bytesio.c
@@ -553,17 +553,18 @@
 "is set not to block as has no data to read.");
 
 static PyObject *
-bytesio_readinto(bytesio *self, PyObject *buffer)
+bytesio_readinto(bytesio *self, PyObject *arg)
 {
-    void *raw_buffer;
+    Py_buffer buffer;
     Py_ssize_t len, n;
 
     CHECK_CLOSED(self, NULL);
 
-    if (PyObject_AsWriteBuffer(buffer, &raw_buffer, &len) == -1)
+    if (!PyArg_Parse(arg, "w*", &buffer))
         return NULL;
 
     /* adjust invalid sizes */
+    len = buffer.len;
     n = self->string_size - self->pos;
     if (len > n) {
         len = n;
@@ -571,10 +572,11 @@
             len = 0;
     }
 
-    memcpy(raw_buffer, self->buf + self->pos, len);
+    memcpy(buffer.buf, self->buf + self->pos, len);
     assert(self->pos + len < PY_SSIZE_T_MAX);
     assert(len >= 0);
     self->pos += len;
+    PyBuffer_Release(&buffer);
 
     return PyLong_FromSsize_t(len);
 }
diff --git a/Modules/_sqlite/connection.c b/Modules/_sqlite/connection.c
--- a/Modules/_sqlite/connection.c
+++ b/Modules/_sqlite/connection.c
@@ -522,19 +522,20 @@
             return -1;
         sqlite3_result_text(context, str, -1, SQLITE_TRANSIENT);
     } else if (PyObject_CheckBuffer(py_val)) {
-        const char* buffer;
-        Py_ssize_t buflen;
-        if (PyObject_AsCharBuffer(py_val, &buffer, &buflen) != 0) {
+        Py_buffer view;
+        if (PyObject_GetBuffer(py_val, &view, PyBUF_SIMPLE) != 0) {
             PyErr_SetString(PyExc_ValueError,
                             "could not convert BLOB to buffer");
             return -1;
         }
-        if (buflen > INT_MAX) {
+        if (view.len > INT_MAX) {
             PyErr_SetString(PyExc_OverflowError,
                             "BLOB longer than INT_MAX bytes");
+            PyBuffer_Release(&view);
             return -1;
         }
-        sqlite3_result_blob(context, buffer, (int)buflen, SQLITE_TRANSIENT);
+        sqlite3_result_blob(context, view.buf, (int)view.len, SQLITE_TRANSIENT);
+        PyBuffer_Release(&view);
     } else {
         return -1;
     }
diff --git a/Modules/_sqlite/statement.c b/Modules/_sqlite/statement.c
--- a/Modules/_sqlite/statement.c
+++ b/Modules/_sqlite/statement.c
@@ -94,7 +94,6 @@
 int pysqlite_statement_bind_parameter(pysqlite_Statement* self, int pos, PyObject* parameter)
 {
     int rc = SQLITE_OK;
-    const char* buffer;
     char* string;
     Py_ssize_t buflen;
     parameter_type paramtype;
@@ -145,18 +144,22 @@
             }
             rc = sqlite3_bind_text(self->st, pos, string, (int)buflen, SQLITE_TRANSIENT);
             break;
-        case TYPE_BUFFER:
-            if (PyObject_AsCharBuffer(parameter, &buffer, &buflen) != 0) {
+        case TYPE_BUFFER: {
+            Py_buffer view;
+            if (PyObject_GetBuffer(parameter, &view, PyBUF_SIMPLE) != 0) {
                 PyErr_SetString(PyExc_ValueError, "could not convert BLOB to buffer");
                 return -1;
             }
-            if (buflen > INT_MAX) {
+            if (view.len > INT_MAX) {
                 PyErr_SetString(PyExc_OverflowError,
                                 "BLOB longer than INT_MAX bytes");
+                PyBuffer_Release(&view);
                 return -1;
             }
-            rc = sqlite3_bind_blob(self->st, pos, buffer, buflen, SQLITE_TRANSIENT);
+            rc = sqlite3_bind_blob(self->st, pos, view.buf, (int)view.len, SQLITE_TRANSIENT);
+            PyBuffer_Release(&view);
             break;
+        }
         case TYPE_UNKNOWN:
             rc = -1;
     }
diff --git a/Modules/_struct.c b/Modules/_struct.c
--- a/Modules/_struct.c
+++ b/Modules/_struct.c
@@ -1842,8 +1842,8 @@
 s_pack_into(PyObject *self, PyObject *args)
 {
     PyStructObject *soself;
-    char *buffer;
-    Py_ssize_t buffer_len, offset;
+    Py_buffer buffer;
+    Py_ssize_t offset;
 
     /* Validate arguments.  +1 is for the first arg as buffer. */
     soself = (PyStructObject *)self;
@@ -1868,34 +1868,37 @@
     }
 
     /* Extract a writable memory buffer from the first argument */
-    if ( PyObject_AsWriteBuffer(PyTuple_GET_ITEM(args, 0),
-                                                            (void**)&buffer, &buffer_len) == -1 ) {
+    if (!PyArg_Parse(PyTuple_GET_ITEM(args, 0), "w*", &buffer))
         return NULL;
-    }
-    assert( buffer_len >= 0 );
+    assert(buffer.len >= 0);
 
     /* Extract the offset from the first argument */
     offset = PyNumber_AsSsize_t(PyTuple_GET_ITEM(args, 1), PyExc_IndexError);
-    if (offset == -1 && PyErr_Occurred())
+    if (offset == -1 && PyErr_Occurred()) {
+        PyBuffer_Release(&buffer);
         return NULL;
+    }
 
     /* Support negative offsets. */
     if (offset < 0)
-        offset += buffer_len;
+        offset += buffer.len;
 
     /* Check boundaries */
-    if (offset < 0 || (buffer_len - offset) < soself->s_size) {
+    if (offset < 0 || (buffer.len - offset) < soself->s_size) {
         PyErr_Format(StructError,
                      "pack_into requires a buffer of at least %zd bytes",
                      soself->s_size);
+        PyBuffer_Release(&buffer);
         return NULL;
     }
 
     /* Call the guts */
-    if ( s_pack_internal(soself, args, 2, buffer + offset) != 0 ) {
+    if (s_pack_internal(soself, args, 2, (char*)buffer.buf + offset) != 0) {
+        PyBuffer_Release(&buffer);
         return NULL;
     }
 
+    PyBuffer_Release(&buffer);
     Py_RETURN_NONE;
 }
 
diff --git a/Objects/abstract.c b/Objects/abstract.c
--- a/Objects/abstract.c
+++ b/Objects/abstract.c
@@ -250,27 +250,7 @@
                       const char **buffer,
                       Py_ssize_t *buffer_len)
 {
-    PyBufferProcs *pb;
-    Py_buffer view;
-
-    if (obj == NULL || buffer == NULL || buffer_len == NULL) {
-        null_error();
-        return -1;
-    }
-    pb = obj->ob_type->tp_as_buffer;
-    if (pb == NULL || pb->bf_getbuffer == NULL) {
-        PyErr_SetString(PyExc_TypeError,
-                        "expected a bytes-like object");
-        return -1;
-    }
-    if ((*pb->bf_getbuffer)(obj, &view, PyBUF_SIMPLE)) return -1;
-
-    *buffer = view.buf;
-    *buffer_len = view.len;
-    if (pb->bf_releasebuffer != NULL)
-        (*pb->bf_releasebuffer)(obj, &view);
-    Py_XDECREF(view.obj);
-    return 0;
+    return PyObject_AsReadBuffer(obj, (const void **)buffer, buffer_len);
 }
 
 int
@@ -294,28 +274,18 @@
                           const void **buffer,
                           Py_ssize_t *buffer_len)
 {
-    PyBufferProcs *pb;
     Py_buffer view;
 
     if (obj == NULL || buffer == NULL || buffer_len == NULL) {
         null_error();
         return -1;
     }
-    pb = obj->ob_type->tp_as_buffer;
-    if (pb == NULL ||
-        pb->bf_getbuffer == NULL) {
-        PyErr_SetString(PyExc_TypeError,
-                        "expected a bytes-like object");
+    if (PyObject_GetBuffer(obj, &view, PyBUF_SIMPLE) != 0)
         return -1;
-    }
-
-    if ((*pb->bf_getbuffer)(obj, &view, PyBUF_SIMPLE)) return -1;
 
     *buffer = view.buf;
     *buffer_len = view.len;
-    if (pb->bf_releasebuffer != NULL)
-        (*pb->bf_releasebuffer)(obj, &view);
-    Py_XDECREF(view.obj);
+    PyBuffer_Release(&view);
     return 0;
 }
 
@@ -341,9 +311,7 @@
 
     *buffer = view.buf;
     *buffer_len = view.len;
-    if (pb->bf_releasebuffer != NULL)
-        (*pb->bf_releasebuffer)(obj, &view);
-    Py_XDECREF(view.obj);
+    PyBuffer_Release(&view);
     return 0;
 }
 
@@ -352,13 +320,15 @@
 int
 PyObject_GetBuffer(PyObject *obj, Py_buffer *view, int flags)
 {
-    if (!PyObject_CheckBuffer(obj)) {
+    PyBufferProcs *pb = obj->ob_type->tp_as_buffer;
+
+    if (pb == NULL || pb->bf_getbuffer == NULL) {
         PyErr_Format(PyExc_TypeError,
                      "a bytes-like object is required, not '%.100s'",
                      Py_TYPE(obj)->tp_name);
         return -1;
     }
-    return (*(obj->ob_type->tp_as_buffer->bf_getbuffer))(obj, view, flags);
+    return (*pb->bf_getbuffer)(obj, view, flags);
 }
 
 static int
@@ -676,10 +646,14 @@
 PyBuffer_Release(Py_buffer *view)
 {
     PyObject *obj = view->obj;
-    if (obj && Py_TYPE(obj)->tp_as_buffer && Py_TYPE(obj)->tp_as_buffer->bf_releasebuffer)
-        Py_TYPE(obj)->tp_as_buffer->bf_releasebuffer(obj, view);
-    Py_XDECREF(obj);
+    PyBufferProcs *pb;
+    if (obj == NULL)
+        return;
+    pb = Py_TYPE(obj)->tp_as_buffer;
+    if (pb && pb->bf_releasebuffer)
+        pb->bf_releasebuffer(obj, view);
     view->obj = NULL;
+    Py_DECREF(obj);
 }
 
 PyObject *
@@ -1288,8 +1262,7 @@
 {
     PyNumberMethods *m;
     PyObject *trunc_func;
-    const char *buffer;
-    Py_ssize_t buffer_len;
+    Py_buffer view;
     _Py_IDENTIFIER(__trunc__);
 
     if (o == NULL)
@@ -1327,21 +1300,22 @@
     if (PyErr_Occurred())
         return NULL;
 
-    if (PyBytes_Check(o))
+    if (PyUnicode_Check(o))
+        /* The below check is done in PyLong_FromUnicode(). */
+        return PyLong_FromUnicodeObject(o, 10);
+
+    if (PyObject_GetBuffer(o, &view, PyBUF_SIMPLE) == 0) {
         /* need to do extra error checking that PyLong_FromString()
          * doesn't do.  In particular int('9\x005') must raise an
          * exception, not truncate at the null.
          */
-        return _PyLong_FromBytes(PyBytes_AS_STRING(o),
-                                 PyBytes_GET_SIZE(o), 10);
-    if (PyUnicode_Check(o))
-        /* The above check is done in PyLong_FromUnicode(). */
-        return PyLong_FromUnicodeObject(o, 10);
-    if (!PyObject_AsCharBuffer(o, &buffer, &buffer_len))
-        return _PyLong_FromBytes(buffer, buffer_len, 10);
-
-    return type_error("int() argument must be a string or a "
-                      "number, not '%.200s'", o);
+        PyObject *result = _PyLong_FromBytes(view.buf, view.len, 10);
+        PyBuffer_Release(&view);
+        return result;
+    }
+
+    return type_error("int() argument must be a string, a bytes-like object "
+                      "or a number, not '%.200s'", o);
 }
 
 PyObject *
diff --git a/Objects/bytearrayobject.c b/Objects/bytearrayobject.c
--- a/Objects/bytearrayobject.c
+++ b/Objects/bytearrayobject.c
@@ -80,24 +80,6 @@
     obj->ob_exports--;
 }
 
-static Py_ssize_t
-_getbuffer(PyObject *obj, Py_buffer *view)
-{
-    PyBufferProcs *buffer = Py_TYPE(obj)->tp_as_buffer;
-
-    if (buffer == NULL || buffer->bf_getbuffer == NULL)
-    {
-        PyErr_Format(PyExc_TypeError,
-                     "a bytes-like object is required, not '%.100s'",
-                     Py_TYPE(obj)->tp_name);
-        return -1;
-    }
-
-    if (buffer->bf_getbuffer(obj, view, PyBUF_SIMPLE) < 0)
-            return -1;
-    return view->len;
-}
-
 static int
 _canresize(PyByteArrayObject *self)
 {
@@ -268,8 +250,8 @@
 
     va.len = -1;
     vb.len = -1;
-    if (_getbuffer(a, &va) < 0  ||
-        _getbuffer(b, &vb) < 0) {
+    if (PyObject_GetBuffer(a, &va, PyBUF_SIMPLE) != 0 ||
+        PyObject_GetBuffer(b, &vb, PyBUF_SIMPLE) != 0) {
             PyErr_Format(PyExc_TypeError, "can't concat %.100s to %.100s",
                          Py_TYPE(a)->tp_name, Py_TYPE(b)->tp_name);
             goto done;
@@ -335,7 +317,7 @@
     Py_ssize_t size;
     Py_buffer vo;
 
-    if (_getbuffer(other, &vo) < 0) {
+    if (PyObject_GetBuffer(other, &vo, PyBUF_SIMPLE) != 0) {
         PyErr_Format(PyExc_TypeError, "can't concat %.100s to %.100s",
                      Py_TYPE(other)->tp_name, Py_TYPE(self)->tp_name);
         return NULL;
@@ -595,14 +577,14 @@
         needed = 0;
     }
     else {
-            if (_getbuffer(values, &vbytes) < 0) {
-                    PyErr_Format(PyExc_TypeError,
-                                 "can't set bytearray slice from %.100s",
-                                 Py_TYPE(values)->tp_name);
-                    return -1;
-            }
-            needed = vbytes.len;
-            bytes = vbytes.buf;
+        if (PyObject_GetBuffer(values, &vbytes, PyBUF_SIMPLE) != 0) {
+            PyErr_Format(PyExc_TypeError,
+                         "can't set bytearray slice from %.100s",
+                         Py_TYPE(values)->tp_name);
+            return -1;
+        }
+        needed = vbytes.len;
+        bytes = vbytes.buf;
     }
 
     if (lo < 0)
@@ -1047,18 +1029,18 @@
         Py_RETURN_NOTIMPLEMENTED;
     }
 
-    self_size = _getbuffer(self, &self_bytes);
-    if (self_size < 0) {
+    if (PyObject_GetBuffer(self, &self_bytes, PyBUF_SIMPLE) != 0) {
         PyErr_Clear();
         Py_RETURN_NOTIMPLEMENTED;
     }
-
-    other_size = _getbuffer(other, &other_bytes);
-    if (other_size < 0) {
+    self_size = self_bytes.len;
+
+    if (PyObject_GetBuffer(other, &other_bytes, PyBUF_SIMPLE) != 0) {
         PyErr_Clear();
         PyBuffer_Release(&self_bytes);
         Py_RETURN_NOTIMPLEMENTED;
     }
+    other_size = other_bytes.len;
 
     if (self_size != other_size && (op == Py_EQ || op == Py_NE)) {
         /* Shortcut: if the lengths differ, the objects differ */
@@ -1170,7 +1152,7 @@
         return -2;
 
     if (subobj) {
-        if (_getbuffer(subobj, &subbuf) < 0)
+        if (PyObject_GetBuffer(subobj, &subbuf, PyBUF_SIMPLE) != 0)
             return -2;
 
         sub = subbuf.buf;
@@ -1238,7 +1220,7 @@
         return NULL;
 
     if (sub_obj) {
-        if (_getbuffer(sub_obj, &vsub) < 0)
+        if (PyObject_GetBuffer(sub_obj, &vsub, PyBUF_SIMPLE) != 0)
             return NULL;
 
         sub = vsub.buf;
@@ -1397,7 +1379,7 @@
         Py_buffer varg;
         Py_ssize_t pos;
         PyErr_Clear();
-        if (_getbuffer(arg, &varg) < 0)
+        if (PyObject_GetBuffer(arg, &varg, PyBUF_SIMPLE) != 0)
             return -1;
         pos = stringlib_find(PyByteArray_AS_STRING(self), Py_SIZE(self),
                              varg.buf, varg.len, 0);
@@ -1428,7 +1410,7 @@
 
     str = PyByteArray_AS_STRING(self);
 
-    if (_getbuffer(substr, &vsubstr) < 0)
+    if (PyObject_GetBuffer(substr, &vsubstr, PyBUF_SIMPLE) != 0)
         return -1;
 
     ADJUST_INDICES(start, end, len);
@@ -1621,7 +1603,7 @@
     if (table == Py_None) {
         table_chars = NULL;
         table = NULL;
-    } else if (_getbuffer(table, &vtable) < 0) {
+    } else if (PyObject_GetBuffer(table, &vtable, PyBUF_SIMPLE) != 0) {
         return NULL;
     } else {
         if (vtable.len != 256) {
@@ -1634,7 +1616,7 @@
     }
 
     if (deletechars != NULL) {
-        if (_getbuffer(deletechars, &vdel) < 0) {
+        if (PyObject_GetBuffer(deletechars, &vdel, PyBUF_SIMPLE) != 0) {
             if (table != NULL)
                 PyBuffer_Release(&vtable);
             return NULL;
@@ -1699,8 +1681,8 @@
 @staticmethod
 bytearray.maketrans
 
-    frm: object
-    to: object
+    frm: Py_buffer
+    to: Py_buffer
     /
 
 Return a translation table useable for the bytes or bytearray translate method.
@@ -1726,28 +1708,35 @@
     {"maketrans", (PyCFunction)bytearray_maketrans, METH_VARARGS|METH_STATIC, bytearray_maketrans__doc__},
 
 static PyObject *
-bytearray_maketrans_impl(PyObject *frm, PyObject *to);
+bytearray_maketrans_impl(Py_buffer *frm, Py_buffer *to);
 
 static PyObject *
 bytearray_maketrans(void *null, PyObject *args)
 {
     PyObject *return_value = NULL;
-    PyObject *frm;
-    PyObject *to;
-
-    if (!PyArg_UnpackTuple(args, "maketrans",
-        2, 2,
+    Py_buffer frm = {NULL, NULL};
+    Py_buffer to = {NULL, NULL};
+
+    if (!PyArg_ParseTuple(args,
+        "y*y*:maketrans",
         &frm, &to))
         goto exit;
-    return_value = bytearray_maketrans_impl(frm, to);
+    return_value = bytearray_maketrans_impl(&frm, &to);
 
 exit:
+    /* Cleanup for frm */
+    if (frm.obj)
+       PyBuffer_Release(&frm);
+    /* Cleanup for to */
+    if (to.obj)
+       PyBuffer_Release(&to);
+
     return return_value;
 }
 
 static PyObject *
-bytearray_maketrans_impl(PyObject *frm, PyObject *to)
-/*[clinic end generated code: output=307752019d9b25b5 input=ea9bdc6b328c15e2]*/
+bytearray_maketrans_impl(Py_buffer *frm, Py_buffer *to)
+/*[clinic end generated code: output=d332622814c26f4b input=5925a81d2fbbf151]*/
 {
     return _Py_bytes_maketrans(frm, to);
 }
@@ -2243,8 +2232,8 @@
 /*[clinic input]
 bytearray.replace
 
-    old: object
-    new: object
+    old: Py_buffer
+    new: Py_buffer
     count: Py_ssize_t = -1
         Maximum number of occurrences to replace.
         -1 (the default value) means replace all occurrences.
@@ -2273,47 +2262,40 @@
     {"replace", (PyCFunction)bytearray_replace, METH_VARARGS, bytearray_replace__doc__},
 
 static PyObject *
-bytearray_replace_impl(PyByteArrayObject *self, PyObject *old, PyObject *new, Py_ssize_t count);
+bytearray_replace_impl(PyByteArrayObject *self, Py_buffer *old, Py_buffer *new, Py_ssize_t count);
 
 static PyObject *
 bytearray_replace(PyByteArrayObject *self, PyObject *args)
 {
     PyObject *return_value = NULL;
-    PyObject *old;
-    PyObject *new;
+    Py_buffer old = {NULL, NULL};
+    Py_buffer new = {NULL, NULL};
     Py_ssize_t count = -1;
 
     if (!PyArg_ParseTuple(args,
-        "OO|n:replace",
+        "y*y*|n:replace",
         &old, &new, &count))
         goto exit;
-    return_value = bytearray_replace_impl(self, old, new, count);
+    return_value = bytearray_replace_impl(self, &old, &new, count);
 
 exit:
+    /* Cleanup for old */
+    if (old.obj)
+       PyBuffer_Release(&old);
+    /* Cleanup for new */
+    if (new.obj)
+       PyBuffer_Release(&new);
+
     return return_value;
 }
 
 static PyObject *
-bytearray_replace_impl(PyByteArrayObject *self, PyObject *old, PyObject *new, Py_ssize_t count)
-/*[clinic end generated code: output=4d2e3c9130da0f96 input=9aaaa123608dfc1f]*/
+bytearray_replace_impl(PyByteArrayObject *self, Py_buffer *old, Py_buffer *new, Py_ssize_t count)
+/*[clinic end generated code: output=9997fbbd5bac4883 input=aa379d988637c7fb]*/
 {
-    PyObject *res;
-    Py_buffer vold, vnew;
-
-    if (_getbuffer(old, &vold) < 0)
-        return NULL;
-    if (_getbuffer(new, &vnew) < 0) {
-        PyBuffer_Release(&vold);
-        return NULL;
-    }
-
-    res = (PyObject *)replace((PyByteArrayObject *) self,
-                              vold.buf, vold.len,
-                              vnew.buf, vnew.len, count);
-
-    PyBuffer_Release(&vold);
-    PyBuffer_Release(&vnew);
-    return res;
+    return (PyObject *)replace((PyByteArrayObject *) self,
+                               old->buf, old->len,
+                               new->buf, new->len, count);
 }
 
 /*[clinic input]
@@ -2383,7 +2365,7 @@
     if (sep == Py_None)
         return stringlib_split_whitespace((PyObject*) self, s, len, maxsplit);
 
-    if (_getbuffer(sep, &vsub) < 0)
+    if (PyObject_GetBuffer(sep, &vsub, PyBUF_SIMPLE) != 0)
         return NULL;
     sub = vsub.buf;
     n = vsub.len;
@@ -2566,7 +2548,7 @@
     if (sep == Py_None)
         return stringlib_rsplit_whitespace((PyObject*) self, s, len, maxsplit);
 
-    if (_getbuffer(sep, &vsub) < 0)
+    if (PyObject_GetBuffer(sep, &vsub, PyBUF_SIMPLE) != 0)
         return NULL;
     sub = vsub.buf;
     n = vsub.len;
@@ -3088,7 +3070,7 @@
         byteslen = 6;
     }
     else {
-        if (_getbuffer(bytes, &vbytes) < 0)
+        if (PyObject_GetBuffer(bytes, &vbytes, PyBUF_SIMPLE) != 0)
             return NULL;
         bytesptr = (char *) vbytes.buf;
         byteslen = vbytes.len;
@@ -3159,7 +3141,7 @@
         byteslen = 6;
     }
     else {
-        if (_getbuffer(bytes, &vbytes) < 0)
+        if (PyObject_GetBuffer(bytes, &vbytes, PyBUF_SIMPLE) != 0)
             return NULL;
         bytesptr = (char *) vbytes.buf;
         byteslen = vbytes.len;
@@ -3227,7 +3209,7 @@
         byteslen = 6;
     }
     else {
-        if (_getbuffer(bytes, &vbytes) < 0)
+        if (PyObject_GetBuffer(bytes, &vbytes, PyBUF_SIMPLE) != 0)
             return NULL;
         bytesptr = (char *) vbytes.buf;
         byteslen = vbytes.len;
diff --git a/Objects/bytes_methods.c b/Objects/bytes_methods.c
--- a/Objects/bytes_methods.c
+++ b/Objects/bytes_methods.c
@@ -363,59 +363,27 @@
 in frm is mapped to the byte at the same position in to.\n\
 The bytes objects frm and to must be of the same length.");
 
-static Py_ssize_t
-_getbuffer(PyObject *obj, Py_buffer *view)
-{
-    PyBufferProcs *buffer = Py_TYPE(obj)->tp_as_buffer;
-
-    if (buffer == NULL || buffer->bf_getbuffer == NULL)
-    {
-        PyErr_Format(PyExc_TypeError,
-                     "a bytes-like object is required, not '%.100s'",
-                     Py_TYPE(obj)->tp_name);
-        return -1;
-    }
-
-    if (buffer->bf_getbuffer(obj, view, PyBUF_SIMPLE) < 0)
-        return -1;
-    return view->len;
-}
-
 PyObject *
-_Py_bytes_maketrans(PyObject *frm, PyObject *to)
+_Py_bytes_maketrans(Py_buffer *frm, Py_buffer *to)
 {
     PyObject *res = NULL;
-    Py_buffer bfrm, bto;
     Py_ssize_t i;
     char *p;
 
-    bfrm.len = -1;
-    bto.len = -1;
-
-    if (_getbuffer(frm, &bfrm) < 0)
-        return NULL;
-    if (_getbuffer(to, &bto) < 0)
-        goto done;
-    if (bfrm.len != bto.len) {
+    if (frm->len != to->len) {
         PyErr_Format(PyExc_ValueError,
                      "maketrans arguments must have same length");
-        goto done;
+        return NULL;
     }
     res = PyBytes_FromStringAndSize(NULL, 256);
-    if (!res) {
-        goto done;
-    }
+    if (!res)
+        return NULL;
     p = PyBytes_AS_STRING(res);
     for (i = 0; i < 256; i++)
         p[i] = (char) i;
-    for (i = 0; i < bfrm.len; i++) {
-        p[((unsigned char *)bfrm.buf)[i]] = ((char *)bto.buf)[i];
+    for (i = 0; i < frm->len; i++) {
+        p[((unsigned char *)frm->buf)[i]] = ((char *)to->buf)[i];
     }
 
-done:
-    if (bfrm.len != -1)
-        PyBuffer_Release(&bfrm);
-    if (bto.len != -1)
-        PyBuffer_Release(&bto);
     return res;
 }
diff --git a/Objects/bytesobject.c b/Objects/bytesobject.c
--- a/Objects/bytesobject.c
+++ b/Objects/bytesobject.c
@@ -12,33 +12,6 @@
 [clinic start generated code]*/
 /*[clinic end generated code: output=da39a3ee5e6b4b0d input=1a1d9102afc1b00c]*/
 
-static Py_ssize_t
-_getbuffer(PyObject *obj, Py_buffer *view)
-{
-    PyBufferProcs *bufferprocs;
-    if (PyBytes_CheckExact(obj)) {
-        /* Fast path, e.g. for .join() of many bytes objects */
-        Py_INCREF(obj);
-        view->obj = obj;
-        view->buf = PyBytes_AS_STRING(obj);
-        view->len = PyBytes_GET_SIZE(obj);
-        return view->len;
-    }
-
-    bufferprocs = Py_TYPE(obj)->tp_as_buffer;
-    if (bufferprocs == NULL || bufferprocs->bf_getbuffer == NULL)
-    {
-        PyErr_Format(PyExc_TypeError,
-                     "a bytes-like object is required, not '%.100s'",
-                     Py_TYPE(obj)->tp_name);
-        return -1;
-    }
-
-    if (bufferprocs->bf_getbuffer(obj, view, PyBUF_SIMPLE) < 0)
-        return -1;
-    return view->len;
-}
-
 #ifdef COUNT_ALLOCS
 Py_ssize_t null_strings, one_strings;
 #endif
@@ -1349,8 +1322,8 @@
 
     va.len = -1;
     vb.len = -1;
-    if (_getbuffer(a, &va) < 0  ||
-        _getbuffer(b, &vb) < 0) {
+    if (PyObject_GetBuffer(a, &va, PyBUF_SIMPLE) != 0 ||
+        PyObject_GetBuffer(b, &vb, PyBUF_SIMPLE) != 0) {
         PyErr_Format(PyExc_TypeError, "can't concat %.100s to %.100s",
                      Py_TYPE(a)->tp_name, Py_TYPE(b)->tp_name);
         goto done;
@@ -1448,7 +1421,7 @@
         Py_buffer varg;
         Py_ssize_t pos;
         PyErr_Clear();
-        if (_getbuffer(arg, &varg) < 0)
+        if (PyObject_GetBuffer(arg, &varg, PyBUF_SIMPLE) != 0)
             return -1;
         pos = stringlib_find(PyBytes_AS_STRING(self), Py_SIZE(self),
                              varg.buf, varg.len, 0);
@@ -1737,7 +1710,7 @@
         maxsplit = PY_SSIZE_T_MAX;
     if (sep == Py_None)
         return stringlib_split_whitespace((PyObject*) self, s, len, maxsplit);
-    if (_getbuffer(sep, &vsub) < 0)
+    if (PyObject_GetBuffer(sep, &vsub, PyBUF_SIMPLE) != 0)
         return NULL;
     sub = vsub.buf;
     n = vsub.len;
@@ -1751,7 +1724,7 @@
 bytes.partition
 
     self: self(type="PyBytesObject *")
-    sep: object
+    sep: Py_buffer
     /
 
 Partition the bytes into three parts using the given separator.
@@ -1778,26 +1751,39 @@
 "object and two empty bytes objects.");
 
 #define BYTES_PARTITION_METHODDEF    \
-    {"partition", (PyCFunction)bytes_partition, METH_O, bytes_partition__doc__},
+    {"partition", (PyCFunction)bytes_partition, METH_VARARGS, bytes_partition__doc__},
 
 static PyObject *
-bytes_partition(PyBytesObject *self, PyObject *sep)
-/*[clinic end generated code: output=b41e119c873c08bc input=6c5b9dcc5a9fd62e]*/
+bytes_partition_impl(PyBytesObject *self, Py_buffer *sep);
+
+static PyObject *
+bytes_partition(PyBytesObject *self, PyObject *args)
 {
-    const char *sep_chars;
-    Py_ssize_t sep_len;
-
-    if (PyBytes_Check(sep)) {
-        sep_chars = PyBytes_AS_STRING(sep);
-        sep_len = PyBytes_GET_SIZE(sep);
-    }
-    else if (PyObject_AsCharBuffer(sep, &sep_chars, &sep_len))
-        return NULL;
-
+    PyObject *return_value = NULL;
+    Py_buffer sep = {NULL, NULL};
+
+    if (!PyArg_ParseTuple(args,
+        "y*:partition",
+        &sep))
+        goto exit;
+    return_value = bytes_partition_impl(self, &sep);
+
+exit:
+    /* Cleanup for sep */
+    if (sep.obj)
+       PyBuffer_Release(&sep);
+
+    return return_value;
+}
+
+static PyObject *
+bytes_partition_impl(PyBytesObject *self, Py_buffer *sep)
+/*[clinic end generated code: output=3006727cfbf83aa4 input=bc855dc63ca949de]*/
+{
     return stringlib_partition(
         (PyObject*) self,
         PyBytes_AS_STRING(self), PyBytes_GET_SIZE(self),
-        sep, sep_chars, sep_len
+        sep->obj, (const char *)sep->buf, sep->len
         );
 }
 
@@ -1805,7 +1791,7 @@
 bytes.rpartition
 
     self: self(type="PyBytesObject *")
-    sep: object
+    sep: Py_buffer
     /
 
 Partition the bytes into three parts using the given separator.
@@ -1832,26 +1818,39 @@
 "objects and the original bytes object.");
 
 #define BYTES_RPARTITION_METHODDEF    \
-    {"rpartition", (PyCFunction)bytes_rpartition, METH_O, bytes_rpartition__doc__},
+    {"rpartition", (PyCFunction)bytes_rpartition, METH_VARARGS, bytes_rpartition__doc__},
 
 static PyObject *
-bytes_rpartition(PyBytesObject *self, PyObject *sep)
-/*[clinic end generated code: output=3a620803657196ee input=79bc2932e78e5ce0]*/
+bytes_rpartition_impl(PyBytesObject *self, Py_buffer *sep);
+
+static PyObject *
+bytes_rpartition(PyBytesObject *self, PyObject *args)
 {
-    const char *sep_chars;
-    Py_ssize_t sep_len;
-
-    if (PyBytes_Check(sep)) {
-        sep_chars = PyBytes_AS_STRING(sep);
-        sep_len = PyBytes_GET_SIZE(sep);
-    }
-    else if (PyObject_AsCharBuffer(sep, &sep_chars, &sep_len))
-        return NULL;
-
+    PyObject *return_value = NULL;
+    Py_buffer sep = {NULL, NULL};
+
+    if (!PyArg_ParseTuple(args,
+        "y*:rpartition",
+        &sep))
+        goto exit;
+    return_value = bytes_rpartition_impl(self, &sep);
+
+exit:
+    /* Cleanup for sep */
+    if (sep.obj)
+       PyBuffer_Release(&sep);
+
+    return return_value;
+}
+
+static PyObject *
+bytes_rpartition_impl(PyBytesObject *self, Py_buffer *sep)
+/*[clinic end generated code: output=57b169dc47fa90e8 input=6588fff262a9170e]*/
+{
     return stringlib_rpartition(
         (PyObject*) self,
         PyBytes_AS_STRING(self), PyBytes_GET_SIZE(self),
-        sep, sep_chars, sep_len
+        sep->obj, (const char *)sep->buf, sep->len
         );
 }
 
@@ -1916,7 +1915,7 @@
         maxsplit = PY_SSIZE_T_MAX;
     if (sep == Py_None)
         return stringlib_rsplit_whitespace((PyObject*) self, s, len, maxsplit);
-    if (_getbuffer(sep, &vsub) < 0)
+    if (PyObject_GetBuffer(sep, &vsub, PyBUF_SIMPLE) != 0)
         return NULL;
     sub = vsub.buf;
     n = vsub.len;
@@ -2003,7 +2002,7 @@
         return -2;
 
     if (subobj) {
-        if (_getbuffer(subobj, &subbuf) < 0)
+        if (PyObject_GetBuffer(subobj, &subbuf, PyBUF_SIMPLE) != 0)
             return -2;
 
         sub = subbuf.buf;
@@ -2118,7 +2117,7 @@
     Py_ssize_t seplen;
     Py_ssize_t i, j;
 
-    if (_getbuffer(sepobj, &vsep) < 0)
+    if (PyObject_GetBuffer(sepobj, &vsep, PyBUF_SIMPLE) != 0)
         return NULL;
     sep = vsep.buf;
     seplen = vsep.len;
@@ -2360,7 +2359,7 @@
         return NULL;
 
     if (sub_obj) {
-        if (_getbuffer(sub_obj, &vsub) < 0)
+        if (PyObject_GetBuffer(sub_obj, &vsub, PyBUF_SIMPLE) != 0)
             return NULL;
 
         sub = vsub.buf;
@@ -2450,6 +2449,8 @@
 /*[clinic end generated code: output=f0f29a57f41df5d8 input=d8fa5519d7cc4be7]*/
 {
     char *input, *output;
+    Py_buffer table_view = {NULL, NULL};
+    Py_buffer del_table_view = {NULL, NULL};
     const char *table_chars;
     Py_ssize_t i, c, changed = 0;
     PyObject *input_obj = (PyObject*)self;
@@ -2466,12 +2467,17 @@
         table_chars = NULL;
         tablen = 256;
     }
-    else if (PyObject_AsCharBuffer(table, &table_chars, &tablen))
-        return NULL;
+    else {
+        if (PyObject_GetBuffer(table, &table_view, PyBUF_SIMPLE) != 0)
+            return NULL;
+        table_chars = table_view.buf;
+        tablen = table_view.len;
+    }
 
     if (tablen != 256) {
         PyErr_SetString(PyExc_ValueError,
           "translation table must be 256 characters long");
+        PyBuffer_Release(&table_view);
         return NULL;
     }
 
@@ -2480,8 +2486,14 @@
             del_table_chars = PyBytes_AS_STRING(deletechars);
             dellen = PyBytes_GET_SIZE(deletechars);
         }
-        else if (PyObject_AsCharBuffer(deletechars, &del_table_chars, &dellen))
-            return NULL;
+        else {
+            if (PyObject_GetBuffer(deletechars, &del_table_view, PyBUF_SIMPLE) != 0) {
+                PyBuffer_Release(&table_view);
+                return NULL;
+            }
+            del_table_chars = del_table_view.buf;
+            dellen = del_table_view.len;
+        }
     }
     else {
         del_table_chars = NULL;
@@ -2490,8 +2502,11 @@
 
     inlen = PyBytes_GET_SIZE(input_obj);
     result = PyBytes_FromStringAndSize((char *)NULL, inlen);
-    if (result == NULL)
+    if (result == NULL) {
+        PyBuffer_Release(&del_table_view);
+        PyBuffer_Release(&table_view);
         return NULL;
+    }
     output_start = output = PyBytes_AsString(result);
     input = PyBytes_AS_STRING(input_obj);
 
@@ -2502,11 +2517,14 @@
             if (Py_CHARMASK((*output++ = table_chars[c])) != c)
                 changed = 1;
         }
-        if (changed || !PyBytes_CheckExact(input_obj))
-            return result;
-        Py_DECREF(result);
-        Py_INCREF(input_obj);
-        return input_obj;
+        if (!changed && PyBytes_CheckExact(input_obj)) {
+            Py_INCREF(input_obj);
+            Py_DECREF(result);
+            result = input_obj;
+        }
+        PyBuffer_Release(&del_table_view);
+        PyBuffer_Release(&table_view);
+        return result;
     }
 
     if (table_chars == NULL) {
@@ -2516,9 +2534,11 @@
         for (i = 0; i < 256; i++)
             trans_table[i] = Py_CHARMASK(table_chars[i]);
     }
+    PyBuffer_Release(&table_view);
 
     for (i = 0; i < dellen; i++)
         trans_table[(int) Py_CHARMASK(del_table_chars[i])] = -1;
+    PyBuffer_Release(&del_table_view);
 
     for (i = inlen; --i >= 0; ) {
         c = Py_CHARMASK(*input++);
@@ -2544,8 +2564,8 @@
 @staticmethod
 bytes.maketrans
 
-    frm: object
-    to: object
+    frm: Py_buffer
+    to: Py_buffer
     /
 
 Return a translation table useable for the bytes or bytearray translate method.
@@ -2571,28 +2591,35 @@
     {"maketrans", (PyCFunction)bytes_maketrans, METH_VARARGS|METH_STATIC, bytes_maketrans__doc__},
 
 static PyObject *
-bytes_maketrans_impl(PyObject *frm, PyObject *to);
+bytes_maketrans_impl(Py_buffer *frm, Py_buffer *to);
 
 static PyObject *
 bytes_maketrans(void *null, PyObject *args)
 {
     PyObject *return_value = NULL;
-    PyObject *frm;
-    PyObject *to;
-
-    if (!PyArg_UnpackTuple(args, "maketrans",
-        2, 2,
+    Py_buffer frm = {NULL, NULL};
+    Py_buffer to = {NULL, NULL};
+
+    if (!PyArg_ParseTuple(args,
+        "y*y*:maketrans",
         &frm, &to))
         goto exit;
-    return_value = bytes_maketrans_impl(frm, to);
+    return_value = bytes_maketrans_impl(&frm, &to);
 
 exit:
+    /* Cleanup for frm */
+    if (frm.obj)
+       PyBuffer_Release(&frm);
+    /* Cleanup for to */
+    if (to.obj)
+       PyBuffer_Release(&to);
+
     return return_value;
 }
 
 static PyObject *
-bytes_maketrans_impl(PyObject *frm, PyObject *to)
-/*[clinic end generated code: output=89a3c3556975e466 input=d204f680f85da382]*/
+bytes_maketrans_impl(Py_buffer *frm, Py_buffer *to)
+/*[clinic end generated code: output=7df47390c476ac60 input=de7a8fc5632bb8f1]*/
 {
     return _Py_bytes_maketrans(frm, to);
 }
@@ -3093,8 +3120,8 @@
 /*[clinic input]
 bytes.replace
 
-    old: object
-    new: object
+    old: Py_buffer
+    new: Py_buffer
     count: Py_ssize_t = -1
         Maximum number of occurrences to replace.
         -1 (the default value) means replace all occurrences.
@@ -3123,50 +3150,40 @@
     {"replace", (PyCFunction)bytes_replace, METH_VARARGS, bytes_replace__doc__},
 
 static PyObject *
-bytes_replace_impl(PyBytesObject*self, PyObject *old, PyObject *new, Py_ssize_t count);
+bytes_replace_impl(PyBytesObject*self, Py_buffer *old, Py_buffer *new, Py_ssize_t count);
 
 static PyObject *
 bytes_replace(PyBytesObject*self, PyObject *args)
 {
     PyObject *return_value = NULL;
-    PyObject *old;
-    PyObject *new;
+    Py_buffer old = {NULL, NULL};
+    Py_buffer new = {NULL, NULL};
     Py_ssize_t count = -1;
 
     if (!PyArg_ParseTuple(args,
-        "OO|n:replace",
+        "y*y*|n:replace",
         &old, &new, &count))
         goto exit;
-    return_value = bytes_replace_impl(self, old, new, count);
+    return_value = bytes_replace_impl(self, &old, &new, count);
 
 exit:
+    /* Cleanup for old */
+    if (old.obj)
+       PyBuffer_Release(&old);
+    /* Cleanup for new */
+    if (new.obj)
+       PyBuffer_Release(&new);
+
     return return_value;
 }
 
 static PyObject *
-bytes_replace_impl(PyBytesObject*self, PyObject *old, PyObject *new, Py_ssize_t count)
-/*[clinic end generated code: output=14ce72f4f9cb91cf input=d3ac254ea50f4ac1]*/
+bytes_replace_impl(PyBytesObject*self, Py_buffer *old, Py_buffer *new, Py_ssize_t count)
+/*[clinic end generated code: output=f07bd9ecf29ee8d8 input=b2fbbf0bf04de8e5]*/
 {
-    const char *old_s, *new_s;
-    Py_ssize_t old_len, new_len;
-
-    if (PyBytes_Check(old)) {
-        old_s = PyBytes_AS_STRING(old);
-        old_len = PyBytes_GET_SIZE(old);
-    }
-    else if (PyObject_AsCharBuffer(old, &old_s, &old_len))
-        return NULL;
-
-    if (PyBytes_Check(new)) {
-        new_s = PyBytes_AS_STRING(new);
-        new_len = PyBytes_GET_SIZE(new);
-    }
-    else if (PyObject_AsCharBuffer(new, &new_s, &new_len))
-        return NULL;
-
     return (PyObject *)replace((PyBytesObject *) self,
-                               old_s, old_len,
-                               new_s, new_len, count);
+                               (const char *)old->buf, old->len,
+                               (const char *)new->buf, new->len, count);
 }
 
 /** End DALKE **/
@@ -3181,6 +3198,7 @@
 {
     Py_ssize_t len = PyBytes_GET_SIZE(self);
     Py_ssize_t slen;
+    Py_buffer sub_view = {NULL, NULL};
     const char* sub;
     const char* str;
 
@@ -3188,8 +3206,12 @@
         sub = PyBytes_AS_STRING(substr);
         slen = PyBytes_GET_SIZE(substr);
     }
-    else if (PyObject_AsCharBuffer(substr, &sub, &slen))
-        return -1;
+    else {
+        if (PyObject_GetBuffer(substr, &sub_view, PyBUF_SIMPLE) != 0)
+            return -1;
+        sub = sub_view.buf;
+        slen = sub_view.len;
+    }
     str = PyBytes_AS_STRING(self);
 
     ADJUST_INDICES(start, end, len);
@@ -3197,17 +3219,25 @@
     if (direction < 0) {
         /* startswith */
         if (start+slen > len)
-            return 0;
+            goto notfound;
     } else {
         /* endswith */
         if (end-start < slen || start > len)
-            return 0;
+            goto notfound;
 
         if (end-slen > start)
             start = end - slen;
     }
-    if (end-start >= slen)
-        return ! memcmp(str+start, sub, slen);
+    if (end-start < slen)
+        goto notfound;
+    if (memcmp(str+start, sub, slen) != 0)
+        goto notfound;
+
+    PyBuffer_Release(&sub_view);
+    return 1;
+
+notfound:
+    PyBuffer_Release(&sub_view);
     return 0;
 }
 
@@ -3978,7 +4008,7 @@
         Py_buffer wb;
 
         wb.len = -1;
-        if (_getbuffer(w, &wb) < 0) {
+        if (PyObject_GetBuffer(w, &wb, PyBUF_SIMPLE) != 0) {
             PyErr_Format(PyExc_TypeError, "can't concat %.100s to %.100s",
                          Py_TYPE(w)->tp_name, Py_TYPE(*pv)->tp_name);
             Py_CLEAR(*pv);
diff --git a/Objects/complexobject.c b/Objects/complexobject.c
--- a/Objects/complexobject.c
+++ b/Objects/complexobject.c
@@ -767,6 +767,7 @@
     int got_bracket=0;
     PyObject *s_buffer = NULL;
     Py_ssize_t len;
+    Py_buffer view = {NULL, NULL};
 
     if (PyUnicode_Check(v)) {
         s_buffer = _PyUnicode_TransformDecimalAndSpaceToASCII(v);
@@ -776,7 +777,11 @@
         if (s == NULL)
             goto error;
     }
-    else if (PyObject_AsCharBuffer(v, &s, &len)) {
+    else if (PyObject_GetBuffer(v, &view, PyBUF_SIMPLE) == 0) {
+        s = (const char *)view.buf;
+        len = view.len;
+    }
+    else {
         PyErr_Format(PyExc_TypeError,
             "complex() argument must be a string or a number, not '%.200s'",
             Py_TYPE(v)->tp_name);
@@ -890,6 +895,7 @@
     if (s-start != len)
         goto parse_error;
 
+    PyBuffer_Release(&view);
     Py_XDECREF(s_buffer);
     return complex_subtype_from_doubles(type, x, y);
 
@@ -897,6 +903,7 @@
     PyErr_SetString(PyExc_ValueError,
                     "complex() arg is a malformed string");
   error:
+    PyBuffer_Release(&view);
     Py_XDECREF(s_buffer);
     return NULL;
 }
diff --git a/Objects/exceptions.c b/Objects/exceptions.c
--- a/Objects/exceptions.c
+++ b/Objects/exceptions.c
@@ -1922,8 +1922,6 @@
 UnicodeDecodeError_init(PyObject *self, PyObject *args, PyObject *kwds)
 {
     PyUnicodeErrorObject *ude;
-    const char *data;
-    Py_ssize_t size;
 
     if (BaseException_init((PyBaseExceptionObject *)self, args, kwds) == -1)
         return -1;
@@ -1944,21 +1942,27 @@
              return -1;
     }
 
+    Py_INCREF(ude->encoding);
+    Py_INCREF(ude->object);
+    Py_INCREF(ude->reason);
+
     if (!PyBytes_Check(ude->object)) {
-        if (PyObject_AsReadBuffer(ude->object, (const void **)&data, &size)) {
-            ude->encoding = ude->object = ude->reason = NULL;
-            return -1;
-        }
-        ude->object = PyBytes_FromStringAndSize(data, size);
+        Py_buffer view;
+        if (PyObject_GetBuffer(ude->object, &view, PyBUF_SIMPLE) != 0)
+            goto error;
+        Py_CLEAR(ude->object);
+        ude->object = PyBytes_FromStringAndSize(view.buf, view.len);
+        PyBuffer_Release(&view);
+        if (!ude->object)
+            goto error;
     }
-    else {
-        Py_INCREF(ude->object);
-    }
-
-    Py_INCREF(ude->encoding);
-    Py_INCREF(ude->reason);
-
     return 0;
+
+error:
+    Py_CLEAR(ude->encoding);
+    Py_CLEAR(ude->object);
+    Py_CLEAR(ude->reason);
+    return -1;
 }
 
 static PyObject *
diff --git a/Objects/floatobject.c b/Objects/floatobject.c
--- a/Objects/floatobject.c
+++ b/Objects/floatobject.c
@@ -131,6 +131,7 @@
     double x;
     PyObject *s_buffer = NULL;
     Py_ssize_t len;
+    Py_buffer view = {NULL, NULL};
     PyObject *result = NULL;
 
     if (PyUnicode_Check(v)) {
@@ -143,7 +144,11 @@
             return NULL;
         }
     }
-    else if (PyObject_AsCharBuffer(v, &s, &len)) {
+    else if (PyObject_GetBuffer(v, &view, PyBUF_SIMPLE) == 0) {
+        s = (const char *)view.buf;
+        len = view.len;
+    }
+    else {
         PyErr_Format(PyExc_TypeError,
             "float() argument must be a string or a number, not '%.200s'",
             Py_TYPE(v)->tp_name);
@@ -170,6 +175,7 @@
     else
         result = PyFloat_FromDouble(x);
 
+    PyBuffer_Release(&view);
     Py_XDECREF(s_buffer);
     return result;
 }
diff --git a/Objects/stringlib/join.h b/Objects/stringlib/join.h
--- a/Objects/stringlib/join.h
+++ b/Objects/stringlib/join.h
@@ -58,7 +58,14 @@
     for (i = 0, nbufs = 0; i < seqlen; i++) {
         Py_ssize_t itemlen;
         item = PySequence_Fast_GET_ITEM(seq, i);
-        if (_getbuffer(item, &buffers[i]) < 0) {
+        if (PyBytes_CheckExact(item)) {
+            /* Fast path. */
+            Py_INCREF(item);
+            buffers[i].obj = item;
+            buffers[i].buf = PyBytes_AS_STRING(item);
+            buffers[i].len = PyBytes_GET_SIZE(item);
+        }
+        else if (PyObject_GetBuffer(item, &buffers[i], PyBUF_SIMPLE) != 0) {
             PyErr_Format(PyExc_TypeError,
                          "sequence item %zd: expected a bytes-like object, "
                          "%.80s found",
diff --git a/Python/bltinmodule.c b/Python/bltinmodule.c
--- a/Python/bltinmodule.c
+++ b/Python/bltinmodule.c
@@ -723,10 +723,10 @@
 }
 
 
-static char *
-source_as_string(PyObject *cmd, char *funcname, char *what, PyCompilerFlags *cf)
+static const char *
+source_as_string(PyObject *cmd, const char *funcname, const char *what, PyCompilerFlags *cf, Py_buffer *view)
 {
-    char *str;
+    const char *str;
     Py_ssize_t size;
 
     if (PyUnicode_Check(cmd)) {
@@ -735,19 +735,21 @@
         if (str == NULL)
             return NULL;
     }
-    else if (!PyObject_CheckReadBuffer(cmd)) {
+    else if (PyObject_GetBuffer(cmd, view, PyBUF_SIMPLE) == 0) {
+        str = (const char *)view->buf;
+        size = view->len;
+    }
+    else {
         PyErr_Format(PyExc_TypeError,
           "%s() arg 1 must be a %s object",
           funcname, what);
         return NULL;
     }
-    else if (PyObject_AsReadBuffer(cmd, (const void **)&str, &size) < 0) {
-        return NULL;
-    }
-
-    if (strlen(str) != (size_t)size)  {
+
+    if (strlen(str) != (size_t)size) {
         PyErr_SetString(PyExc_ValueError,
                         "source code string cannot contain null bytes");
+        PyBuffer_Release(view);
         return NULL;
     }
     return str;
@@ -827,7 +829,8 @@
 builtin_compile_impl(PyModuleDef *module, PyObject *source, PyObject *filename, const char *mode, int flags, int dont_inherit, int optimize)
 /*[clinic end generated code: output=c72d197809d178fc input=c6212a9d21472f7e]*/
 {
-    char *str;
+    Py_buffer view = {NULL, NULL};
+    const char *str;
     int compile_mode = -1;
     int is_ast;
     PyCompilerFlags cf;
@@ -898,11 +901,12 @@
         goto finally;
     }
 
-    str = source_as_string(source, "compile", "string, bytes or AST", &cf);
+    str = source_as_string(source, "compile", "string, bytes or AST", &cf, &view);
     if (str == NULL)
         goto error;
 
     result = Py_CompileStringObject(str, filename, start[compile_mode], &cf, optimize);
+    PyBuffer_Release(&view);
     goto finally;
 
 error:
@@ -1042,7 +1046,8 @@
 /*[clinic end generated code: output=644fd59012538ce6 input=31e42c1d2125b50b]*/
 {
     PyObject *result, *tmp = NULL;
-    char *str;
+    Py_buffer view = {NULL, NULL};
+    const char *str;
     PyCompilerFlags cf;
 
     if (locals != Py_None && !PyMapping_Check(locals)) {
@@ -1089,7 +1094,7 @@
     }
 
     cf.cf_flags = PyCF_SOURCE_IS_UTF8;
-    str = source_as_string(source, "eval", "string, bytes or code", &cf);
+    str = source_as_string(source, "eval", "string, bytes or code", &cf, &view);
     if (str == NULL)
         return NULL;
 
@@ -1098,6 +1103,7 @@
 
     (void)PyEval_MergeCompilerFlags(&cf);
     result = PyRun_StringFlags(str, Py_eval_input, globals, locals, &cf);
+    PyBuffer_Release(&view);
     Py_XDECREF(tmp);
     return result;
 }
@@ -1204,11 +1210,12 @@
         v = PyEval_EvalCode(source, globals, locals);
     }
     else {
-        char *str;
+        Py_buffer view = {NULL, NULL};
+        const char *str;
         PyCompilerFlags cf;
         cf.cf_flags = PyCF_SOURCE_IS_UTF8;
         str = source_as_string(source, "exec",
-                                       "string, bytes or code", &cf);
+                                       "string, bytes or code", &cf, &view);
         if (str == NULL)
             return NULL;
         if (PyEval_MergeCompilerFlags(&cf))
@@ -1216,6 +1223,7 @@
                                   locals, &cf);
         else
             v = PyRun_String(str, Py_file_input, globals, locals);
+        PyBuffer_Release(&view);
     }
     if (v == NULL)
         return NULL;

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Tue Feb  3 01:05:58 2015
From: python-checkins at python.org (serhiy.storchaka)
Date: Tue, 03 Feb 2015 00:05:58 +0000
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIyODk2?=
 =?utf-8?q?=3A_Avoid_to_use_PyObject=5FAsCharBuffer=28=29=2C_PyObject=5FAs?=
 =?utf-8?q?ReadBuffer=28=29?=
Message-ID: <20150203000514.39292.99884@psf.io>

https://hg.python.org/cpython/rev/1da9630e9b7f
changeset:   94474:1da9630e9b7f
branch:      3.4
parent:      94472:b883bb8bd744
user:        Serhiy Storchaka <storchaka at gmail.com>
date:        Tue Feb 03 01:21:08 2015 +0200
summary:
  Issue #22896: Avoid to use PyObject_AsCharBuffer(), PyObject_AsReadBuffer()
and PyObject_AsWriteBuffer().

files:
  Lib/ctypes/test/test_frombuffer.py |   50 +++-
  Misc/NEWS                          |    3 +
  Modules/_codecsmodule.c            |   24 +-
  Modules/_ctypes/_ctypes.c          |   59 +++--
  Modules/_io/bytesio.c              |   10 +-
  Modules/_sqlite/connection.c       |   11 +-
  Modules/_sqlite/statement.c        |   13 +-
  Modules/_struct.c                  |   23 +-
  Objects/abstract.c                 |   81 ++-----
  Objects/bytearrayobject.c          |   98 +++-----
  Objects/bytes_methods.c            |   33 +--
  Objects/bytesobject.c              |  179 ++++++++--------
  Objects/complexobject.c            |    9 +-
  Objects/exceptions.c               |   32 +-
  Objects/floatobject.c              |    8 +-
  Objects/stringlib/join.h           |    9 +-
  Python/bltinmodule.c               |   34 +-
  17 files changed, 334 insertions(+), 342 deletions(-)


diff --git a/Lib/ctypes/test/test_frombuffer.py b/Lib/ctypes/test/test_frombuffer.py
--- a/Lib/ctypes/test/test_frombuffer.py
+++ b/Lib/ctypes/test/test_frombuffer.py
@@ -10,7 +10,7 @@
         self._init_called = True
 
 class Test(unittest.TestCase):
-    def test_fom_buffer(self):
+    def test_from_buffer(self):
         a = array.array("i", range(16))
         x = (c_int * 16).from_buffer(a)
 
@@ -23,25 +23,37 @@
         a[0], a[-1] = 200, -200
         self.assertEqual(x[:], a.tolist())
 
-        self.assertIn(a, x._objects.values())
+        self.assertRaises(BufferError, a.append, 100)
+        self.assertRaises(BufferError, a.pop)
 
-        self.assertRaises(ValueError,
-                          c_int.from_buffer, a, -1)
+        del x; del y; gc.collect(); gc.collect(); gc.collect()
+        a.append(100)
+        a.pop()
+        x = (c_int * 16).from_buffer(a)
+
+        self.assertIn(a, [obj.obj if isinstance(obj, memoryview) else obj
+                          for obj in x._objects.values()])
 
         expected = x[:]
         del a; gc.collect(); gc.collect(); gc.collect()
         self.assertEqual(x[:], expected)
 
-        self.assertRaises(TypeError,
-                          (c_char * 16).from_buffer, "a" * 16)
+        with self.assertRaises(TypeError):
+            (c_char * 16).from_buffer(b"a" * 16)
+        with self.assertRaises(TypeError):
+            (c_char * 16).from_buffer("a" * 16)
 
-    def test_fom_buffer_with_offset(self):
+    def test_from_buffer_with_offset(self):
         a = array.array("i", range(16))
         x = (c_int * 15).from_buffer(a, sizeof(c_int))
 
         self.assertEqual(x[:], a.tolist()[1:])
-        self.assertRaises(ValueError, lambda: (c_int * 16).from_buffer(a, sizeof(c_int)))
-        self.assertRaises(ValueError, lambda: (c_int * 1).from_buffer(a, 16 * sizeof(c_int)))
+        with self.assertRaises(ValueError):
+            c_int.from_buffer(a, -1)
+        with self.assertRaises(ValueError):
+            (c_int * 16).from_buffer(a, sizeof(c_int))
+        with self.assertRaises(ValueError):
+            (c_int * 1).from_buffer(a, 16 * sizeof(c_int))
 
     def test_from_buffer_copy(self):
         a = array.array("i", range(16))
@@ -56,26 +68,30 @@
         a[0], a[-1] = 200, -200
         self.assertEqual(x[:], list(range(16)))
 
+        a.append(100)
+        self.assertEqual(x[:], list(range(16)))
+
         self.assertEqual(x._objects, None)
 
-        self.assertRaises(ValueError,
-                          c_int.from_buffer, a, -1)
-
         del a; gc.collect(); gc.collect(); gc.collect()
         self.assertEqual(x[:], list(range(16)))
 
         x = (c_char * 16).from_buffer_copy(b"a" * 16)
         self.assertEqual(x[:], b"a" * 16)
+        with self.assertRaises(TypeError):
+            (c_char * 16).from_buffer_copy("a" * 16)
 
-    def test_fom_buffer_copy_with_offset(self):
+    def test_from_buffer_copy_with_offset(self):
         a = array.array("i", range(16))
         x = (c_int * 15).from_buffer_copy(a, sizeof(c_int))
 
         self.assertEqual(x[:], a.tolist()[1:])
-        self.assertRaises(ValueError,
-                          (c_int * 16).from_buffer_copy, a, sizeof(c_int))
-        self.assertRaises(ValueError,
-                          (c_int * 1).from_buffer_copy, a, 16 * sizeof(c_int))
+        with self.assertRaises(ValueError):
+            c_int.from_buffer_copy(a, -1)
+        with self.assertRaises(ValueError):
+            (c_int * 16).from_buffer_copy(a, sizeof(c_int))
+        with self.assertRaises(ValueError):
+            (c_int * 1).from_buffer_copy(a, 16 * sizeof(c_int))
 
 if __name__ == '__main__':
     unittest.main()
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -11,6 +11,9 @@
 Core and Builtins
 -----------------
 
+- Issue #22896: Avoid using PyObject_AsCharBuffer(), PyObject_AsReadBuffer()
+  and PyObject_AsWriteBuffer().
+
 - Issue #21295: Revert some changes (issue #16795) to AST line numbers and
   column offsets that constituted a regression.
 
diff --git a/Modules/_codecsmodule.c b/Modules/_codecsmodule.c
--- a/Modules/_codecsmodule.c
+++ b/Modules/_codecsmodule.c
@@ -284,8 +284,6 @@
 {
     PyObject *obj;
     const char *errors = NULL;
-    const char *data;
-    Py_ssize_t size;
 
     if (!PyArg_ParseTuple(args, "O|z:unicode_internal_decode",
                           &obj, &errors))
@@ -298,11 +296,16 @@
         return codec_tuple(obj, PyUnicode_GET_LENGTH(obj));
     }
     else {
-        if (PyObject_AsReadBuffer(obj, (const void **)&data, &size))
+        Py_buffer view;
+        PyObject *result;
+        if (PyObject_GetBuffer(obj, &view, PyBUF_SIMPLE) != 0)
             return NULL;
 
-        return codec_tuple(_PyUnicode_DecodeUnicodeInternal(data, size, errors),
-                           size);
+        result = codec_tuple(
+                _PyUnicode_DecodeUnicodeInternal(view.buf, view.len, errors),
+                view.len);
+        PyBuffer_Release(&view);
+        return result;
     }
 }
 
@@ -727,8 +730,6 @@
 {
     PyObject *obj;
     const char *errors = NULL;
-    const char *data;
-    Py_ssize_t len, size;
 
     if (PyErr_WarnEx(PyExc_DeprecationWarning,
                      "unicode_internal codec has been deprecated",
@@ -741,6 +742,7 @@
 
     if (PyUnicode_Check(obj)) {
         Py_UNICODE *u;
+        Py_ssize_t len, size;
 
         if (PyUnicode_READY(obj) < 0)
             return NULL;
@@ -755,9 +757,13 @@
                            PyUnicode_GET_LENGTH(obj));
     }
     else {
-        if (PyObject_AsReadBuffer(obj, (const void **)&data, &size))
+        Py_buffer view;
+        PyObject *result;
+        if (PyObject_GetBuffer(obj, &view, PyBUF_SIMPLE) != 0)
             return NULL;
-        return codec_tuple(PyBytes_FromStringAndSize(data, size), size);
+        result = codec_tuple(PyBytes_FromStringAndSize(view.buf, view.len), view.len);
+        PyBuffer_Release(&view);
+        return result;
     }
 }
 
diff --git a/Modules/_ctypes/_ctypes.c b/Modules/_ctypes/_ctypes.c
--- a/Modules/_ctypes/_ctypes.c
+++ b/Modules/_ctypes/_ctypes.c
@@ -463,39 +463,45 @@
 static PyObject *
 CDataType_from_buffer(PyObject *type, PyObject *args)
 {
-    void *buffer;
-    Py_ssize_t buffer_len;
+    Py_buffer buffer;
     Py_ssize_t offset = 0;
-    PyObject *obj, *result;
+    PyObject *result, *mv;
     StgDictObject *dict = PyType_stgdict(type);
     assert (dict);
 
-    if (!PyArg_ParseTuple(args, "O|n:from_buffer", &obj, &offset))
-        return NULL;
-
-    if (-1 == PyObject_AsWriteBuffer(obj, &buffer, &buffer_len))
+    if (!PyArg_ParseTuple(args, "w*|n:from_buffer", &buffer, &offset))
         return NULL;
 
     if (offset < 0) {
         PyErr_SetString(PyExc_ValueError,
                         "offset cannot be negative");
+        PyBuffer_Release(&buffer);
         return NULL;
     }
-    if (dict->size > buffer_len - offset) {
+    if (dict->size > buffer.len - offset) {
         PyErr_Format(PyExc_ValueError,
                      "Buffer size too small (%zd instead of at least %zd bytes)",
-                     buffer_len, dict->size + offset);
+                     buffer.len, dict->size + offset);
+        PyBuffer_Release(&buffer);
         return NULL;
     }
 
-    result = PyCData_AtAddress(type, (char *)buffer + offset);
-    if (result == NULL)
+    result = PyCData_AtAddress(type, (char *)buffer.buf + offset);
+    if (result == NULL) {
+        PyBuffer_Release(&buffer);
         return NULL;
-
-    Py_INCREF(obj);
-    if (-1 == KeepRef((CDataObject *)result, -1, obj)) {
+    }
+
+    mv = PyMemoryView_FromBuffer(&buffer);
+    if (mv == NULL) {
+        PyBuffer_Release(&buffer);
         return NULL;
     }
+    /* Hack the memoryview so that it will release the buffer. */
+    ((PyMemoryViewObject *)mv)->mbuf->master.obj = buffer.obj;
+    ((PyMemoryViewObject *)mv)->view.obj = buffer.obj;
+    if (-1 == KeepRef((CDataObject *)result, -1, mv))
+        result = NULL;
     return result;
 }
 
@@ -508,37 +514,36 @@
 static PyObject *
 CDataType_from_buffer_copy(PyObject *type, PyObject *args)
 {
-    const void *buffer;
-    Py_ssize_t buffer_len;
+    Py_buffer buffer;
     Py_ssize_t offset = 0;
-    PyObject *obj, *result;
+    PyObject *result;
     StgDictObject *dict = PyType_stgdict(type);
     assert (dict);
 
-    if (!PyArg_ParseTuple(args, "O|n:from_buffer", &obj, &offset))
-        return NULL;
-
-    if (-1 == PyObject_AsReadBuffer(obj, (const void**)&buffer, &buffer_len))
+    if (!PyArg_ParseTuple(args, "y*|n:from_buffer", &buffer, &offset))
         return NULL;
 
     if (offset < 0) {
         PyErr_SetString(PyExc_ValueError,
                         "offset cannot be negative");
+        PyBuffer_Release(&buffer);
         return NULL;
     }
 
-    if (dict->size > buffer_len - offset) {
+    if (dict->size > buffer.len - offset) {
         PyErr_Format(PyExc_ValueError,
                      "Buffer size too small (%zd instead of at least %zd bytes)",
-                     buffer_len, dict->size + offset);
+                     buffer.len, dict->size + offset);
+        PyBuffer_Release(&buffer);
         return NULL;
     }
 
     result = GenericPyCData_new((PyTypeObject *)type, NULL, NULL);
-    if (result == NULL)
-        return NULL;
-    memcpy(((CDataObject *)result)->b_ptr,
-           (char *)buffer+offset, dict->size);
+    if (result != NULL) {
+        memcpy(((CDataObject *)result)->b_ptr,
+               (char *)buffer.buf + offset, dict->size);
+    }
+    PyBuffer_Release(&buffer);
     return result;
 }
 
diff --git a/Modules/_io/bytesio.c b/Modules/_io/bytesio.c
--- a/Modules/_io/bytesio.c
+++ b/Modules/_io/bytesio.c
@@ -437,17 +437,18 @@
 "is set not to block as has no data to read.");
 
 static PyObject *
-bytesio_readinto(bytesio *self, PyObject *buffer)
+bytesio_readinto(bytesio *self, PyObject *arg)
 {
-    void *raw_buffer;
+    Py_buffer buffer;
     Py_ssize_t len, n;
 
     CHECK_CLOSED(self);
 
-    if (PyObject_AsWriteBuffer(buffer, &raw_buffer, &len) == -1)
+    if (!PyArg_Parse(arg, "w*", &buffer))
         return NULL;
 
     /* adjust invalid sizes */
+    len = buffer.len;
     n = self->string_size - self->pos;
     if (len > n) {
         len = n;
@@ -455,10 +456,11 @@
             len = 0;
     }
 
-    memcpy(raw_buffer, self->buf + self->pos, len);
+    memcpy(buffer.buf, self->buf + self->pos, len);
     assert(self->pos + len < PY_SSIZE_T_MAX);
     assert(len >= 0);
     self->pos += len;
+    PyBuffer_Release(&buffer);
 
     return PyLong_FromSsize_t(len);
 }
diff --git a/Modules/_sqlite/connection.c b/Modules/_sqlite/connection.c
--- a/Modules/_sqlite/connection.c
+++ b/Modules/_sqlite/connection.c
@@ -522,19 +522,20 @@
             return -1;
         sqlite3_result_text(context, str, -1, SQLITE_TRANSIENT);
     } else if (PyObject_CheckBuffer(py_val)) {
-        const char* buffer;
-        Py_ssize_t buflen;
-        if (PyObject_AsCharBuffer(py_val, &buffer, &buflen) != 0) {
+        Py_buffer view;
+        if (PyObject_GetBuffer(py_val, &view, PyBUF_SIMPLE) != 0) {
             PyErr_SetString(PyExc_ValueError,
                             "could not convert BLOB to buffer");
             return -1;
         }
-        if (buflen > INT_MAX) {
+        if (view.len > INT_MAX) {
             PyErr_SetString(PyExc_OverflowError,
                             "BLOB longer than INT_MAX bytes");
+            PyBuffer_Release(&view);
             return -1;
         }
-        sqlite3_result_blob(context, buffer, (int)buflen, SQLITE_TRANSIENT);
+        sqlite3_result_blob(context, view.buf, (int)view.len, SQLITE_TRANSIENT);
+        PyBuffer_Release(&view);
     } else {
         return -1;
     }
diff --git a/Modules/_sqlite/statement.c b/Modules/_sqlite/statement.c
--- a/Modules/_sqlite/statement.c
+++ b/Modules/_sqlite/statement.c
@@ -94,7 +94,6 @@
 int pysqlite_statement_bind_parameter(pysqlite_Statement* self, int pos, PyObject* parameter)
 {
     int rc = SQLITE_OK;
-    const char* buffer;
     char* string;
     Py_ssize_t buflen;
     parameter_type paramtype;
@@ -145,18 +144,22 @@
             }
             rc = sqlite3_bind_text(self->st, pos, string, (int)buflen, SQLITE_TRANSIENT);
             break;
-        case TYPE_BUFFER:
-            if (PyObject_AsCharBuffer(parameter, &buffer, &buflen) != 0) {
+        case TYPE_BUFFER: {
+            Py_buffer view;
+            if (PyObject_GetBuffer(parameter, &view, PyBUF_SIMPLE) != 0) {
                 PyErr_SetString(PyExc_ValueError, "could not convert BLOB to buffer");
                 return -1;
             }
-            if (buflen > INT_MAX) {
+            if (view.len > INT_MAX) {
                 PyErr_SetString(PyExc_OverflowError,
                                 "BLOB longer than INT_MAX bytes");
+                PyBuffer_Release(&view);
                 return -1;
             }
-            rc = sqlite3_bind_blob(self->st, pos, buffer, buflen, SQLITE_TRANSIENT);
+            rc = sqlite3_bind_blob(self->st, pos, view.buf, (int)view.len, SQLITE_TRANSIENT);
+            PyBuffer_Release(&view);
             break;
+        }
         case TYPE_UNKNOWN:
             rc = -1;
     }
diff --git a/Modules/_struct.c b/Modules/_struct.c
--- a/Modules/_struct.c
+++ b/Modules/_struct.c
@@ -1842,8 +1842,8 @@
 s_pack_into(PyObject *self, PyObject *args)
 {
     PyStructObject *soself;
-    char *buffer;
-    Py_ssize_t buffer_len, offset;
+    Py_buffer buffer;
+    Py_ssize_t offset;
 
     /* Validate arguments.  +1 is for the first arg as buffer. */
     soself = (PyStructObject *)self;
@@ -1868,34 +1868,37 @@
     }
 
     /* Extract a writable memory buffer from the first argument */
-    if ( PyObject_AsWriteBuffer(PyTuple_GET_ITEM(args, 0),
-                                                            (void**)&buffer, &buffer_len) == -1 ) {
+    if (!PyArg_Parse(PyTuple_GET_ITEM(args, 0), "w*", &buffer))
         return NULL;
-    }
-    assert( buffer_len >= 0 );
+    assert(buffer.len >= 0);
 
     /* Extract the offset from the first argument */
     offset = PyNumber_AsSsize_t(PyTuple_GET_ITEM(args, 1), PyExc_IndexError);
-    if (offset == -1 && PyErr_Occurred())
+    if (offset == -1 && PyErr_Occurred()) {
+        PyBuffer_Release(&buffer);
         return NULL;
+    }
 
     /* Support negative offsets. */
     if (offset < 0)
-        offset += buffer_len;
+        offset += buffer.len;
 
     /* Check boundaries */
-    if (offset < 0 || (buffer_len - offset) < soself->s_size) {
+    if (offset < 0 || (buffer.len - offset) < soself->s_size) {
         PyErr_Format(StructError,
                      "pack_into requires a buffer of at least %zd bytes",
                      soself->s_size);
+        PyBuffer_Release(&buffer);
         return NULL;
     }
 
     /* Call the guts */
-    if ( s_pack_internal(soself, args, 2, buffer + offset) != 0 ) {
+    if (s_pack_internal(soself, args, 2, (char*)buffer.buf + offset) != 0) {
+        PyBuffer_Release(&buffer);
         return NULL;
     }
 
+    PyBuffer_Release(&buffer);
     Py_RETURN_NONE;
 }
 
diff --git a/Objects/abstract.c b/Objects/abstract.c
--- a/Objects/abstract.c
+++ b/Objects/abstract.c
@@ -250,28 +250,7 @@
                       const char **buffer,
                       Py_ssize_t *buffer_len)
 {
-    PyBufferProcs *pb;
-    Py_buffer view;
-
-    if (obj == NULL || buffer == NULL || buffer_len == NULL) {
-        null_error();
-        return -1;
-    }
-    pb = obj->ob_type->tp_as_buffer;
-    if (pb == NULL || pb->bf_getbuffer == NULL) {
-        PyErr_SetString(PyExc_TypeError,
-                        "expected bytes, bytearray "
-                        "or buffer compatible object");
-        return -1;
-    }
-    if ((*pb->bf_getbuffer)(obj, &view, PyBUF_SIMPLE)) return -1;
-
-    *buffer = view.buf;
-    *buffer_len = view.len;
-    if (pb->bf_releasebuffer != NULL)
-        (*pb->bf_releasebuffer)(obj, &view);
-    Py_XDECREF(view.obj);
-    return 0;
+    return PyObject_AsReadBuffer(obj, (const void **)buffer, buffer_len);
 }
 
 int
@@ -295,28 +274,18 @@
                           const void **buffer,
                           Py_ssize_t *buffer_len)
 {
-    PyBufferProcs *pb;
     Py_buffer view;
 
     if (obj == NULL || buffer == NULL || buffer_len == NULL) {
         null_error();
         return -1;
     }
-    pb = obj->ob_type->tp_as_buffer;
-    if (pb == NULL ||
-        pb->bf_getbuffer == NULL) {
-        PyErr_SetString(PyExc_TypeError,
-                        "expected an object with a buffer interface");
+    if (PyObject_GetBuffer(obj, &view, PyBUF_SIMPLE) != 0)
         return -1;
-    }
-
-    if ((*pb->bf_getbuffer)(obj, &view, PyBUF_SIMPLE)) return -1;
 
     *buffer = view.buf;
     *buffer_len = view.len;
-    if (pb->bf_releasebuffer != NULL)
-        (*pb->bf_releasebuffer)(obj, &view);
-    Py_XDECREF(view.obj);
+    PyBuffer_Release(&view);
     return 0;
 }
 
@@ -342,9 +311,7 @@
 
     *buffer = view.buf;
     *buffer_len = view.len;
-    if (pb->bf_releasebuffer != NULL)
-        (*pb->bf_releasebuffer)(obj, &view);
-    Py_XDECREF(view.obj);
+    PyBuffer_Release(&view);
     return 0;
 }
 
@@ -353,13 +320,15 @@
 int
 PyObject_GetBuffer(PyObject *obj, Py_buffer *view, int flags)
 {
-    if (!PyObject_CheckBuffer(obj)) {
+    PyBufferProcs *pb = obj->ob_type->tp_as_buffer;
+
+    if (pb == NULL || pb->bf_getbuffer == NULL) {
         PyErr_Format(PyExc_TypeError,
                      "'%.100s' does not support the buffer interface",
                      Py_TYPE(obj)->tp_name);
         return -1;
     }
-    return (*(obj->ob_type->tp_as_buffer->bf_getbuffer))(obj, view, flags);
+    return (*pb->bf_getbuffer)(obj, view, flags);
 }
 
 static int
@@ -652,10 +621,14 @@
 PyBuffer_Release(Py_buffer *view)
 {
     PyObject *obj = view->obj;
-    if (obj && Py_TYPE(obj)->tp_as_buffer && Py_TYPE(obj)->tp_as_buffer->bf_releasebuffer)
-        Py_TYPE(obj)->tp_as_buffer->bf_releasebuffer(obj, view);
-    Py_XDECREF(obj);
+    PyBufferProcs *pb;
+    if (obj == NULL)
+        return;
+    pb = Py_TYPE(obj)->tp_as_buffer;
+    if (pb && pb->bf_releasebuffer)
+        pb->bf_releasebuffer(obj, view);
     view->obj = NULL;
+    Py_DECREF(obj);
 }
 
 PyObject *
@@ -1249,8 +1222,7 @@
 {
     PyNumberMethods *m;
     PyObject *trunc_func;
-    const char *buffer;
-    Py_ssize_t buffer_len;
+    Py_buffer view;
     _Py_IDENTIFIER(__trunc__);
 
     if (o == NULL)
@@ -1288,21 +1260,22 @@
     if (PyErr_Occurred())
         return NULL;
 
-    if (PyBytes_Check(o))
+    if (PyUnicode_Check(o))
+        /* The below check is done in PyLong_FromUnicode(). */
+        return PyLong_FromUnicodeObject(o, 10);
+
+    if (PyObject_GetBuffer(o, &view, PyBUF_SIMPLE) == 0) {
         /* need to do extra error checking that PyLong_FromString()
          * doesn't do.  In particular int('9\x005') must raise an
          * exception, not truncate at the null.
          */
-        return _PyLong_FromBytes(PyBytes_AS_STRING(o),
-                                 PyBytes_GET_SIZE(o), 10);
-    if (PyUnicode_Check(o))
-        /* The above check is done in PyLong_FromUnicode(). */
-        return PyLong_FromUnicodeObject(o, 10);
-    if (!PyObject_AsCharBuffer(o, &buffer, &buffer_len))
-        return _PyLong_FromBytes(buffer, buffer_len, 10);
+        PyObject *result = _PyLong_FromBytes(view.buf, view.len, 10);
+        PyBuffer_Release(&view);
+        return result;
+    }
 
-    return type_error("int() argument must be a string or a "
-                      "number, not '%.200s'", o);
+    return type_error("int() argument must be a string, a bytes-like object "
+                      "or a number, not '%.200s'", o);
 }
 
 PyObject *
diff --git a/Objects/bytearrayobject.c b/Objects/bytearrayobject.c
--- a/Objects/bytearrayobject.c
+++ b/Objects/bytearrayobject.c
@@ -74,24 +74,6 @@
     obj->ob_exports--;
 }
 
-static Py_ssize_t
-_getbuffer(PyObject *obj, Py_buffer *view)
-{
-    PyBufferProcs *buffer = Py_TYPE(obj)->tp_as_buffer;
-
-    if (buffer == NULL || buffer->bf_getbuffer == NULL)
-    {
-        PyErr_Format(PyExc_TypeError,
-                     "Type %.100s doesn't support the buffer API",
-                     Py_TYPE(obj)->tp_name);
-        return -1;
-    }
-
-    if (buffer->bf_getbuffer(obj, view, PyBUF_SIMPLE) < 0)
-            return -1;
-    return view->len;
-}
-
 static int
 _canresize(PyByteArrayObject *self)
 {
@@ -262,8 +244,8 @@
 
     va.len = -1;
     vb.len = -1;
-    if (_getbuffer(a, &va) < 0  ||
-        _getbuffer(b, &vb) < 0) {
+    if (PyObject_GetBuffer(a, &va, PyBUF_SIMPLE) != 0 ||
+        PyObject_GetBuffer(b, &vb, PyBUF_SIMPLE) != 0) {
             PyErr_Format(PyExc_TypeError, "can't concat %.100s to %.100s",
                          Py_TYPE(a)->tp_name, Py_TYPE(b)->tp_name);
             goto done;
@@ -304,7 +286,7 @@
     Py_ssize_t size;
     Py_buffer vo;
 
-    if (_getbuffer(other, &vo) < 0) {
+    if (PyObject_GetBuffer(other, &vo, PyBUF_SIMPLE) != 0) {
         PyErr_Format(PyExc_TypeError, "can't concat %.100s to %.100s",
                      Py_TYPE(other)->tp_name, Py_TYPE(self)->tp_name);
         return NULL;
@@ -562,14 +544,14 @@
         needed = 0;
     }
     else {
-            if (_getbuffer(values, &vbytes) < 0) {
-                    PyErr_Format(PyExc_TypeError,
-                                 "can't set bytearray slice from %.100s",
-                                 Py_TYPE(values)->tp_name);
-                    return -1;
-            }
-            needed = vbytes.len;
-            bytes = vbytes.buf;
+        if (PyObject_GetBuffer(values, &vbytes, PyBUF_SIMPLE) != 0) {
+            PyErr_Format(PyExc_TypeError,
+                         "can't set bytearray slice from %.100s",
+                         Py_TYPE(values)->tp_name);
+            return -1;
+        }
+        needed = vbytes.len;
+        bytes = vbytes.buf;
     }
 
     if (lo < 0)
@@ -1012,18 +994,18 @@
         Py_RETURN_NOTIMPLEMENTED;
     }
 
-    self_size = _getbuffer(self, &self_bytes);
-    if (self_size < 0) {
+    if (PyObject_GetBuffer(self, &self_bytes, PyBUF_SIMPLE) != 0) {
         PyErr_Clear();
         Py_RETURN_NOTIMPLEMENTED;
     }
-
-    other_size = _getbuffer(other, &other_bytes);
-    if (other_size < 0) {
+    self_size = self_bytes.len;
+
+    if (PyObject_GetBuffer(other, &other_bytes, PyBUF_SIMPLE) != 0) {
         PyErr_Clear();
         PyBuffer_Release(&self_bytes);
         Py_RETURN_NOTIMPLEMENTED;
     }
+    other_size = other_bytes.len;
 
     if (self_size != other_size && (op == Py_EQ || op == Py_NE)) {
         /* Shortcut: if the lengths differ, the objects differ */
@@ -1135,7 +1117,7 @@
         return -2;
 
     if (subobj) {
-        if (_getbuffer(subobj, &subbuf) < 0)
+        if (PyObject_GetBuffer(subobj, &subbuf, PyBUF_SIMPLE) != 0)
             return -2;
 
         sub = subbuf.buf;
@@ -1203,7 +1185,7 @@
         return NULL;
 
     if (sub_obj) {
-        if (_getbuffer(sub_obj, &vsub) < 0)
+        if (PyObject_GetBuffer(sub_obj, &vsub, PyBUF_SIMPLE) != 0)
             return NULL;
 
         sub = vsub.buf;
@@ -1318,7 +1300,7 @@
         Py_buffer varg;
         Py_ssize_t pos;
         PyErr_Clear();
-        if (_getbuffer(arg, &varg) < 0)
+        if (PyObject_GetBuffer(arg, &varg, PyBUF_SIMPLE) != 0)
             return -1;
         pos = stringlib_find(PyByteArray_AS_STRING(self), Py_SIZE(self),
                              varg.buf, varg.len, 0);
@@ -1349,7 +1331,7 @@
 
     str = PyByteArray_AS_STRING(self);
 
-    if (_getbuffer(substr, &vsubstr) < 0)
+    if (PyObject_GetBuffer(substr, &vsubstr, PyBUF_SIMPLE) != 0)
         return -1;
 
     ADJUST_INDICES(start, end, len);
@@ -1493,7 +1475,7 @@
     if (tableobj == Py_None) {
         table = NULL;
         tableobj = NULL;
-    } else if (_getbuffer(tableobj, &vtable) < 0) {
+    } else if (PyObject_GetBuffer(tableobj, &vtable, PyBUF_SIMPLE) != 0) {
         return NULL;
     } else {
         if (vtable.len != 256) {
@@ -1506,7 +1488,7 @@
     }
 
     if (delobj != NULL) {
-        if (_getbuffer(delobj, &vdel) < 0) {
+        if (PyObject_GetBuffer(delobj, &vdel, PyBUF_SIMPLE) != 0) {
             if (tableobj != NULL)
                 PyBuffer_Release(&vtable);
             return NULL;
@@ -2070,26 +2052,20 @@
 static PyObject *
 bytearray_replace(PyByteArrayObject *self, PyObject *args)
 {
+    PyObject *res;
+    Py_buffer old = {NULL, NULL};
+    Py_buffer new = {NULL, NULL};
     Py_ssize_t count = -1;
-    PyObject *from, *to, *res;
-    Py_buffer vfrom, vto;
-
-    if (!PyArg_ParseTuple(args, "OO|n:replace", &from, &to, &count))
+
+    if (!PyArg_ParseTuple(args, "y*y*|n:replace", &old, &new, &count))
         return NULL;
 
-    if (_getbuffer(from, &vfrom) < 0)
-        return NULL;
-    if (_getbuffer(to, &vto) < 0) {
-        PyBuffer_Release(&vfrom);
-        return NULL;
-    }
-
     res = (PyObject *)replace((PyByteArrayObject *) self,
-                              vfrom.buf, vfrom.len,
-                              vto.buf, vto.len, count);
-
-    PyBuffer_Release(&vfrom);
-    PyBuffer_Release(&vto);
+                              (const char *)old.buf, old.len,
+                              (const char *)new.buf, new.len, count);
+
+    PyBuffer_Release(&old);
+    PyBuffer_Release(&new);
     return res;
 }
 
@@ -2120,7 +2096,7 @@
     if (subobj == Py_None)
         return stringlib_split_whitespace((PyObject*) self, s, len, maxsplit);
 
-    if (_getbuffer(subobj, &vsub) < 0)
+    if (PyObject_GetBuffer(subobj, &vsub, PyBUF_SIMPLE) != 0)
         return NULL;
     sub = vsub.buf;
     n = vsub.len;
@@ -2215,7 +2191,7 @@
     if (subobj == Py_None)
         return stringlib_rsplit_whitespace((PyObject*) self, s, len, maxsplit);
 
-    if (_getbuffer(subobj, &vsub) < 0)
+    if (PyObject_GetBuffer(subobj, &vsub, PyBUF_SIMPLE) != 0)
         return NULL;
     sub = vsub.buf;
     n = vsub.len;
@@ -2503,7 +2479,7 @@
         argsize = 6;
     }
     else {
-        if (_getbuffer(arg, &varg) < 0)
+        if (PyObject_GetBuffer(arg, &varg, PyBUF_SIMPLE) != 0)
             return NULL;
         argptr = (char *) varg.buf;
         argsize = varg.len;
@@ -2540,7 +2516,7 @@
         argsize = 6;
     }
     else {
-        if (_getbuffer(arg, &varg) < 0)
+        if (PyObject_GetBuffer(arg, &varg, PyBUF_SIMPLE) != 0)
             return NULL;
         argptr = (char *) varg.buf;
         argsize = varg.len;
@@ -2574,7 +2550,7 @@
         argsize = 6;
     }
     else {
-        if (_getbuffer(arg, &varg) < 0)
+        if (PyObject_GetBuffer(arg, &varg, PyBUF_SIMPLE) != 0)
             return NULL;
         argptr = (char *) varg.buf;
         argsize = varg.len;
diff --git a/Objects/bytes_methods.c b/Objects/bytes_methods.c
--- a/Objects/bytes_methods.c
+++ b/Objects/bytes_methods.c
@@ -363,41 +363,20 @@
 in frm is mapped to the byte at the same position in to.\n\
 The bytes objects frm and to must be of the same length.");
 
-static Py_ssize_t
-_getbuffer(PyObject *obj, Py_buffer *view)
-{
-    PyBufferProcs *buffer = Py_TYPE(obj)->tp_as_buffer;
-
-    if (buffer == NULL || buffer->bf_getbuffer == NULL)
-    {
-        PyErr_Format(PyExc_TypeError,
-                     "Type %.100s doesn't support the buffer API",
-                     Py_TYPE(obj)->tp_name);
-        return -1;
-    }
-
-    if (buffer->bf_getbuffer(obj, view, PyBUF_SIMPLE) < 0)
-        return -1;
-    return view->len;
-}
-
 PyObject *
 _Py_bytes_maketrans(PyObject *args)
 {
-    PyObject *frm, *to, *res = NULL;
-    Py_buffer bfrm, bto;
+    PyObject *res = NULL;
+    Py_buffer bfrm = {NULL, NULL};
+    Py_buffer bto = {NULL, NULL};
     Py_ssize_t i;
     char *p;
 
     bfrm.len = -1;
     bto.len = -1;
 
-    if (!PyArg_ParseTuple(args, "OO:maketrans", &frm, &to))
+    if (!PyArg_ParseTuple(args, "y*y*:maketrans", &bfrm, &bto))
         return NULL;
-    if (_getbuffer(frm, &bfrm) < 0)
-        return NULL;
-    if (_getbuffer(to, &bto) < 0)
-        goto done;
     if (bfrm.len != bto.len) {
         PyErr_Format(PyExc_ValueError,
                      "maketrans arguments must have same length");
@@ -415,9 +394,9 @@
     }
 
 done:
-    if (bfrm.len != -1)
+    if (bfrm.obj != NULL)
         PyBuffer_Release(&bfrm);
-    if (bto.len != -1)
+    if (bfrm.obj != NULL)
         PyBuffer_Release(&bto);
     return res;
 }
diff --git a/Objects/bytesobject.c b/Objects/bytesobject.c
--- a/Objects/bytesobject.c
+++ b/Objects/bytesobject.c
@@ -7,33 +7,6 @@
 #include "bytes_methods.h"
 #include <stddef.h>
 
-static Py_ssize_t
-_getbuffer(PyObject *obj, Py_buffer *view)
-{
-    PyBufferProcs *bufferprocs;
-    if (PyBytes_CheckExact(obj)) {
-        /* Fast path, e.g. for .join() of many bytes objects */
-        Py_INCREF(obj);
-        view->obj = obj;
-        view->buf = PyBytes_AS_STRING(obj);
-        view->len = PyBytes_GET_SIZE(obj);
-        return view->len;
-    }
-
-    bufferprocs = Py_TYPE(obj)->tp_as_buffer;
-    if (bufferprocs == NULL || bufferprocs->bf_getbuffer == NULL)
-    {
-        PyErr_Format(PyExc_TypeError,
-                     "Type %.100s doesn't support the buffer API",
-                     Py_TYPE(obj)->tp_name);
-        return -1;
-    }
-
-    if (bufferprocs->bf_getbuffer(obj, view, PyBUF_SIMPLE) < 0)
-        return -1;
-    return view->len;
-}
-
 #ifdef COUNT_ALLOCS
 Py_ssize_t null_strings, one_strings;
 #endif
@@ -695,8 +668,8 @@
 
     va.len = -1;
     vb.len = -1;
-    if (_getbuffer(a, &va) < 0  ||
-        _getbuffer(b, &vb) < 0) {
+    if (PyObject_GetBuffer(a, &va, PyBUF_SIMPLE) != 0 ||
+        PyObject_GetBuffer(b, &vb, PyBUF_SIMPLE) != 0) {
         PyErr_Format(PyExc_TypeError, "can't concat %.100s to %.100s",
                      Py_TYPE(a)->tp_name, Py_TYPE(b)->tp_name);
         goto done;
@@ -794,7 +767,7 @@
         Py_buffer varg;
         Py_ssize_t pos;
         PyErr_Clear();
-        if (_getbuffer(arg, &varg) < 0)
+        if (PyObject_GetBuffer(arg, &varg, PyBUF_SIMPLE) != 0)
             return -1;
         pos = stringlib_find(PyBytes_AS_STRING(self), Py_SIZE(self),
                              varg.buf, varg.len, 0);
@@ -1048,7 +1021,7 @@
         maxsplit = PY_SSIZE_T_MAX;
     if (subobj == Py_None)
         return stringlib_split_whitespace((PyObject*) self, s, len, maxsplit);
-    if (_getbuffer(subobj, &vsub) < 0)
+    if (PyObject_GetBuffer(subobj, &vsub, PyBUF_SIMPLE) != 0)
         return NULL;
     sub = vsub.buf;
     n = vsub.len;
@@ -1068,21 +1041,19 @@
 static PyObject *
 bytes_partition(PyBytesObject *self, PyObject *sep_obj)
 {
-    const char *sep;
-    Py_ssize_t sep_len;
-
-    if (PyBytes_Check(sep_obj)) {
-        sep = PyBytes_AS_STRING(sep_obj);
-        sep_len = PyBytes_GET_SIZE(sep_obj);
-    }
-    else if (PyObject_AsCharBuffer(sep_obj, &sep, &sep_len))
+    Py_buffer sep = {NULL, NULL};
+    PyObject *res;
+
+    if (PyObject_GetBuffer(sep_obj, &sep, PyBUF_SIMPLE) != 0)
         return NULL;
 
-    return stringlib_partition(
+    res = stringlib_partition(
         (PyObject*) self,
         PyBytes_AS_STRING(self), PyBytes_GET_SIZE(self),
-        sep_obj, sep, sep_len
+        sep_obj, sep.buf, sep.len
         );
+    PyBuffer_Release(&sep);
+    return res;
 }
 
 PyDoc_STRVAR(rpartition__doc__,
@@ -1096,21 +1067,19 @@
 static PyObject *
 bytes_rpartition(PyBytesObject *self, PyObject *sep_obj)
 {
-    const char *sep;
-    Py_ssize_t sep_len;
-
-    if (PyBytes_Check(sep_obj)) {
-        sep = PyBytes_AS_STRING(sep_obj);
-        sep_len = PyBytes_GET_SIZE(sep_obj);
-    }
-    else if (PyObject_AsCharBuffer(sep_obj, &sep, &sep_len))
+    Py_buffer sep = {NULL, NULL};
+    PyObject *res;
+
+    if (PyObject_GetBuffer(sep_obj, &sep, PyBUF_SIMPLE) != 0)
         return NULL;
 
-    return stringlib_rpartition(
+    res = stringlib_rpartition(
         (PyObject*) self,
         PyBytes_AS_STRING(self), PyBytes_GET_SIZE(self),
-        sep_obj, sep, sep_len
+        sep_obj, sep.buf, sep.len
         );
+    PyBuffer_Release(&sep);
+    return res;
 }
 
 PyDoc_STRVAR(rsplit__doc__,
@@ -1140,7 +1109,7 @@
         maxsplit = PY_SSIZE_T_MAX;
     if (subobj == Py_None)
         return stringlib_rsplit_whitespace((PyObject*) self, s, len, maxsplit);
-    if (_getbuffer(subobj, &vsub) < 0)
+    if (PyObject_GetBuffer(subobj, &vsub, PyBUF_SIMPLE) != 0)
         return NULL;
     sub = vsub.buf;
     n = vsub.len;
@@ -1202,7 +1171,7 @@
         return -2;
 
     if (subobj) {
-        if (_getbuffer(subobj, &subbuf) < 0)
+        if (PyObject_GetBuffer(subobj, &subbuf, PyBUF_SIMPLE) != 0)
             return -2;
 
         sub = subbuf.buf;
@@ -1317,7 +1286,7 @@
     Py_ssize_t seplen;
     Py_ssize_t i, j;
 
-    if (_getbuffer(sepobj, &vsep) < 0)
+    if (PyObject_GetBuffer(sepobj, &vsep, PyBUF_SIMPLE) != 0)
         return NULL;
     sep = vsep.buf;
     seplen = vsep.len;
@@ -1462,7 +1431,7 @@
         return NULL;
 
     if (sub_obj) {
-        if (_getbuffer(sub_obj, &vsub) < 0)
+        if (PyObject_GetBuffer(sub_obj, &vsub, PyBUF_SIMPLE) != 0)
             return NULL;
 
         sub = vsub.buf;
@@ -1498,6 +1467,8 @@
 bytes_translate(PyBytesObject *self, PyObject *args)
 {
     char *input, *output;
+    Py_buffer table_view = {NULL, NULL};
+    Py_buffer del_table_view = {NULL, NULL};
     const char *table;
     Py_ssize_t i, c, changed = 0;
     PyObject *input_obj = (PyObject*)self;
@@ -1519,12 +1490,17 @@
         table = NULL;
         tablen = 256;
     }
-    else if (PyObject_AsCharBuffer(tableobj, &table, &tablen))
-        return NULL;
+    else {
+        if (PyObject_GetBuffer(tableobj, &table_view, PyBUF_SIMPLE) != 0)
+            return NULL;
+        table = table_view.buf;
+        tablen = table_view.len;
+    }
 
     if (tablen != 256) {
         PyErr_SetString(PyExc_ValueError,
           "translation table must be 256 characters long");
+        PyBuffer_Release(&table_view);
         return NULL;
     }
 
@@ -1533,8 +1509,14 @@
             del_table = PyBytes_AS_STRING(delobj);
             dellen = PyBytes_GET_SIZE(delobj);
         }
-        else if (PyObject_AsCharBuffer(delobj, &del_table, &dellen))
-            return NULL;
+        else {
+            if (PyObject_GetBuffer(delobj, &del_table_view, PyBUF_SIMPLE) != 0) {
+                PyBuffer_Release(&table_view);
+                return NULL;
+            }
+            del_table = del_table_view.buf;
+            dellen = del_table_view.len;
+        }
     }
     else {
         del_table = NULL;
@@ -1543,8 +1525,11 @@
 
     inlen = PyBytes_GET_SIZE(input_obj);
     result = PyBytes_FromStringAndSize((char *)NULL, inlen);
-    if (result == NULL)
+    if (result == NULL) {
+        PyBuffer_Release(&del_table_view);
+        PyBuffer_Release(&table_view);
         return NULL;
+    }
     output_start = output = PyBytes_AsString(result);
     input = PyBytes_AS_STRING(input_obj);
 
@@ -1555,11 +1540,14 @@
             if (Py_CHARMASK((*output++ = table[c])) != c)
                 changed = 1;
         }
-        if (changed || !PyBytes_CheckExact(input_obj))
-            return result;
-        Py_DECREF(result);
-        Py_INCREF(input_obj);
-        return input_obj;
+        if (!changed && PyBytes_CheckExact(input_obj)) {
+            Py_INCREF(input_obj);
+            Py_DECREF(result);
+            result = input_obj;
+        }
+        PyBuffer_Release(&del_table_view);
+        PyBuffer_Release(&table_view);
+        return result;
     }
 
     if (table == NULL) {
@@ -1569,9 +1557,11 @@
         for (i = 0; i < 256; i++)
             trans_table[i] = Py_CHARMASK(table[i]);
     }
+    PyBuffer_Release(&table_view);
 
     for (i = 0; i < dellen; i++)
         trans_table[(int) Py_CHARMASK(del_table[i])] = -1;
+    PyBuffer_Release(&del_table_view);
 
     for (i = inlen; --i >= 0; ) {
         c = Py_CHARMASK(*input++);
@@ -2100,31 +2090,21 @@
 static PyObject *
 bytes_replace(PyBytesObject *self, PyObject *args)
 {
+    PyObject *res;
+    Py_buffer old = {NULL, NULL};
+    Py_buffer new = {NULL, NULL};
     Py_ssize_t count = -1;
-    PyObject *from, *to;
-    const char *from_s, *to_s;
-    Py_ssize_t from_len, to_len;
-
-    if (!PyArg_ParseTuple(args, "OO|n:replace", &from, &to, &count))
+
+    if (!PyArg_ParseTuple(args, "y*y*|n:replace", &old, &new, &count))
         return NULL;
 
-    if (PyBytes_Check(from)) {
-        from_s = PyBytes_AS_STRING(from);
-        from_len = PyBytes_GET_SIZE(from);
-    }
-    else if (PyObject_AsCharBuffer(from, &from_s, &from_len))
-        return NULL;
-
-    if (PyBytes_Check(to)) {
-        to_s = PyBytes_AS_STRING(to);
-        to_len = PyBytes_GET_SIZE(to);
-    }
-    else if (PyObject_AsCharBuffer(to, &to_s, &to_len))
-        return NULL;
-
-    return (PyObject *)replace((PyBytesObject *) self,
-                               from_s, from_len,
-                               to_s, to_len, count);
+    res = (PyObject *)replace((PyBytesObject *) self,
+                              (const char *)old.buf, old.len,
+                              (const char *)new.buf, new.len, count);
+
+    PyBuffer_Release(&old);
+    PyBuffer_Release(&new);
+    return res;
 }
 
 /** End DALKE **/
@@ -2139,6 +2119,7 @@
 {
     Py_ssize_t len = PyBytes_GET_SIZE(self);
     Py_ssize_t slen;
+    Py_buffer sub_view = {NULL, NULL};
     const char* sub;
     const char* str;
 
@@ -2146,8 +2127,12 @@
         sub = PyBytes_AS_STRING(substr);
         slen = PyBytes_GET_SIZE(substr);
     }
-    else if (PyObject_AsCharBuffer(substr, &sub, &slen))
-        return -1;
+    else {
+        if (PyObject_GetBuffer(substr, &sub_view, PyBUF_SIMPLE) != 0)
+            return -1;
+        sub = sub_view.buf;
+        slen = sub_view.len;
+    }
     str = PyBytes_AS_STRING(self);
 
     ADJUST_INDICES(start, end, len);
@@ -2155,17 +2140,25 @@
     if (direction < 0) {
         /* startswith */
         if (start+slen > len)
-            return 0;
+            goto notfound;
     } else {
         /* endswith */
         if (end-start < slen || start > len)
-            return 0;
+            goto notfound;
 
         if (end-slen > start)
             start = end - slen;
     }
-    if (end-start >= slen)
-        return ! memcmp(str+start, sub, slen);
+    if (end-start < slen)
+        goto notfound;
+    if (memcmp(str+start, sub, slen) != 0)
+        goto notfound;
+
+    PyBuffer_Release(&sub_view);
+    return 1;
+
+notfound:
+    PyBuffer_Release(&sub_view);
     return 0;
 }
 
diff --git a/Objects/complexobject.c b/Objects/complexobject.c
--- a/Objects/complexobject.c
+++ b/Objects/complexobject.c
@@ -767,6 +767,7 @@
     int got_bracket=0;
     PyObject *s_buffer = NULL;
     Py_ssize_t len;
+    Py_buffer view = {NULL, NULL};
 
     if (PyUnicode_Check(v)) {
         s_buffer = _PyUnicode_TransformDecimalAndSpaceToASCII(v);
@@ -776,7 +777,11 @@
         if (s == NULL)
             goto error;
     }
-    else if (PyObject_AsCharBuffer(v, &s, &len)) {
+    else if (PyObject_GetBuffer(v, &view, PyBUF_SIMPLE) == 0) {
+        s = (const char *)view.buf;
+        len = view.len;
+    }
+    else {
         PyErr_Format(PyExc_TypeError,
             "complex() argument must be a string or a number, not '%.200s'",
             Py_TYPE(v)->tp_name);
@@ -890,6 +895,7 @@
     if (s-start != len)
         goto parse_error;
 
+    PyBuffer_Release(&view);
     Py_XDECREF(s_buffer);
     return complex_subtype_from_doubles(type, x, y);
 
@@ -897,6 +903,7 @@
     PyErr_SetString(PyExc_ValueError,
                     "complex() arg is a malformed string");
   error:
+    PyBuffer_Release(&view);
     Py_XDECREF(s_buffer);
     return NULL;
 }
diff --git a/Objects/exceptions.c b/Objects/exceptions.c
--- a/Objects/exceptions.c
+++ b/Objects/exceptions.c
@@ -1922,8 +1922,6 @@
 UnicodeDecodeError_init(PyObject *self, PyObject *args, PyObject *kwds)
 {
     PyUnicodeErrorObject *ude;
-    const char *data;
-    Py_ssize_t size;
 
     if (BaseException_init((PyBaseExceptionObject *)self, args, kwds) == -1)
         return -1;
@@ -1944,21 +1942,27 @@
              return -1;
     }
 
+    Py_INCREF(ude->encoding);
+    Py_INCREF(ude->object);
+    Py_INCREF(ude->reason);
+
     if (!PyBytes_Check(ude->object)) {
-        if (PyObject_AsReadBuffer(ude->object, (const void **)&data, &size)) {
-            ude->encoding = ude->object = ude->reason = NULL;
-            return -1;
-        }
-        ude->object = PyBytes_FromStringAndSize(data, size);
+        Py_buffer view;
+        if (PyObject_GetBuffer(ude->object, &view, PyBUF_SIMPLE) != 0)
+            goto error;
+        Py_CLEAR(ude->object);
+        ude->object = PyBytes_FromStringAndSize(view.buf, view.len);
+        PyBuffer_Release(&view);
+        if (!ude->object)
+            goto error;
     }
-    else {
-        Py_INCREF(ude->object);
-    }
-
-    Py_INCREF(ude->encoding);
-    Py_INCREF(ude->reason);
-
     return 0;
+
+error:
+    Py_CLEAR(ude->encoding);
+    Py_CLEAR(ude->object);
+    Py_CLEAR(ude->reason);
+    return -1;
 }
 
 static PyObject *
diff --git a/Objects/floatobject.c b/Objects/floatobject.c
--- a/Objects/floatobject.c
+++ b/Objects/floatobject.c
@@ -131,6 +131,7 @@
     double x;
     PyObject *s_buffer = NULL;
     Py_ssize_t len;
+    Py_buffer view = {NULL, NULL};
     PyObject *result = NULL;
 
     if (PyUnicode_Check(v)) {
@@ -143,7 +144,11 @@
             return NULL;
         }
     }
-    else if (PyObject_AsCharBuffer(v, &s, &len)) {
+    else if (PyObject_GetBuffer(v, &view, PyBUF_SIMPLE) == 0) {
+        s = (const char *)view.buf;
+        len = view.len;
+    }
+    else {
         PyErr_Format(PyExc_TypeError,
             "float() argument must be a string or a number, not '%.200s'",
             Py_TYPE(v)->tp_name);
@@ -170,6 +175,7 @@
     else
         result = PyFloat_FromDouble(x);
 
+    PyBuffer_Release(&view);
     Py_XDECREF(s_buffer);
     return result;
 }
diff --git a/Objects/stringlib/join.h b/Objects/stringlib/join.h
--- a/Objects/stringlib/join.h
+++ b/Objects/stringlib/join.h
@@ -58,7 +58,14 @@
     for (i = 0, nbufs = 0; i < seqlen; i++) {
         Py_ssize_t itemlen;
         item = PySequence_Fast_GET_ITEM(seq, i);
-        if (_getbuffer(item, &buffers[i]) < 0) {
+        if (PyBytes_CheckExact(item)) {
+            /* Fast path. */
+            Py_INCREF(item);
+            buffers[i].obj = item;
+            buffers[i].buf = PyBytes_AS_STRING(item);
+            buffers[i].len = PyBytes_GET_SIZE(item);
+        }
+        else if (PyObject_GetBuffer(item, &buffers[i], PyBUF_SIMPLE) != 0) {
             PyErr_Format(PyExc_TypeError,
                          "sequence item %zd: expected a bytes-like object, "
                          "%.80s found",
diff --git a/Python/bltinmodule.c b/Python/bltinmodule.c
--- a/Python/bltinmodule.c
+++ b/Python/bltinmodule.c
@@ -559,10 +559,10 @@
 Return a Unicode string of one character with ordinal i; 0 <= i <= 0x10ffff.");
 
 
-static char *
-source_as_string(PyObject *cmd, char *funcname, char *what, PyCompilerFlags *cf)
+static const char *
+source_as_string(PyObject *cmd, const char *funcname, const char *what, PyCompilerFlags *cf, Py_buffer *view)
 {
-    char *str;
+    const char *str;
     Py_ssize_t size;
 
     if (PyUnicode_Check(cmd)) {
@@ -571,19 +571,21 @@
         if (str == NULL)
             return NULL;
     }
-    else if (!PyObject_CheckReadBuffer(cmd)) {
+    else if (PyObject_GetBuffer(cmd, view, PyBUF_SIMPLE) == 0) {
+        str = (const char *)view->buf;
+        size = view->len;
+    }
+    else {
         PyErr_Format(PyExc_TypeError,
           "%s() arg 1 must be a %s object",
           funcname, what);
         return NULL;
     }
-    else if (PyObject_AsReadBuffer(cmd, (const void **)&str, &size) < 0) {
-        return NULL;
-    }
 
     if (strlen(str) != size) {
         PyErr_SetString(PyExc_TypeError,
                         "source code string cannot contain null bytes");
+        PyBuffer_Release(view);
         return NULL;
     }
     return str;
@@ -592,7 +594,8 @@
 static PyObject *
 builtin_compile(PyObject *self, PyObject *args, PyObject *kwds)
 {
-    char *str;
+    Py_buffer view = {NULL, NULL};
+    const char *str;
     PyObject *filename;
     char *startstr;
     int mode = -1;
@@ -678,11 +681,12 @@
         goto finally;
     }
 
-    str = source_as_string(cmd, "compile", "string, bytes or AST", &cf);
+    str = source_as_string(cmd, "compile", "string, bytes or AST", &cf, &view);
     if (str == NULL)
         goto error;
 
     result = Py_CompileStringObject(str, filename, start[mode], &cf, optimize);
+    PyBuffer_Release(&view);
     goto finally;
 
 error:
@@ -752,7 +756,8 @@
 {
     PyObject *cmd, *result, *tmp = NULL;
     PyObject *globals = Py_None, *locals = Py_None;
-    char *str;
+    Py_buffer view = {NULL, NULL};
+    const char *str;
     PyCompilerFlags cf;
 
     if (!PyArg_UnpackTuple(args, "eval", 1, 3, &cmd, &globals, &locals))
@@ -801,7 +806,7 @@
     }
 
     cf.cf_flags = PyCF_SOURCE_IS_UTF8;
-    str = source_as_string(cmd, "eval", "string, bytes or code", &cf);
+    str = source_as_string(cmd, "eval", "string, bytes or code", &cf, &view);
     if (str == NULL)
         return NULL;
 
@@ -810,6 +815,7 @@
 
     (void)PyEval_MergeCompilerFlags(&cf);
     result = PyRun_StringFlags(str, Py_eval_input, globals, locals, &cf);
+    PyBuffer_Release(&view);
     Py_XDECREF(tmp);
     return result;
 }
@@ -876,11 +882,12 @@
         v = PyEval_EvalCode(prog, globals, locals);
     }
     else {
-        char *str;
+        Py_buffer view = {NULL, NULL};
+        const char *str;
         PyCompilerFlags cf;
         cf.cf_flags = PyCF_SOURCE_IS_UTF8;
         str = source_as_string(prog, "exec",
-                                     "string, bytes or code", &cf);
+                                     "string, bytes or code", &cf, &view);
         if (str == NULL)
             return NULL;
         if (PyEval_MergeCompilerFlags(&cf))
@@ -888,6 +895,7 @@
                                   locals, &cf);
         else
             v = PyRun_String(str, Py_file_input, globals, locals);
+        PyBuffer_Release(&view);
     }
     if (v == NULL)
         return NULL;

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Tue Feb  3 07:44:50 2015
From: python-checkins at python.org (berker.peksag)
Date: Tue, 03 Feb 2015 06:44:50 +0000
Subject: [Python-checkins] =?utf-8?q?peps=3A_PEP_328=3A_Fix_a_broken_link?=
	=?utf-8?q?=2E?=
Message-ID: <20150203064444.39274.19498@psf.io>

https://hg.python.org/peps/rev/8e6299fb8454
changeset:   5691:8e6299fb8454
user:        Berker Peksag <berker.peksag at gmail.com>
date:        Tue Feb 03 08:44:51 2015 +0200
summary:
  PEP 328: Fix a broken link.

files:
  pep-0328.txt |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/pep-0328.txt b/pep-0328.txt
--- a/pep-0328.txt
+++ b/pep-0328.txt
@@ -315,7 +315,7 @@
 
 .. [1] http://mail.python.org/pipermail/python-dev/2004-March/043739.html
 
-.. [2] http://www.python.org/doc/essays/packages.html
+.. [2] https://www.python.org/doc/essays/packages/
 
 
 Copyright

-- 
Repository URL: https://hg.python.org/peps

From python-checkins at python.org  Tue Feb  3 08:31:44 2015
From: python-checkins at python.org (serhiy.storchaka)
Date: Tue, 03 Feb 2015 07:31:44 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?q?=29=3A_Issue_=2323099=3A_Closing_io=2EBytesIO_with_exported_buf?=
 =?utf-8?q?fer_is_rejected_now_to?=
Message-ID: <20150203073130.25847.70451@psf.io>

https://hg.python.org/cpython/rev/b9d4c013b09a
changeset:   94481:b9d4c013b09a
parent:      94479:4cb316fe6bf2
parent:      94480:e62d54128bd3
user:        Serhiy Storchaka <storchaka at gmail.com>
date:        Tue Feb 03 09:30:51 2015 +0200
summary:
  Issue #23099: Closing io.BytesIO with exported buffer is rejected now to
prevent corrupting exported buffer.

files:
  Doc/library/io.rst        |  13 +++++++------
  Lib/_pyio.py              |   6 ++++++
  Lib/test/test_memoryio.py |   7 ++++++-
  Misc/NEWS                 |   3 +++
  Modules/_io/bytesio.c     |   1 +
  5 files changed, 23 insertions(+), 7 deletions(-)


diff --git a/Doc/library/io.rst b/Doc/library/io.rst
--- a/Doc/library/io.rst
+++ b/Doc/library/io.rst
@@ -578,7 +578,8 @@
 .. class:: BytesIO([initial_bytes])
 
    A stream implementation using an in-memory bytes buffer.  It inherits
-   :class:`BufferedIOBase`.
+   :class:`BufferedIOBase`.  The buffer is discarded when the
+   :meth:`~IOBase.close` method is called.
 
    The argument *initial_bytes* contains optional initial :class:`bytes` data.
 
@@ -599,7 +600,7 @@
 
       .. note::
          As long as the view exists, the :class:`BytesIO` object cannot be
-         resized.
+         resized or closed.
 
       .. versionadded:: 3.2
 
@@ -607,6 +608,7 @@
 
       Return :class:`bytes` containing the entire contents of the buffer.
 
+
    .. method:: read1()
 
       In :class:`BytesIO`, this is the same as :meth:`read`.
@@ -880,7 +882,8 @@
 
 .. class:: StringIO(initial_value='', newline='\\n')
 
-   An in-memory stream for text I/O.
+   An in-memory stream for text I/O.  The text buffer is discarded when the
+   :meth:`~IOBase.close` method is called.
 
    The initial value of the buffer (an empty string by default) can be set by
    providing *initial_value*.  The *newline* argument works like that of
@@ -892,9 +895,7 @@
 
    .. method:: getvalue()
 
-      Return a ``str`` containing the entire contents of the buffer at any
-      time before the :class:`StringIO` object's :meth:`close` method is
-      called.
+      Return a ``str`` containing the entire contents of the buffer.
 
    Example usage::
 
diff --git a/Lib/_pyio.py b/Lib/_pyio.py
--- a/Lib/_pyio.py
+++ b/Lib/_pyio.py
@@ -852,8 +852,14 @@
     def getbuffer(self):
         """Return a readable and writable view of the buffer.
         """
+        if self.closed:
+            raise ValueError("getbuffer on closed file")
         return memoryview(self._buffer)
 
+    def close(self):
+        self._buffer.clear()
+        super().close()
+
     def read(self, size=None):
         if self.closed:
             raise ValueError("read from closed file")
diff --git a/Lib/test/test_memoryio.py b/Lib/test/test_memoryio.py
--- a/Lib/test/test_memoryio.py
+++ b/Lib/test/test_memoryio.py
@@ -399,14 +399,19 @@
         # raises a BufferError.
         self.assertRaises(BufferError, memio.write, b'x' * 100)
         self.assertRaises(BufferError, memio.truncate)
+        self.assertRaises(BufferError, memio.close)
+        self.assertFalse(memio.closed)
         # Mutating the buffer updates the BytesIO
         buf[3:6] = b"abc"
         self.assertEqual(bytes(buf), b"123abc7890")
         self.assertEqual(memio.getvalue(), b"123abc7890")
-        # After the buffer gets released, we can resize the BytesIO again
+        # After the buffer gets released, we can resize and close the BytesIO
+        # again
         del buf
         support.gc_collect()
         memio.truncate()
+        memio.close()
+        self.assertRaises(ValueError, memio.getbuffer)
 
 
 class PyBytesIOTest(MemoryTestMixin, MemorySeekTestMixin,
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -232,6 +232,9 @@
 Library
 -------
 
+- Issue #23099: Closing io.BytesIO with exported buffer is rejected now to
+  prevent corrupting exported buffer.
+
 - Issue #23326: Removed __ne__ implementations.  Since fixing default __ne__
   implementation in issue #21408 they are redundant.
 
diff --git a/Modules/_io/bytesio.c b/Modules/_io/bytesio.c
--- a/Modules/_io/bytesio.c
+++ b/Modules/_io/bytesio.c
@@ -779,6 +779,7 @@
 static PyObject *
 bytesio_close(bytesio *self)
 {
+    CHECK_EXPORTS(self);
     reset(self);
     Py_RETURN_NONE;
 }

-- 
Repository URL: https://hg.python.org/cpython

From solipsis at pitrou.net  Tue Feb  3 09:02:33 2015
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Tue, 03 Feb 2015 09:02:33 +0100
Subject: [Python-checkins] Daily reference leaks (4cb316fe6bf2): sum=3
Message-ID: <E1YIYRN-0001sp-QB@ssl.pitrou.net>

results for 4cb316fe6bf2 on branch "default"
--------------------------------------------

test_functools leaked [0, 0, 3] memory blocks, sum=3


Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogbMOxGv', '-x']

From python-checkins at python.org  Tue Feb  3 10:05:15 2015
From: python-checkins at python.org (serhiy.storchaka)
Date: Tue, 03 Feb 2015 09:05:15 +0000
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2322818=3A_Splittin?=
 =?utf-8?q?g_on_a_pattern_that_could_match_an_empty_string_now?=
Message-ID: <20150203090510.106309.14683@psf.io>

https://hg.python.org/cpython/rev/7c667d8ae10d
changeset:   94482:7c667d8ae10d
user:        Serhiy Storchaka <storchaka at gmail.com>
date:        Tue Feb 03 11:04:19 2015 +0200
summary:
  Issue #22818: Splitting on a pattern that could match an empty string now
raises a warning.  Patterns that can only match empty strings are now
rejected.

files:
  Doc/library/re.rst   |  32 +++++++++++++++++++++----
  Doc/whatsnew/3.5.rst |   7 +++++
  Lib/sre_compile.py   |  10 ++++----
  Lib/test/test_re.py  |  39 ++++++++++++++++++++++++-------
  Misc/NEWS            |   4 +++
  Modules/_sre.c       |  13 ++++++++++
  6 files changed, 85 insertions(+), 20 deletions(-)


diff --git a/Doc/library/re.rst b/Doc/library/re.rst
--- a/Doc/library/re.rst
+++ b/Doc/library/re.rst
@@ -626,17 +626,37 @@
    That way, separator components are always found at the same relative
    indices within the result list.
 
-   Note that *split* will never split a string on an empty pattern match.
-   For example:
+   .. note::
 
-      >>> re.split('x*', 'foo')
-      ['foo']
-      >>> re.split("(?m)^$", "foo\n\nbar\n")
-      ['foo\n\nbar\n']
+      :func:`split` doesn't currently split a string on an empty pattern match.
+      For example:
+
+         >>> re.split('x*', 'axbc')
+         ['a', 'bc']
+
+      Even though ``'x*'`` also matches 0 'x' before 'a', between 'b' and 'c',
+      and after 'c', currently these matches are ignored.  The correct behavior
+      (i.e. splitting on empty matches too and returning ``['', 'a', 'b', 'c',
+      '']``) will be implemented in future versions of Python, but since this
+      is a backward incompatible change, a :exc:`FutureWarning` will be raised
+      in the meanwhile.
+
+      Patterns that can only match empty strings currently never split the
+      string.  Since this doesn't match the expected behavior, a
+      :exc:`ValueError` will be raised starting from Python 3.5::
+
+         >>> re.split("^$", "foo\n\nbar\n", flags=re.M)
+         Traceback (most recent call last):
+           File "<stdin>", line 1, in <module>
+           ...
+         ValueError: split() requires a non-empty pattern match.
 
    .. versionchanged:: 3.1
       Added the optional flags argument.
 
+   .. versionchanged:: 3.5
+      Splitting on a pattern that could match an empty string now raises
+      a warning.  Patterns that can only match empty strings are now rejected.
 
 .. function:: findall(pattern, string, flags=0)
 
diff --git a/Doc/whatsnew/3.5.rst b/Doc/whatsnew/3.5.rst
--- a/Doc/whatsnew/3.5.rst
+++ b/Doc/whatsnew/3.5.rst
@@ -482,6 +482,13 @@
   simply define :meth:`~importlib.machinery.Loader.create_module` to return
   ``None`` (:issue:`23014`).
 
+* :func:`re.split` always ignored empty pattern matches, so the ``'x*'``
+  pattern worked the same as ``'x+'``, and the ``'\b'`` pattern never worked.
+  Now :func:`re.split` raises a warning if the pattern could match
+  an empty string.  For compatibility use patterns that never match an empty
+  string (e.g. ``'x+'`` instead of ``'x*'``).  Patterns that could only match
+  an empty string (such as ``'\b'``) now raise an error.
+
 Changes in the C API
 --------------------
 
diff --git a/Lib/sre_compile.py b/Lib/sre_compile.py
--- a/Lib/sre_compile.py
+++ b/Lib/sre_compile.py
@@ -414,8 +414,11 @@
     # this contains min/max pattern width, and an optional literal
     # prefix or a character map
     lo, hi = pattern.getwidth()
+    if hi > MAXCODE:
+        hi = MAXCODE
     if lo == 0:
-        return # not worth it
+        code.extend([INFO, 4, 0, lo, hi])
+        return
     # look for a literal prefix
     prefix = []
     prefixappend = prefix.append
@@ -495,10 +498,7 @@
     else:
         emit(MAXCODE)
         prefix = prefix[:MAXCODE]
-    if hi < MAXCODE:
-        emit(hi)
-    else:
-        emit(0)
+    emit(min(hi, MAXCODE))
     # add literal prefix
     if prefix:
         emit(len(prefix)) # length
diff --git a/Lib/test/test_re.py b/Lib/test/test_re.py
--- a/Lib/test/test_re.py
+++ b/Lib/test/test_re.py
@@ -251,28 +251,28 @@
         for string in ":a:b::c", S(":a:b::c"):
             self.assertTypedEqual(re.split(":", string),
                                   ['', 'a', 'b', '', 'c'])
-            self.assertTypedEqual(re.split(":*", string),
+            self.assertTypedEqual(re.split(":+", string),
                                   ['', 'a', 'b', 'c'])
-            self.assertTypedEqual(re.split("(:*)", string),
+            self.assertTypedEqual(re.split("(:+)", string),
                                   ['', ':', 'a', ':', 'b', '::', 'c'])
         for string in (b":a:b::c", B(b":a:b::c"), bytearray(b":a:b::c"),
                        memoryview(b":a:b::c")):
             self.assertTypedEqual(re.split(b":", string),
                                   [b'', b'a', b'b', b'', b'c'])
-            self.assertTypedEqual(re.split(b":*", string),
+            self.assertTypedEqual(re.split(b":+", string),
                                   [b'', b'a', b'b', b'c'])
-            self.assertTypedEqual(re.split(b"(:*)", string),
+            self.assertTypedEqual(re.split(b"(:+)", string),
                                   [b'', b':', b'a', b':', b'b', b'::', b'c'])
         for a, b, c in ("\xe0\xdf\xe7", "\u0430\u0431\u0432",
                         "\U0001d49c\U0001d49e\U0001d4b5"):
             string = ":%s:%s::%s" % (a, b, c)
             self.assertEqual(re.split(":", string), ['', a, b, '', c])
-            self.assertEqual(re.split(":*", string), ['', a, b, c])
-            self.assertEqual(re.split("(:*)", string),
+            self.assertEqual(re.split(":+", string), ['', a, b, c])
+            self.assertEqual(re.split("(:+)", string),
                              ['', ':', a, ':', b, '::', c])
 
-        self.assertEqual(re.split("(?::*)", ":a:b::c"), ['', 'a', 'b', 'c'])
-        self.assertEqual(re.split("(:)*", ":a:b::c"),
+        self.assertEqual(re.split("(?::+)", ":a:b::c"), ['', 'a', 'b', 'c'])
+        self.assertEqual(re.split("(:)+", ":a:b::c"),
                          ['', ':', 'a', ':', 'b', ':', 'c'])
         self.assertEqual(re.split("([b:]+)", ":a:b::c"),
                          ['', ':', 'a', ':b::', 'c'])
@@ -282,13 +282,34 @@
         self.assertEqual(re.split("(?:b)|(?::+)", ":a:b::c"),
                          ['', 'a', '', '', 'c'])
 
+        for sep, expected in [
+            (':*', ['', 'a', 'b', 'c']),
+            ('(?::*)', ['', 'a', 'b', 'c']),
+            ('(:*)', ['', ':', 'a', ':', 'b', '::', 'c']),
+            ('(:)*', ['', ':', 'a', ':', 'b', ':', 'c']),
+        ]:
+            with self.subTest(sep=sep), self.assertWarns(FutureWarning):
+                self.assertTypedEqual(re.split(sep, ':a:b::c'), expected)
+
+        for sep, expected in [
+            ('', [':a:b::c']),
+            (r'\b', [':a:b::c']),
+            (r'(?=:)', [':a:b::c']),
+            (r'(?<=:)', [':a:b::c']),
+        ]:
+            with self.subTest(sep=sep), self.assertRaises(ValueError):
+                self.assertTypedEqual(re.split(sep, ':a:b::c'), expected)
+
     def test_qualified_re_split(self):
         self.assertEqual(re.split(":", ":a:b::c", maxsplit=2), ['', 'a', 'b::c'])
         self.assertEqual(re.split(':', 'a:b:c:d', maxsplit=2), ['a', 'b', 'c:d'])
         self.assertEqual(re.split("(:)", ":a:b::c", maxsplit=2),
                          ['', ':', 'a', ':', 'b::c'])
-        self.assertEqual(re.split("(:*)", ":a:b::c", maxsplit=2),
+        self.assertEqual(re.split("(:+)", ":a:b::c", maxsplit=2),
                          ['', ':', 'a', ':', 'b::c'])
+        with self.assertWarns(FutureWarning):
+            self.assertEqual(re.split("(:*)", ":a:b::c", maxsplit=2),
+                             ['', ':', 'a', ':', 'b::c'])
 
     def test_re_findall(self):
         self.assertEqual(re.findall(":+", "abc"), [])
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -232,6 +232,10 @@
 Library
 -------
 
+- Issue #22818: Splitting on a pattern that could match an empty string now
+  raises a warning.  Patterns that can only match empty strings are now
+  rejected.
+
 - Issue #23099: Closing io.BytesIO with exported buffer is rejected now to
   prevent corrupting exported buffer.
 
diff --git a/Modules/_sre.c b/Modules/_sre.c
--- a/Modules/_sre.c
+++ b/Modules/_sre.c
@@ -863,6 +863,19 @@
     if (!string)
         return NULL;
 
+    assert(self->codesize != 0);
+    if (self->code[0] != SRE_OP_INFO || self->code[3] == 0) {
+        if (self->code[0] == SRE_OP_INFO && self->code[4] == 0) {
+            PyErr_SetString(PyExc_ValueError,
+                            "split() requires a non-empty pattern match.");
+            return NULL;
+        }
+        if (PyErr_WarnEx(PyExc_FutureWarning,
+                         "split() requires a non-empty pattern match.",
+                         1) < 0)
+            return NULL;
+    }
+
     string = state_init(&state, self, string, 0, PY_SSIZE_T_MAX);
     if (!string)
         return NULL;

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Tue Feb  3 10:31:13 2015
From: python-checkins at python.org (serhiy.storchaka)
Date: Tue, 03 Feb 2015 09:31:13 +0000
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2315381=3A_Optimize?=
 =?utf-8?q?d_io=2EBytesIO_to_make_less_allocations_and_copyings=2E?=
Message-ID: <20150203093057.34390.35293@psf.io>

https://hg.python.org/cpython/rev/2e29d54843a4
changeset:   94483:2e29d54843a4
user:        Serhiy Storchaka <storchaka at gmail.com>
date:        Tue Feb 03 11:30:10 2015 +0200
summary:
  Issue #15381: Optimized io.BytesIO to make less allocations and copyings.

files:
  Doc/whatsnew/3.5.rst      |    3 +
  Lib/test/test_memoryio.py |    5 +-
  Misc/NEWS                 |    2 +
  Modules/_io/bytesio.c     |  338 +++++++++++--------------
  4 files changed, 159 insertions(+), 189 deletions(-)


diff --git a/Doc/whatsnew/3.5.rst b/Doc/whatsnew/3.5.rst
--- a/Doc/whatsnew/3.5.rst
+++ b/Doc/whatsnew/3.5.rst
@@ -362,6 +362,9 @@
   The speed up can range from 3x to 15x.
   (:issue:`21486`, :issue:`21487`, :issue:`20826`)
 
+* Many operations on :class:`io.BytesIO` are now 50% to 100% faster.
+  (Contributed by Serhiy Storchaka in :issue:`15381`.)
+
 
 Build and C API Changes
 =======================
diff --git a/Lib/test/test_memoryio.py b/Lib/test/test_memoryio.py
--- a/Lib/test/test_memoryio.py
+++ b/Lib/test/test_memoryio.py
@@ -718,12 +718,11 @@
 
     @support.cpython_only
     def test_sizeof(self):
-        basesize = support.calcobjsize('P2nN2PnP')
+        basesize = support.calcobjsize('P2n2Pn')
         check = self.check_sizeof
         self.assertEqual(object.__sizeof__(io.BytesIO()), basesize)
         check(io.BytesIO(), basesize )
-        check(io.BytesIO(b'a'), basesize + 1 )
-        check(io.BytesIO(b'a' * 1000), basesize + 1000)
+        check(io.BytesIO(b'a' * 1000), basesize + sys.getsizeof(b'a' * 1000))
 
     # Various tests of copy-on-write behaviour for BytesIO.
 
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -232,6 +232,8 @@
 Library
 -------
 
+- Issue #15381: Optimized io.BytesIO to make less allocations and copyings.
+
 - Issue #22818: Splitting on a pattern that could match an empty string now
   raises a warning.  Patterns that can only match empty strings are now
   rejected.
diff --git a/Modules/_io/bytesio.c b/Modules/_io/bytesio.c
--- a/Modules/_io/bytesio.c
+++ b/Modules/_io/bytesio.c
@@ -4,17 +4,12 @@
 
 typedef struct {
     PyObject_HEAD
-    char *buf;
+    PyObject *buf;
     Py_ssize_t pos;
     Py_ssize_t string_size;
-    size_t buf_size;
     PyObject *dict;
     PyObject *weakreflist;
     Py_ssize_t exports;
-    /** If `initvalue' != NULL, `buf' is a read-only pointer into the PyBytes
-     * referenced by `initvalue'. It must be copied prior to mutation, and
-     * released during finalization */
-    PyObject *initvalue;
 } bytesio;
 
 typedef struct {
@@ -22,12 +17,18 @@
     bytesio *source;
 } bytesiobuf;
 
+/* The bytesio object can be in three states:
+  * Py_REFCNT(buf) == 1, exports == 0.
+  * Py_REFCNT(buf) > 1.  exports == 0, string_size == PyBytes_GET_SIZE(buf),
+    first modification or export causes the internal buffer copying.
+  * exports > 0.  Py_REFCNT(buf) == 1, any modifications are forbidden.
+*/
 
-#define CHECK_CLOSED(self, ret)                             \
+#define CHECK_CLOSED(self)                                  \
     if ((self)->buf == NULL) {                              \
         PyErr_SetString(PyExc_ValueError,                   \
                         "I/O operation on closed file.");   \
-        return ret;                                         \
+        return NULL;                                        \
     }
 
 #define CHECK_EXPORTS(self) \
@@ -37,47 +38,8 @@
         return NULL; \
     }
 
-/* Ensure we have a buffer suitable for writing, in the case that an initvalue
- * object was provided, and we're currently borrowing its buffer. `size'
- * indicates the new buffer size allocated as part of unsharing, to avoid a
- * redundant reallocation caused by any subsequent mutation. `truncate'
- * indicates whether truncation should occur if `size` < self->string_size.
- *
- * Do nothing if the buffer wasn't shared. Returns 0 on success, or sets an
- * exception and returns -1 on failure. Existing state is preserved on failure.
- */
-static int
-unshare(bytesio *self, size_t preferred_size, int truncate)
-{
-    if (self->initvalue) {
-        Py_ssize_t copy_size;
-        char *new_buf;
+#define SHARED_BUF(self) (Py_REFCNT((self)->buf) > 1)
 
-        if((! truncate) && preferred_size < (size_t)self->string_size) {
-            preferred_size = self->string_size;
-        }
-
-        /* PyMem_Malloc() returns NULL if preferred_size is bigger
-           than PY_SSIZE_T_MAX */
-        new_buf = (char *)PyMem_Malloc(preferred_size);
-        if (new_buf == NULL) {
-            PyErr_NoMemory();
-            return -1;
-        }
-
-        copy_size = self->string_size;
-        if ((size_t)copy_size > preferred_size) {
-            copy_size = preferred_size;
-        }
-
-        memcpy(new_buf, self->buf, copy_size);
-        Py_CLEAR(self->initvalue);
-        self->buf = new_buf;
-        self->buf_size = preferred_size;
-        self->string_size = (Py_ssize_t) copy_size;
-    }
-    return 0;
-}
 
 /* Internal routine to get a line from the buffer of a BytesIO
    object. Returns the length between the current position to the
@@ -91,7 +53,7 @@
     assert(self->buf != NULL);
 
     /* Move to the end of the line, up to the end of the string, s. */
-    start = self->buf + self->pos;
+    start = PyBytes_AS_STRING(self->buf) + self->pos;
     maxlen = self->string_size - self->pos;
     if (len < 0 || len > maxlen)
         len = maxlen;
@@ -109,6 +71,27 @@
     return len;
 }
 
+/* Internal routine for detaching the shared buffer of BytesIO objects.
+   The caller should ensure that the 'size' argument is non-negative and
+   not lesser than self->string_size.  Returns 0 on success, -1 otherwise. */
+static int
+unshare_buffer(bytesio *self, size_t size)
+{
+    PyObject *new_buf, *old_buf;
+    assert(SHARED_BUF(self));
+    assert(self->exports == 0);
+    assert(size >= (size_t)self->string_size);
+    new_buf = PyBytes_FromStringAndSize(NULL, size);
+    if (new_buf == NULL)
+        return -1;
+    memcpy(PyBytes_AS_STRING(new_buf), PyBytes_AS_STRING(self->buf),
+           self->string_size);
+    old_buf = self->buf;
+    self->buf = new_buf;
+    Py_DECREF(old_buf);
+    return 0;
+}
+
 /* Internal routine for changing the size of the buffer of BytesIO objects.
    The caller should ensure that the 'size' argument is non-negative.  Returns
    0 on success, -1 otherwise. */
@@ -117,8 +100,7 @@
 {
     /* Here, unsigned types are used to avoid dealing with signed integer
        overflow, which is undefined in C. */
-    size_t alloc = self->buf_size;
-    char *new_buf = NULL;
+    size_t alloc = PyBytes_GET_SIZE(self->buf);
 
     assert(self->buf != NULL);
 
@@ -146,13 +128,15 @@
 
     if (alloc > ((size_t)-1) / sizeof(char))
         goto overflow;
-    new_buf = (char *)PyMem_Realloc(self->buf, alloc * sizeof(char));
-    if (new_buf == NULL) {
-        PyErr_NoMemory();
-        return -1;
+
+    if (SHARED_BUF(self)) {
+        if (unshare_buffer(self, alloc) < 0)
+            return -1;
     }
-    self->buf_size = alloc;
-    self->buf = new_buf;
+    else {
+        if (_PyBytes_Resize(&self->buf, alloc) < 0)
+            return -1;
+    }
 
     return 0;
 
@@ -167,19 +151,16 @@
 static Py_ssize_t
 write_bytes(bytesio *self, const char *bytes, Py_ssize_t len)
 {
-    size_t desired;
-
     assert(self->buf != NULL);
     assert(self->pos >= 0);
     assert(len >= 0);
 
-    desired = (size_t)self->pos + len;
-    if (unshare(self, desired, 0) < 0) {
-        return -1;
+    if ((size_t)self->pos + len > (size_t)PyBytes_GET_SIZE(self->buf)) {
+        if (resize_buffer(self, (size_t)self->pos + len) < 0)
+            return -1;
     }
-
-    if (desired > self->buf_size) {
-        if (resize_buffer(self, (size_t)self->pos + len) < 0)
+    else if (SHARED_BUF(self)) {
+        if (unshare_buffer(self, self->string_size) < 0)
             return -1;
     }
 
@@ -192,13 +173,13 @@
           |   |            <--to pad-->|<---to write--->    |
           0   buf                   position
         */
-        memset(self->buf + self->string_size, '\0',
+        memset(PyBytes_AS_STRING(self->buf) + self->string_size, '\0',
                (self->pos - self->string_size) * sizeof(char));
     }
 
     /* Copy the data to the internal buffer, overwriting some of the existing
        data if self->pos < self->string_size. */
-    memcpy(self->buf + self->pos, bytes, len);
+    memcpy(PyBytes_AS_STRING(self->buf) + self->pos, bytes, len);
     self->pos += len;
 
     /* Set the new length of the internal string if it has changed. */
@@ -209,74 +190,6 @@
     return len;
 }
 
-/* Release or free any existing buffer, and place the BytesIO in the closed
- * state. */
-static void
-reset(bytesio *self)
-{
-    if (self->initvalue) {
-        Py_CLEAR(self->initvalue);
-    } else if (self->buf) {
-        PyMem_Free(self->buf);
-    }
-    self->buf = NULL;
-    self->string_size = 0;
-    self->pos = 0;
-}
-
-/* Reinitialize with a new heap-allocated buffer of size `size`. Returns 0 on
- * success, or sets an exception and returns -1 on failure. Existing state is
- * preserved on failure. */
-static int
-reinit_private(bytesio *self, Py_ssize_t size)
-{
-    char *tmp = (char *)PyMem_Malloc(size);
-    if (tmp == NULL) {
-        PyErr_NoMemory();
-        return -1;
-    }
-    reset(self);
-    self->buf = tmp;
-    self->buf_size = size;
-    return 0;
-}
-
-/* Internal version of BytesIO.__init__; resets the object to its initial
- * (closed) state before repopulating it, optionally by sharing a PyBytes
- * buffer provided by `initvalue'. Returns 0 on success, or sets an exception
- * and returns -1 on failure. */
-static int
-reinit(bytesio *self, PyObject *initvalue)
-{
-    CHECK_CLOSED(self, -1);
-
-    if (initvalue == NULL || initvalue == Py_None) {
-        if (reinit_private(self, 0) < 0) {
-            return -1;
-        }
-    } else if (PyBytes_CheckExact(initvalue)) {
-        reset(self);
-        Py_INCREF(initvalue);
-        self->initvalue = initvalue;
-        self->buf = PyBytes_AS_STRING(initvalue);
-        self->buf_size = PyBytes_GET_SIZE(initvalue);
-        self->string_size = PyBytes_GET_SIZE(initvalue);
-    } else {
-        Py_buffer buf;
-        if (PyObject_GetBuffer(initvalue, &buf, PyBUF_CONTIG_RO) < 0) {
-            return -1;
-        }
-        if (reinit_private(self, buf.len) < 0) {
-            PyBuffer_Release(&buf);
-            return -1;
-        }
-        memcpy(self->buf, buf.buf, buf.len);
-        self->string_size = buf.len;
-        PyBuffer_Release(&buf);
-    }
-    return 0;
-}
-
 static PyObject *
 bytesio_get_closed(bytesio *self)
 {
@@ -301,7 +214,7 @@
 static PyObject *
 return_not_closed(bytesio *self)
 {
-    CHECK_CLOSED(self, NULL);
+    CHECK_CLOSED(self);
     Py_RETURN_TRUE;
 }
 
@@ -311,7 +224,7 @@
 static PyObject *
 bytesio_flush(bytesio *self)
 {
-    CHECK_CLOSED(self, NULL);
+    CHECK_CLOSED(self);
     Py_RETURN_NONE;
 }
 
@@ -327,7 +240,7 @@
     bytesiobuf *buf;
     PyObject *view;
 
-    CHECK_CLOSED(self, NULL);
+    CHECK_CLOSED(self);
 
     buf = (bytesiobuf *) type->tp_alloc(type, 0);
     if (buf == NULL)
@@ -347,8 +260,23 @@
 static PyObject *
 bytesio_getvalue(bytesio *self)
 {
-    CHECK_CLOSED(self, NULL);
-    return PyBytes_FromStringAndSize(self->buf, self->string_size);
+    CHECK_CLOSED(self);
+    if (self->string_size <= 1 || self->exports > 0)
+        return PyBytes_FromStringAndSize(PyBytes_AS_STRING(self->buf),
+                                         self->string_size);
+
+    if (self->string_size != PyBytes_GET_SIZE(self->buf)) {
+        if (SHARED_BUF(self)) {
+            if (unshare_buffer(self, self->string_size) < 0)
+                return NULL;
+        }
+        else {
+            if (_PyBytes_Resize(&self->buf, self->string_size) < 0)
+                return NULL;
+        }
+    }
+    Py_INCREF(self->buf);
+    return self->buf;
 }
 
 PyDoc_STRVAR(isatty_doc,
@@ -360,7 +288,7 @@
 static PyObject *
 bytesio_isatty(bytesio *self)
 {
-    CHECK_CLOSED(self, NULL);
+    CHECK_CLOSED(self);
     Py_RETURN_FALSE;
 }
 
@@ -370,10 +298,29 @@
 static PyObject *
 bytesio_tell(bytesio *self)
 {
-    CHECK_CLOSED(self, NULL);
+    CHECK_CLOSED(self);
     return PyLong_FromSsize_t(self->pos);
 }
 
+static PyObject *
+read_bytes(bytesio *self, Py_ssize_t size)
+{
+    char *output;
+
+    assert(self->buf != NULL);
+    if (size > 1 &&
+        self->pos == 0 && size == PyBytes_GET_SIZE(self->buf) &&
+        self->exports == 0) {
+        self->pos += size;
+        Py_INCREF(self->buf);
+        return self->buf;
+    }
+
+    output = PyBytes_AS_STRING(self->buf) + self->pos;
+    self->pos += size;
+    return PyBytes_FromStringAndSize(output, size);
+}
+
 PyDoc_STRVAR(read_doc,
 "read([size]) -> read at most size bytes, returned as a string.\n"
 "\n"
@@ -384,10 +331,9 @@
 bytesio_read(bytesio *self, PyObject *args)
 {
     Py_ssize_t size, n;
-    char *output;
     PyObject *arg = Py_None;
 
-    CHECK_CLOSED(self, NULL);
+    CHECK_CLOSED(self);
 
     if (!PyArg_ParseTuple(args, "|O:read", &arg))
         return NULL;
@@ -415,11 +361,7 @@
             size = 0;
     }
 
-    assert(self->buf != NULL);
-    output = self->buf + self->pos;
-    self->pos += size;
-
-    return PyBytes_FromStringAndSize(output, size);
+    return read_bytes(self, size);
 }
 
 
@@ -453,10 +395,9 @@
 bytesio_readline(bytesio *self, PyObject *args)
 {
     Py_ssize_t size, n;
-    char *output;
     PyObject *arg = Py_None;
 
-    CHECK_CLOSED(self, NULL);
+    CHECK_CLOSED(self);
 
     if (!PyArg_ParseTuple(args, "|O:readline", &arg))
         return NULL;
@@ -478,9 +419,7 @@
 
     n = scan_eol(self, size);
 
-    output = self->buf + self->pos;
-    self->pos += n;
-    return PyBytes_FromStringAndSize(output, n);
+    return read_bytes(self, n);
 }
 
 PyDoc_STRVAR(readlines_doc,
@@ -498,7 +437,7 @@
     char *output;
     PyObject *arg = Py_None;
 
-    CHECK_CLOSED(self, NULL);
+    CHECK_CLOSED(self);
 
     if (!PyArg_ParseTuple(args, "|O:readlines", &arg))
         return NULL;
@@ -523,7 +462,7 @@
     if (!result)
         return NULL;
 
-    output = self->buf + self->pos;
+    output = PyBytes_AS_STRING(self->buf) + self->pos;
     while ((n = scan_eol(self, -1)) != 0) {
         self->pos += n;
         line = PyBytes_FromStringAndSize(output, n);
@@ -558,7 +497,7 @@
     Py_buffer buffer;
     Py_ssize_t len, n;
 
-    CHECK_CLOSED(self, NULL);
+    CHECK_CLOSED(self);
 
     if (!PyArg_Parse(arg, "w*", &buffer))
         return NULL;
@@ -572,7 +511,7 @@
             len = 0;
     }
 
-    memcpy(buffer.buf, self->buf + self->pos, len);
+    memcpy(buffer.buf, PyBytes_AS_STRING(self->buf) + self->pos, len);
     assert(self->pos + len < PY_SSIZE_T_MAX);
     assert(len >= 0);
     self->pos += len;
@@ -593,7 +532,7 @@
     Py_ssize_t size;
     PyObject *arg = Py_None;
 
-    CHECK_CLOSED(self, NULL);
+    CHECK_CLOSED(self);
     CHECK_EXPORTS(self);
 
     if (!PyArg_ParseTuple(args, "|O:truncate", &arg))
@@ -620,10 +559,6 @@
         return NULL;
     }
 
-    if (unshare(self, size, 1) < 0) {
-        return NULL;
-    }
-
     if (size < self->string_size) {
         self->string_size = size;
         if (resize_buffer(self, size) < 0)
@@ -636,19 +571,16 @@
 static PyObject *
 bytesio_iternext(bytesio *self)
 {
-    const char *next;
     Py_ssize_t n;
 
-    CHECK_CLOSED(self, NULL);
+    CHECK_CLOSED(self);
 
     n = scan_eol(self, -1);
 
     if (n == 0)
         return NULL;
 
-    next = self->buf + self->pos;
-    self->pos += n;
-    return PyBytes_FromStringAndSize(next, n);
+    return read_bytes(self, n);
 }
 
 PyDoc_STRVAR(seek_doc,
@@ -666,7 +598,7 @@
     Py_ssize_t pos;
     int mode = 0;
 
-    CHECK_CLOSED(self, NULL);
+    CHECK_CLOSED(self);
 
     if (!PyArg_ParseTuple(args, "n|i:seek", &pos, &mode))
         return NULL;
@@ -721,7 +653,7 @@
     Py_buffer buf;
     PyObject *result = NULL;
 
-    CHECK_CLOSED(self, NULL);
+    CHECK_CLOSED(self);
     CHECK_EXPORTS(self);
 
     if (PyObject_GetBuffer(obj, &buf, PyBUF_CONTIG_RO) < 0)
@@ -749,7 +681,7 @@
     PyObject *it, *item;
     PyObject *ret;
 
-    CHECK_CLOSED(self, NULL);
+    CHECK_CLOSED(self);
 
     it = PyObject_GetIter(v);
     if (it == NULL)
@@ -780,7 +712,7 @@
 bytesio_close(bytesio *self)
 {
     CHECK_EXPORTS(self);
-    reset(self);
+    Py_CLEAR(self->buf);
     Py_RETURN_NONE;
 }
 
@@ -828,11 +760,11 @@
 static PyObject *
 bytesio_setstate(bytesio *self, PyObject *state)
 {
+    PyObject *result;
     PyObject *position_obj;
     PyObject *dict;
     Py_ssize_t pos;
 
-    CHECK_EXPORTS(self);
     assert(state != NULL);
 
     /* We allow the state tuple to be longer than 3, because we may need
@@ -844,13 +776,18 @@
                      Py_TYPE(self)->tp_name, Py_TYPE(state)->tp_name);
         return NULL;
     }
+    CHECK_EXPORTS(self);
+    /* Reset the object to its default state. This is only needed to handle
+       the case of repeated calls to __setstate__. */
+    self->string_size = 0;
+    self->pos = 0;
 
-    /* Reset the object to its default state and set the value of the internal
-     * buffer. If state[0] does not support the buffer protocol, reinit() will
-     * raise the appropriate TypeError. */
-    if (reinit(self, PyTuple_GET_ITEM(state, 0)) < 0) {
+    /* Set the value of the internal buffer. If state[0] does not support the
+       buffer protocol, bytesio_write will raise the appropriate TypeError. */
+    result = bytesio_write(self, PyTuple_GET_ITEM(state, 0));
+    if (result == NULL)
         return NULL;
-    }
+    Py_DECREF(result);
 
     /* Set carefully the position value. Alternatively, we could use the seek
        method instead of modifying self->pos directly to better protect the
@@ -905,9 +842,7 @@
                         "deallocated BytesIO object has exported buffers");
         PyErr_Print();
     }
-
-    reset(self);
-
+    Py_CLEAR(self->buf);
     Py_CLEAR(self->dict);
     if (self->weakreflist != NULL)
         PyObject_ClearWeakRefs((PyObject *) self);
@@ -927,7 +862,7 @@
     /* tp_alloc initializes all the fields to zero. So we don't have to
        initialize them here. */
 
-    self->buf = (char *)PyMem_Malloc(0);
+    self->buf = PyBytes_FromStringAndSize(NULL, 0);
     if (self->buf == NULL) {
         Py_DECREF(self);
         return PyErr_NoMemory();
@@ -946,7 +881,33 @@
                                      &initvalue))
         return -1;
 
-    return reinit(self, initvalue);
+    /* In case, __init__ is called multiple times. */
+    self->string_size = 0;
+    self->pos = 0;
+
+    if (self->exports > 0) {
+        PyErr_SetString(PyExc_BufferError,
+                        "Existing exports of data: object cannot be re-sized");
+        return -1;
+    }
+    if (initvalue && initvalue != Py_None) {
+        if (PyBytes_CheckExact(initvalue)) {
+            Py_INCREF(initvalue);
+            Py_XDECREF(self->buf);
+            self->buf = initvalue;
+            self->string_size = PyBytes_GET_SIZE(initvalue);
+        }
+        else {
+            PyObject *res;
+            res = bytesio_write(self, initvalue);
+            if (res == NULL)
+                return -1;
+            Py_DECREF(res);
+            self->pos = 0;
+        }
+    }
+
+    return 0;
 }
 
 static PyObject *
@@ -955,8 +916,8 @@
     Py_ssize_t res;
 
     res = sizeof(bytesio);
-    if (self->buf)
-        res += self->buf_size;
+    if (self->buf && !SHARED_BUF(self))
+        res += _PySys_GetSizeOf(self->buf);
     return PyLong_FromSsize_t(res);
 }
 
@@ -1066,11 +1027,16 @@
 {
     int ret;
     bytesio *b = (bytesio *) obj->source;
+    if (SHARED_BUF(b)) {
+        if (unshare_buffer(b, b->string_size) < 0)
+            return -1;
+    }
     if (view == NULL) {
         b->exports++;
         return 0;
     }
-    ret = PyBuffer_FillInfo(view, (PyObject*)obj, b->buf, b->string_size,
+    ret = PyBuffer_FillInfo(view, (PyObject*)obj,
+                            PyBytes_AS_STRING(b->buf), b->string_size,
                             0, flags);
     if (ret >= 0) {
         b->exports++;

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Tue Feb  3 10:55:27 2015
From: python-checkins at python.org (berker.peksag)
Date: Tue, 03 Feb 2015 09:55:27 +0000
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIzMzU4?=
 =?utf-8?q?=3A_Add_missing_BaseServer_entry_to_socketserver=2E=5F=5Fall=5F?=
 =?utf-8?b?Xy4=?=
Message-ID: <20150203095525.39294.46495@psf.io>

https://hg.python.org/cpython/rev/e614ff664877
changeset:   94484:e614ff664877
branch:      3.4
parent:      94480:e62d54128bd3
user:        Berker Peksag <berker.peksag at gmail.com>
date:        Tue Feb 03 11:55:09 2015 +0200
summary:
  Issue #23358: Add missing BaseServer entry to socketserver.__all__.

Patch by Martin Panter.

files:
  Lib/socketserver.py           |   8 ++++----
  Lib/test/test_socketserver.py |  19 ++++++++++++-------
  2 files changed, 16 insertions(+), 11 deletions(-)


diff --git a/Lib/socketserver.py b/Lib/socketserver.py
--- a/Lib/socketserver.py
+++ b/Lib/socketserver.py
@@ -138,10 +138,10 @@
 except ImportError:
     import dummy_threading as threading
 
-__all__ = ["TCPServer","UDPServer","ForkingUDPServer","ForkingTCPServer",
-           "ThreadingUDPServer","ThreadingTCPServer","BaseRequestHandler",
-           "StreamRequestHandler","DatagramRequestHandler",
-           "ThreadingMixIn", "ForkingMixIn"]
+__all__ = ["BaseServer", "TCPServer", "UDPServer", "ForkingUDPServer",
+           "ForkingTCPServer", "ThreadingUDPServer", "ThreadingTCPServer",
+           "BaseRequestHandler", "StreamRequestHandler",
+           "DatagramRequestHandler", "ThreadingMixIn", "ForkingMixIn"]
 if hasattr(socket, "AF_UNIX"):
     __all__.extend(["UnixStreamServer","UnixDatagramServer",
                     "ThreadingUnixStreamServer",
diff --git a/Lib/test/test_socketserver.py b/Lib/test/test_socketserver.py
--- a/Lib/test/test_socketserver.py
+++ b/Lib/test/test_socketserver.py
@@ -2,7 +2,6 @@
 Test suite for socketserver.
 """
 
-import _imp as imp
 import contextlib
 import os
 import select
@@ -313,12 +312,18 @@
                                        socketserver.StreamRequestHandler)
 
 
-def test_main():
-    if imp.lock_held():
-        # If the import lock is held, the threads will hang
-        raise unittest.SkipTest("can't run when import lock is held")
+class MiscTestCase(unittest.TestCase):
 
-    test.support.run_unittest(SocketServerTest)
+    def test_all(self):
+        # objects defined in the module should be in __all__
+        expected = []
+        for name in dir(socketserver):
+            if not name.startswith('_'):
+                mod_object = getattr(socketserver, name)
+                if getattr(mod_object, '__module__', None) == 'socketserver':
+                    expected.append(name)
+        self.assertCountEqual(socketserver.__all__, expected)
+
 
 if __name__ == "__main__":
-    test_main()
+    unittest.main()

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Tue Feb  3 10:55:27 2015
From: python-checkins at python.org (berker.peksag)
Date: Tue, 03 Feb 2015 09:55:27 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?q?=29=3A_Issue_=2323358=3A_Add_missing_BaseServer_entry_to_socket?=
 =?utf-8?b?c2VydmVyLl9fYWxsX18u?=
Message-ID: <20150203095525.106346.89159@psf.io>

https://hg.python.org/cpython/rev/80d282505d21
changeset:   94485:80d282505d21
parent:      94483:2e29d54843a4
parent:      94484:e614ff664877
user:        Berker Peksag <berker.peksag at gmail.com>
date:        Tue Feb 03 11:55:32 2015 +0200
summary:
  Issue #23358: Add missing BaseServer entry to socketserver.__all__.

Patch by Martin Panter.

files:
  Lib/socketserver.py           |   8 ++++----
  Lib/test/test_socketserver.py |  19 ++++++++++++-------
  2 files changed, 16 insertions(+), 11 deletions(-)


diff --git a/Lib/socketserver.py b/Lib/socketserver.py
--- a/Lib/socketserver.py
+++ b/Lib/socketserver.py
@@ -138,10 +138,10 @@
     import dummy_threading as threading
 from time import monotonic as time
 
-__all__ = ["TCPServer","UDPServer","ForkingUDPServer","ForkingTCPServer",
-           "ThreadingUDPServer","ThreadingTCPServer","BaseRequestHandler",
-           "StreamRequestHandler","DatagramRequestHandler",
-           "ThreadingMixIn", "ForkingMixIn"]
+__all__ = ["BaseServer", "TCPServer", "UDPServer", "ForkingUDPServer",
+           "ForkingTCPServer", "ThreadingUDPServer", "ThreadingTCPServer",
+           "BaseRequestHandler", "StreamRequestHandler",
+           "DatagramRequestHandler", "ThreadingMixIn", "ForkingMixIn"]
 if hasattr(socket, "AF_UNIX"):
     __all__.extend(["UnixStreamServer","UnixDatagramServer",
                     "ThreadingUnixStreamServer",
diff --git a/Lib/test/test_socketserver.py b/Lib/test/test_socketserver.py
--- a/Lib/test/test_socketserver.py
+++ b/Lib/test/test_socketserver.py
@@ -2,7 +2,6 @@
 Test suite for socketserver.
 """
 
-import _imp as imp
 import contextlib
 import os
 import select
@@ -281,12 +280,18 @@
                                        socketserver.StreamRequestHandler)
 
 
-def test_main():
-    if imp.lock_held():
-        # If the import lock is held, the threads will hang
-        raise unittest.SkipTest("can't run when import lock is held")
+class MiscTestCase(unittest.TestCase):
 
-    test.support.run_unittest(SocketServerTest)
+    def test_all(self):
+        # objects defined in the module should be in __all__
+        expected = []
+        for name in dir(socketserver):
+            if not name.startswith('_'):
+                mod_object = getattr(socketserver, name)
+                if getattr(mod_object, '__module__', None) == 'socketserver':
+                    expected.append(name)
+        self.assertCountEqual(socketserver.__all__, expected)
+
 
 if __name__ == "__main__":
-    test_main()
+    unittest.main()

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Tue Feb  3 11:22:04 2015
From: python-checkins at python.org (berker.peksag)
Date: Tue, 03 Feb 2015 10:22:04 +0000
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2313128=3A_Print_re?=
 =?utf-8?q?sponse_headers_for_CONNECT_requests_when_debuglevel_=3E_0=2E?=
Message-ID: <20150203102203.106325.68325@psf.io>

https://hg.python.org/cpython/rev/dfbd07cdc031
changeset:   94486:dfbd07cdc031
user:        Berker Peksag <berker.peksag at gmail.com>
date:        Tue Feb 03 12:22:11 2015 +0200
summary:
  Issue #13128: Print response headers for CONNECT requests when debuglevel > 0.

Patch by Demian Brecht.

files:
  Lib/http/client.py       |   3 +++
  Lib/test/test_httplib.py |  23 ++++++++++++++++++-----
  Misc/NEWS                |   3 +++
  3 files changed, 24 insertions(+), 5 deletions(-)


diff --git a/Lib/http/client.py b/Lib/http/client.py
--- a/Lib/http/client.py
+++ b/Lib/http/client.py
@@ -776,6 +776,9 @@
             if line in (b'\r\n', b'\n', b''):
                 break
 
+            if self.debuglevel > 0:
+                print('header:', line.decode())
+
     def connect(self):
         """Connect to the host and port specified in __init__."""
         self.sock = self._create_connection(
diff --git a/Lib/test/test_httplib.py b/Lib/test/test_httplib.py
--- a/Lib/test/test_httplib.py
+++ b/Lib/test/test_httplib.py
@@ -1269,17 +1269,18 @@
             'HTTP/1.1 200 OK\r\n' # Reply to HEAD
             'Content-Length: 42\r\n\r\n'
         )
-
-        def create_connection(address, timeout=None, source_address=None):
-            return FakeSocket(response_text, host=address[0], port=address[1])
-
         self.host = 'proxy.com'
         self.conn = client.HTTPConnection(self.host)
-        self.conn._create_connection = create_connection
+        self.conn._create_connection = self._create_connection(response_text)
 
     def tearDown(self):
         self.conn.close()
 
+    def _create_connection(self, response_text):
+        def create_connection(address, timeout=None, source_address=None):
+            return FakeSocket(response_text, host=address[0], port=address[1])
+        return create_connection
+
     def test_set_tunnel_host_port_headers(self):
         tunnel_host = 'destination.com'
         tunnel_port = 8888
@@ -1320,6 +1321,18 @@
         self.assertIn(b'CONNECT destination.com', self.conn.sock.data)
         self.assertIn(b'Host: destination.com', self.conn.sock.data)
 
+    def test_tunnel_debuglog(self):
+        expected_header = 'X-Dummy: 1'
+        response_text = 'HTTP/1.0 200 OK\r\n{}\r\n\r\n'.format(expected_header)
+
+        self.conn.set_debuglevel(1)
+        self.conn._create_connection = self._create_connection(response_text)
+        self.conn.set_tunnel('destination.com')
+
+        with support.captured_stdout() as output:
+            self.conn.request('PUT', '/', '')
+        lines = output.getvalue().splitlines()
+        self.assertIn('header: {}'.format(expected_header), lines)
 
 
 @support.reap_threads
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -232,6 +232,9 @@
 Library
 -------
 
+- Issue #13128: Print response headers for CONNECT requests when debuglevel
+  > 0. Patch by Demian Brecht.
+
 - Issue #15381: Optimized io.BytesIO to make less allocations and copyings.
 
 - Issue #22818: Splitting on a pattern that could match an empty string now

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Tue Feb  3 13:58:47 2015
From: python-checkins at python.org (serhiy.storchaka)
Date: Tue, 03 Feb 2015 12:58:47 +0000
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2315381=3A_Try_to_f?=
 =?utf-8?q?ix_refcount_bug=2E_Empty_and_1-byte_buffers_are_always?=
Message-ID: <20150203125839.39298.2829@psf.io>

https://hg.python.org/cpython/rev/3fdbdf4d2ac7
changeset:   94487:3fdbdf4d2ac7
user:        Serhiy Storchaka <storchaka at gmail.com>
date:        Tue Feb 03 14:57:49 2015 +0200
summary:
  Issue #15381: Try to fix refcount bug. Empty and 1-byte buffers are always shared.

files:
  Modules/_io/bytesio.c |  4 +++-
  1 files changed, 3 insertions(+), 1 deletions(-)


diff --git a/Modules/_io/bytesio.c b/Modules/_io/bytesio.c
--- a/Modules/_io/bytesio.c
+++ b/Modules/_io/bytesio.c
@@ -38,7 +38,8 @@
         return NULL; \
     }
 
-#define SHARED_BUF(self) (Py_REFCNT((self)->buf) > 1)
+#define SHARED_BUF(self) (Py_REFCNT((self)->buf) > 1 || \
+                          PyBytes_GET_SIZE((self)->buf) <= 1)
 
 
 /* Internal routine to get a line from the buffer of a BytesIO
@@ -308,6 +309,7 @@
     char *output;
 
     assert(self->buf != NULL);
+    assert(size <= self->string_size);
     if (size > 1 &&
         self->pos == 0 && size == PyBytes_GET_SIZE(self->buf) &&
         self->exports == 0) {

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Tue Feb  3 15:13:55 2015
From: python-checkins at python.org (victor.stinner)
Date: Tue, 03 Feb 2015 14:13:55 +0000
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbywgVHVs?=
 =?utf-8?q?ip_issue_221=3A_Fix_doc_of_QueueEmpty_and_QueueFull?=
Message-ID: <20150203141351.34400.63641@psf.io>

https://hg.python.org/cpython/rev/fd445d2dd107
changeset:   94488:fd445d2dd107
branch:      3.4
parent:      94484:e614ff664877
user:        Victor Stinner <victor.stinner at gmail.com>
date:        Tue Feb 03 15:09:24 2015 +0100
summary:
  asyncio, Tulip issue 221: Fix doc of QueueEmpty and QueueFull

files:
  Doc/library/asyncio-sync.rst |  10 ++++------
  Lib/asyncio/queues.py        |   8 ++++++--
  2 files changed, 10 insertions(+), 8 deletions(-)


diff --git a/Doc/library/asyncio-sync.rst b/Doc/library/asyncio-sync.rst
--- a/Doc/library/asyncio-sync.rst
+++ b/Doc/library/asyncio-sync.rst
@@ -428,13 +428,11 @@
 
 .. exception:: QueueEmpty
 
-   Exception raised when non-blocking :meth:`~Queue.get` (or
-   :meth:`~Queue.get_nowait`) is called
-   on a :class:`Queue` object which is empty.
+   Exception raised when the :meth:`~Queue.get_nowait` method is called on a
+   :class:`Queue` object which is empty.
 
 
 .. exception:: QueueFull
 
-   Exception raised when non-blocking :meth:`~Queue.put` (or
-   :meth:`~Queue.put_nowait`) is called
-   on a :class:`Queue` object which is full.
+   Exception raised when the :meth:`~Queue.put_nowait` method is called on a
+   :class:`Queue` object which is full.
diff --git a/Lib/asyncio/queues.py b/Lib/asyncio/queues.py
--- a/Lib/asyncio/queues.py
+++ b/Lib/asyncio/queues.py
@@ -13,12 +13,16 @@
 
 
 class QueueEmpty(Exception):
-    'Exception raised by Queue.get(block=0)/get_nowait().'
+    """Exception raised when Queue.get_nowait() is called on a Queue object
+    which is empty.
+    """
     pass
 
 
 class QueueFull(Exception):
-    'Exception raised by Queue.put(block=0)/put_nowait().'
+    """Exception raised when the Queue.put_nowait() method is called on a Queue
+    object which is full.
+    """
     pass
 
 

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Tue Feb  3 15:13:55 2015
From: python-checkins at python.org (victor.stinner)
Date: Tue, 03 Feb 2015 14:13:55 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio_doc=29?=
Message-ID: <20150203141351.25855.56305@psf.io>

https://hg.python.org/cpython/rev/7494f3972726
changeset:   94489:7494f3972726
parent:      94487:3fdbdf4d2ac7
parent:      94488:fd445d2dd107
user:        Victor Stinner <victor.stinner at gmail.com>
date:        Tue Feb 03 15:12:13 2015 +0100
summary:
  Merge 3.4 (asyncio doc)

files:
  Doc/library/asyncio-sync.rst |  10 ++++------
  Lib/asyncio/queues.py        |   8 ++++++--
  2 files changed, 10 insertions(+), 8 deletions(-)


diff --git a/Doc/library/asyncio-sync.rst b/Doc/library/asyncio-sync.rst
--- a/Doc/library/asyncio-sync.rst
+++ b/Doc/library/asyncio-sync.rst
@@ -428,13 +428,11 @@
 
 .. exception:: QueueEmpty
 
-   Exception raised when non-blocking :meth:`~Queue.get` (or
-   :meth:`~Queue.get_nowait`) is called
-   on a :class:`Queue` object which is empty.
+   Exception raised when the :meth:`~Queue.get_nowait` method is called on a
+   :class:`Queue` object which is empty.
 
 
 .. exception:: QueueFull
 
-   Exception raised when non-blocking :meth:`~Queue.put` (or
-   :meth:`~Queue.put_nowait`) is called
-   on a :class:`Queue` object which is full.
+   Exception raised when the :meth:`~Queue.put_nowait` method is called on a
+   :class:`Queue` object which is full.
diff --git a/Lib/asyncio/queues.py b/Lib/asyncio/queues.py
--- a/Lib/asyncio/queues.py
+++ b/Lib/asyncio/queues.py
@@ -13,12 +13,16 @@
 
 
 class QueueEmpty(Exception):
-    'Exception raised by Queue.get(block=0)/get_nowait().'
+    """Exception raised when Queue.get_nowait() is called on a Queue object
+    which is empty.
+    """
     pass
 
 
 class QueueFull(Exception):
-    'Exception raised by Queue.put(block=0)/put_nowait().'
+    """Exception raised when the Queue.put_nowait() method is called on a Queue
+    object which is full.
+    """
     pass
 
 

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Tue Feb  3 16:57:57 2015
From: python-checkins at python.org (stefan.krah)
Date: Tue, 03 Feb 2015 15:57:57 +0000
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2314203=3A_Remove_o?=
 =?utf-8?q?bsolete_support_for_view=3D=3DNULL_in_PyBuffer=5FFillInfo=28=29?=
Message-ID: <20150203155754.39296.26507@psf.io>

https://hg.python.org/cpython/rev/e8fe32d43c96
changeset:   94490:e8fe32d43c96
user:        Stefan Krah <skrah at bytereef.org>
date:        Tue Feb 03 16:57:21 2015 +0100
summary:
  Issue #14203: Remove obsolete support for view==NULL in PyBuffer_FillInfo()
and bytearray_getbuffer().  Both functions now raise BufferError in that
case.

files:
  Misc/NEWS                 |   4 +++
  Modules/_testcapimodule.c |  34 +++++++++++++++++++++++++++
  Objects/abstract.c        |   7 ++++-
  Objects/bytearrayobject.c |  15 +++++------
  4 files changed, 51 insertions(+), 9 deletions(-)


diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -1560,6 +1560,10 @@
 C API
 -----
 
+- Issue #14203: Remove obsolete support for view==NULL in PyBuffer_FillInfo()
+  and bytearray_getbuffer().  Both functions now raise BufferError in that
+  case.
+
 - Issue #22445: PyBuffer_IsContiguous() now implements precise contiguity
   tests, compatible with NumPy's NPY_RELAXED_STRIDES_CHECKING compilation
   flag.  Previously the function reported false negatives for corner cases.
diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c
--- a/Modules/_testcapimodule.c
+++ b/Modules/_testcapimodule.c
@@ -2518,6 +2518,39 @@
 
     Py_RETURN_NONE;
 }
+ 
+static PyObject *
+test_pep3118_obsolete_write_locks(PyObject* self, PyObject *noargs)
+{
+    PyObject *b;
+    char *dummy[1];
+    int ret, match;
+
+    ret = PyBuffer_FillInfo(NULL, NULL, dummy, 1, 0, PyBUF_SIMPLE);
+    match = PyErr_Occurred() && PyErr_ExceptionMatches(PyExc_BufferError);
+    PyErr_Clear();
+    if (ret != -1 || match == 0)
+        goto error;
+
+    b = PyByteArray_FromStringAndSize("", 0);
+    if (b == NULL) {
+        return NULL;
+    }
+
+    ret = PyObject_GetBuffer(b, NULL, PyBUF_SIMPLE);
+    Py_DECREF(b);
+    match = PyErr_Occurred() && PyErr_ExceptionMatches(PyExc_BufferError);
+    PyErr_Clear();
+    if (ret != -1 || match == 0)
+        goto error;
+
+    Py_RETURN_NONE;
+
+error:
+    PyErr_SetString(TestError,
+        "test_pep3118_obsolete_write_locks: failure");
+    return NULL;
+}
 
 /* Test that the fatal error from not having a current thread doesn't
    cause an infinite loop.  Run via Lib/test/test_capi.py */
@@ -3179,6 +3212,7 @@
     {"test_unicode_compare_with_ascii", (PyCFunction)test_unicode_compare_with_ascii, METH_NOARGS},
     {"test_capsule", (PyCFunction)test_capsule, METH_NOARGS},
     {"test_from_contiguous", (PyCFunction)test_from_contiguous, METH_NOARGS},
+    {"test_pep3118_obsolete_write_locks", (PyCFunction)test_pep3118_obsolete_write_locks, METH_NOARGS},
     {"getargs_tuple",           getargs_tuple,                   METH_VARARGS},
     {"getargs_keywords", (PyCFunction)getargs_keywords,
       METH_VARARGS|METH_KEYWORDS},
diff --git a/Objects/abstract.c b/Objects/abstract.c
--- a/Objects/abstract.c
+++ b/Objects/abstract.c
@@ -612,7 +612,12 @@
 PyBuffer_FillInfo(Py_buffer *view, PyObject *obj, void *buf, Py_ssize_t len,
                   int readonly, int flags)
 {
-    if (view == NULL) return 0; /* XXX why not -1? */
+    if (view == NULL) {
+        PyErr_SetString(PyExc_BufferError,
+            "PyBuffer_FillInfo: view==NULL argument is obsolete");
+        return -1;
+    }
+
     if (((flags & PyBUF_WRITABLE) == PyBUF_WRITABLE) &&
         (readonly == 1)) {
         PyErr_SetString(PyExc_BufferError,
diff --git a/Objects/bytearrayobject.c b/Objects/bytearrayobject.c
--- a/Objects/bytearrayobject.c
+++ b/Objects/bytearrayobject.c
@@ -60,18 +60,17 @@
 static int
 bytearray_getbuffer(PyByteArrayObject *obj, Py_buffer *view, int flags)
 {
-    int ret;
     void *ptr;
     if (view == NULL) {
-        obj->ob_exports++;
-        return 0;
+        PyErr_SetString(PyExc_BufferError,
+            "bytearray_getbuffer: view==NULL argument is obsolete");
+        return -1;
     }
     ptr = (void *) PyByteArray_AS_STRING(obj);
-    ret = PyBuffer_FillInfo(view, (PyObject*)obj, ptr, Py_SIZE(obj), 0, flags);
-    if (ret >= 0) {
-        obj->ob_exports++;
-    }
-    return ret;
+    /* cannot fail if view != NULL and readonly == 0 */
+    (void)PyBuffer_FillInfo(view, (PyObject*)obj, ptr, Py_SIZE(obj), 0, flags);
+    obj->ob_exports++;
+    return 0;
 }
 
 static void

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Tue Feb  3 17:15:53 2015
From: python-checkins at python.org (raymond.hettinger)
Date: Tue, 03 Feb 2015 16:15:53 +0000
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_23359=3A__Reduce_siz?=
 =?utf-8?q?e_of_code_in_set=5Flookkey=2E_Only_do_linear_probes_when?=
Message-ID: <20150203161548.96088.40866@psf.io>

https://hg.python.org/cpython/rev/17cda5a92b6a
changeset:   94491:17cda5a92b6a
user:        Raymond Hettinger <python at rcn.com>
date:        Tue Feb 03 08:15:30 2015 -0800
summary:
  Issue 23359:  Reduce size of code in set_lookkey. Only do linear probes when there is no wrap-around.

Nice simplification contributed by Serhiy Storchaka :-)

files:
  Objects/setobject.c |  33 ---------------------------------
  1 files changed, 0 insertions(+), 33 deletions(-)


diff --git a/Objects/setobject.c b/Objects/setobject.c
--- a/Objects/setobject.c
+++ b/Objects/setobject.c
@@ -115,33 +115,6 @@
                 if (entry->key == dummy && freeslot == NULL)
                     freeslot = entry;
             }
-        } else {
-            for (j = 1 ; j <= LINEAR_PROBES ; j++) {
-                entry = &table[(i + j) & mask];
-                if (entry->key == NULL)
-                    goto found_null;
-                if (entry->hash == hash) {
-                    PyObject *startkey = entry->key;
-                    assert(startkey != dummy);
-                    if (startkey == key)
-                        return entry;
-                    if (PyUnicode_CheckExact(startkey)
-                        && PyUnicode_CheckExact(key)
-                        && unicode_eq(startkey, key))
-                        return entry;
-                    Py_INCREF(startkey);
-                    cmp = PyObject_RichCompareBool(startkey, key, Py_EQ);
-                    Py_DECREF(startkey);
-                    if (cmp < 0)
-                        return NULL;
-                    if (table != so->table || entry->key != startkey)
-                        return set_lookkey(so, key, hash);
-                    if (cmp > 0)
-                        return entry;
-                }
-                if (entry->key == dummy && freeslot == NULL)
-                    freeslot = entry;
-            }
         }
 
         perturb >>= PERTURB_SHIFT;
@@ -183,12 +156,6 @@
                 if (entry->key == NULL)
                     goto found_null;
             }
-        } else {
-            for (j = 1; j <= LINEAR_PROBES; j++) {
-                entry = &table[(i + j) & mask];
-                if (entry->key == NULL)
-                    goto found_null;
-            }
         }
         perturb >>= PERTURB_SHIFT;
         i = (i * 5 + 1 + perturb) & mask;

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Tue Feb  3 17:53:35 2015
From: python-checkins at python.org (serhiy.storchaka)
Date: Tue, 03 Feb 2015 16:53:35 +0000
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2315381=3A_Fixed_a_?=
 =?utf-8?q?bug_in_BytesIO=2Ewrite=28=29=2E?=
Message-ID: <20150203165256.39276.88303@psf.io>

https://hg.python.org/cpython/rev/ea33b61cac74
changeset:   94492:ea33b61cac74
user:        Serhiy Storchaka <storchaka at gmail.com>
date:        Tue Feb 03 18:51:58 2015 +0200
summary:
  Issue #15381: Fixed a bug in BytesIO.write().
It was expected that string_size == PyBytes_GET_SIZE(buf) if the buffer is
shared, but truncate() and __setstate__() can set string_size without
unsharing the buffer.

files:
  Modules/_io/bytesio.c |  19 ++++++++++---------
  1 files changed, 10 insertions(+), 9 deletions(-)


diff --git a/Modules/_io/bytesio.c b/Modules/_io/bytesio.c
--- a/Modules/_io/bytesio.c
+++ b/Modules/_io/bytesio.c
@@ -19,7 +19,7 @@
 
 /* The bytesio object can be in three states:
   * Py_REFCNT(buf) == 1, exports == 0.
-  * Py_REFCNT(buf) > 1.  exports == 0, string_size == PyBytes_GET_SIZE(buf),
+  * Py_REFCNT(buf) > 1.  exports == 0,
     first modification or export causes the internal buffer copying.
   * exports > 0.  Py_REFCNT(buf) == 1, any modifications are forbidden.
 */
@@ -38,8 +38,7 @@
         return NULL; \
     }
 
-#define SHARED_BUF(self) (Py_REFCNT((self)->buf) > 1 || \
-                          PyBytes_GET_SIZE((self)->buf) <= 1)
+#define SHARED_BUF(self) (Py_REFCNT((self)->buf) > 1)
 
 
 /* Internal routine to get a line from the buffer of a BytesIO
@@ -152,16 +151,18 @@
 static Py_ssize_t
 write_bytes(bytesio *self, const char *bytes, Py_ssize_t len)
 {
+    size_t endpos;
     assert(self->buf != NULL);
     assert(self->pos >= 0);
     assert(len >= 0);
 
-    if ((size_t)self->pos + len > (size_t)PyBytes_GET_SIZE(self->buf)) {
-        if (resize_buffer(self, (size_t)self->pos + len) < 0)
+    endpos = (size_t)self->pos + len;
+    if (endpos > (size_t)PyBytes_GET_SIZE(self->buf)) {
+        if (resize_buffer(self, endpos) < 0)
             return -1;
     }
     else if (SHARED_BUF(self)) {
-        if (unshare_buffer(self, self->string_size) < 0)
+        if (unshare_buffer(self, Py_MAX(endpos, (size_t)self->string_size)) < 0)
             return -1;
     }
 
@@ -181,11 +182,11 @@
     /* Copy the data to the internal buffer, overwriting some of the existing
        data if self->pos < self->string_size. */
     memcpy(PyBytes_AS_STRING(self->buf) + self->pos, bytes, len);
-    self->pos += len;
+    self->pos = endpos;
 
     /* Set the new length of the internal string if it has changed. */
-    if (self->string_size < self->pos) {
-        self->string_size = self->pos;
+    if ((size_t)self->string_size < endpos) {
+        self->string_size = endpos;
     }
 
     return len;

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Tue Feb  3 21:43:53 2015
From: python-checkins at python.org (stefan.krah)
Date: Tue, 03 Feb 2015 20:43:53 +0000
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2314203=3A_Remove_o?=
 =?utf-8?q?bsolete_support_for_view=3D=3DNULL_in_bytesiobuf=5Fgetbuffer=28?=
 =?utf-8?q?=29?=
Message-ID: <20150203204350.34384.84889@psf.io>

https://hg.python.org/cpython/rev/0d5095a2422f
changeset:   94493:0d5095a2422f
user:        Stefan Krah <skrah at bytereef.org>
date:        Tue Feb 03 21:43:23 2015 +0100
summary:
  Issue #14203: Remove obsolete support for view==NULL in bytesiobuf_getbuffer()
and array_buffer_getbuf().

files:
  Lib/test/test_array.py    |   5 +++++
  Lib/test/test_bytes.py    |   4 ++++
  Misc/NEWS                 |   6 +++---
  Modules/_io/bytesio.c     |  21 +++++++++++----------
  Modules/_testcapimodule.c |  22 ++++++++++++++++++++--
  Modules/arraymodule.c     |   7 +++++--
  6 files changed, 48 insertions(+), 17 deletions(-)


diff --git a/Lib/test/test_array.py b/Lib/test/test_array.py
--- a/Lib/test/test_array.py
+++ b/Lib/test/test_array.py
@@ -1041,6 +1041,11 @@
             a = array.array(self.typecode, "foo")
             a = array.array(self.typecode, array.array('u', 'foo'))
 
+    @support.cpython_only
+    def test_obsolete_write_lock(self):
+        from _testcapi import getbuffer_with_null_view
+        a = array.array('B', b"")
+        self.assertRaises(BufferError, getbuffer_with_null_view, a)
 
 class StringTest(BaseTest):
 
diff --git a/Lib/test/test_bytes.py b/Lib/test/test_bytes.py
--- a/Lib/test/test_bytes.py
+++ b/Lib/test/test_bytes.py
@@ -1224,6 +1224,10 @@
         self.assertRaises(BufferError, delslice)
         self.assertEqual(b, orig)
 
+    @test.support.cpython_only
+    def test_obsolete_write_lock(self):
+        from _testcapi import getbuffer_with_null_view
+        self.assertRaises(BufferError, getbuffer_with_null_view, bytearray())
 
 class AssortedBytesTest(unittest.TestCase):
     #
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -1560,9 +1560,9 @@
 C API
 -----
 
-- Issue #14203: Remove obsolete support for view==NULL in PyBuffer_FillInfo()
-  and bytearray_getbuffer().  Both functions now raise BufferError in that
-  case.
+- Issue #14203: Remove obsolete support for view==NULL in PyBuffer_FillInfo(),
+  bytearray_getbuffer(), bytesiobuf_getbuffer() and array_buffer_getbuf().
+  All functions now raise BufferError in that case.
 
 - Issue #22445: PyBuffer_IsContiguous() now implements precise contiguity
   tests, compatible with NumPy's NPY_RELAXED_STRIDES_CHECKING compilation
diff --git a/Modules/_io/bytesio.c b/Modules/_io/bytesio.c
--- a/Modules/_io/bytesio.c
+++ b/Modules/_io/bytesio.c
@@ -1028,23 +1028,24 @@
 static int
 bytesiobuf_getbuffer(bytesiobuf *obj, Py_buffer *view, int flags)
 {
-    int ret;
     bytesio *b = (bytesio *) obj->source;
+
+    if (view == NULL) {
+        PyErr_SetString(PyExc_BufferError,
+            "bytesiobuf_getbuffer: view==NULL argument is obsolete");
+        return -1;
+    }
     if (SHARED_BUF(b)) {
         if (unshare_buffer(b, b->string_size) < 0)
             return -1;
     }
-    if (view == NULL) {
-        b->exports++;
-        return 0;
-    }
-    ret = PyBuffer_FillInfo(view, (PyObject*)obj,
+
+    /* cannot fail if view != NULL and readonly == 0 */
+    (void)PyBuffer_FillInfo(view, (PyObject*)obj,
                             PyBytes_AS_STRING(b->buf), b->string_size,
                             0, flags);
-    if (ret >= 0) {
-        b->exports++;
-    }
-    return ret;
+    b->exports++;
+    return 0;
 }
 
 static void
diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c
--- a/Modules/_testcapimodule.c
+++ b/Modules/_testcapimodule.c
@@ -2518,21 +2518,26 @@
 
     Py_RETURN_NONE;
 }
- 
+
+extern PyTypeObject _PyBytesIOBuffer_Type;
+
 static PyObject *
 test_pep3118_obsolete_write_locks(PyObject* self, PyObject *noargs)
 {
+    PyTypeObject *type = &_PyBytesIOBuffer_Type;
     PyObject *b;
     char *dummy[1];
     int ret, match;
 
+    /* PyBuffer_FillInfo() */
     ret = PyBuffer_FillInfo(NULL, NULL, dummy, 1, 0, PyBUF_SIMPLE);
     match = PyErr_Occurred() && PyErr_ExceptionMatches(PyExc_BufferError);
     PyErr_Clear();
     if (ret != -1 || match == 0)
         goto error;
 
-    b = PyByteArray_FromStringAndSize("", 0);
+    /* bytesiobuf_getbuffer() */
+    b = type->tp_alloc(type, 0);
     if (b == NULL) {
         return NULL;
     }
@@ -2552,6 +2557,18 @@
     return NULL;
 }
 
+/* This tests functions that historically supported write locks.  It is
+   wrong to call getbuffer() with view==NULL and a compliant getbufferproc
+   is entitled to segfault in that case. */
+static PyObject *
+getbuffer_with_null_view(PyObject* self, PyObject *obj)
+{
+    if (PyObject_GetBuffer(obj, NULL, PyBUF_SIMPLE) < 0)
+        return NULL;
+
+    Py_RETURN_NONE;
+}
+
 /* Test that the fatal error from not having a current thread doesn't
    cause an infinite loop.  Run via Lib/test/test_capi.py */
 static PyObject *
@@ -3213,6 +3230,7 @@
     {"test_capsule", (PyCFunction)test_capsule, METH_NOARGS},
     {"test_from_contiguous", (PyCFunction)test_from_contiguous, METH_NOARGS},
     {"test_pep3118_obsolete_write_locks", (PyCFunction)test_pep3118_obsolete_write_locks, METH_NOARGS},
+    {"getbuffer_with_null_view", getbuffer_with_null_view, METH_O},
     {"getargs_tuple",           getargs_tuple,                   METH_VARARGS},
     {"getargs_keywords", (PyCFunction)getargs_keywords,
       METH_VARARGS|METH_KEYWORDS},
diff --git a/Modules/arraymodule.c b/Modules/arraymodule.c
--- a/Modules/arraymodule.c
+++ b/Modules/arraymodule.c
@@ -2530,7 +2530,11 @@
 static int
 array_buffer_getbuf(arrayobject *self, Py_buffer *view, int flags)
 {
-    if (view==NULL) goto finish;
+    if (view == NULL) {
+        PyErr_SetString(PyExc_BufferError,
+            "array_buffer_getbuf: view==NULL argument is obsolete");
+        return -1;
+    }
 
     view->buf = (void *)self->ob_item;
     view->obj = (PyObject*)self;
@@ -2560,7 +2564,6 @@
 #endif
     }
 
- finish:
     self->ob_exports++;
     return 0;
 }

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Tue Feb  3 22:27:50 2015
From: python-checkins at python.org (stefan.krah)
Date: Tue, 03 Feb 2015 21:27:50 +0000
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2314203=3A__Tempora?=
 =?utf-8?q?ry_fix_for_the_compile_failure_on_Windows=2E?=
Message-ID: <20150203212749.96076.79334@psf.io>

https://hg.python.org/cpython/rev/17a8c5f8ca48
changeset:   94494:17a8c5f8ca48
user:        Stefan Krah <skrah at bytereef.org>
date:        Tue Feb 03 22:27:21 2015 +0100
summary:
  Issue #14203:  Temporary fix for the compile failure on Windows.

files:
  Modules/_testcapimodule.c |  4 ++++
  1 files changed, 4 insertions(+), 0 deletions(-)


diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c
--- a/Modules/_testcapimodule.c
+++ b/Modules/_testcapimodule.c
@@ -2519,6 +2519,7 @@
     Py_RETURN_NONE;
 }
 
+#if (defined(__linux__) || defined(__FreeBSD__)) && defined(__GNUC__)
 extern PyTypeObject _PyBytesIOBuffer_Type;
 
 static PyObject *
@@ -2556,6 +2557,7 @@
         "test_pep3118_obsolete_write_locks: failure");
     return NULL;
 }
+#endif
 
 /* This tests functions that historically supported write locks.  It is
    wrong to call getbuffer() with view==NULL and a compliant getbufferproc
@@ -3229,7 +3231,9 @@
     {"test_unicode_compare_with_ascii", (PyCFunction)test_unicode_compare_with_ascii, METH_NOARGS},
     {"test_capsule", (PyCFunction)test_capsule, METH_NOARGS},
     {"test_from_contiguous", (PyCFunction)test_from_contiguous, METH_NOARGS},
+#if (defined(__linux__) || defined(__FreeBSD__)) && defined(__GNUC__)
     {"test_pep3118_obsolete_write_locks", (PyCFunction)test_pep3118_obsolete_write_locks, METH_NOARGS},
+#endif
     {"getbuffer_with_null_view", getbuffer_with_null_view, METH_O},
     {"getargs_tuple",           getargs_tuple,                   METH_VARARGS},
     {"getargs_keywords", (PyCFunction)getargs_keywords,

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Wed Feb  4 07:09:38 2015
From: python-checkins at python.org (serhiy.storchaka)
Date: Wed, 04 Feb 2015 06:09:38 +0000
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWVzICMyMzM2?=
 =?utf-8?q?3=2C_=2323364=2C_=2323365=2C_=2323366=3A_Fixed_itertools_overfl?=
 =?utf-8?q?ow_tests=2E?=
Message-ID: <20150204060935.25871.27595@psf.io>

https://hg.python.org/cpython/rev/887526ebb013
changeset:   94495:887526ebb013
branch:      2.7
parent:      94466:83b75b016c77
user:        Serhiy Storchaka <storchaka at gmail.com>
date:        Tue Feb 03 01:34:09 2015 +0200
summary:
  Issues #23363, #23364, #23365, #23366: Fixed itertools overflow tests.
Used PyMem_New to check overflow.

files:
  Lib/test/test_itertools.py |  12 +++++-------
  Modules/itertoolsmodule.c  |  26 ++++++--------------------
  2 files changed, 11 insertions(+), 27 deletions(-)


diff --git a/Lib/test/test_itertools.py b/Lib/test/test_itertools.py
--- a/Lib/test/test_itertools.py
+++ b/Lib/test/test_itertools.py
@@ -139,7 +139,7 @@
 
     @test_support.bigaddrspacetest
     def test_combinations_overflow(self):
-        with self.assertRaises(OverflowError):
+        with self.assertRaises((OverflowError, MemoryError)):
             combinations("AA", 2**29)
 
     @test_support.impl_detail("tuple reuse is specific to CPython")
@@ -215,7 +215,7 @@
 
     @test_support.bigaddrspacetest
     def test_combinations_with_replacement_overflow(self):
-        with self.assertRaises(OverflowError):
+        with self.assertRaises((OverflowError, MemoryError)):
             combinations_with_replacement("AA", 2**30)
 
     @test_support.impl_detail("tuple reuse is specific to CPython")
@@ -286,10 +286,8 @@
 
     @test_support.bigaddrspacetest
     def test_permutations_overflow(self):
-        with self.assertRaises(OverflowError):
+        with self.assertRaises((OverflowError, MemoryError)):
             permutations("A", 2**30)
-        with self.assertRaises(OverflowError):
-            permutations("A", 2, 2**30)
 
     @test_support.impl_detail("tuple reuse is specific to CPython")
     def test_permutations_tuple_reuse(self):
@@ -711,8 +709,8 @@
 
     @test_support.bigaddrspacetest
     def test_product_overflow(self):
-        with self.assertRaises(OverflowError):
-            product(["a"]*(2**16), repeat=2**16)
+        with self.assertRaises((OverflowError, MemoryError)):
+            product(*(['ab']*2**5), repeat=2**25)
 
     @test_support.impl_detail("tuple reuse is specific to CPython")
     def test_product_tuple_reuse(self):
diff --git a/Modules/itertoolsmodule.c b/Modules/itertoolsmodule.c
--- a/Modules/itertoolsmodule.c
+++ b/Modules/itertoolsmodule.c
@@ -1847,15 +1847,14 @@
         nargs = 0;
     } else {
         nargs = PyTuple_GET_SIZE(args);
-        if (repeat > PY_SSIZE_T_MAX/sizeof(Py_ssize_t) ||
-            nargs > PY_SSIZE_T_MAX/(repeat * sizeof(Py_ssize_t))) {
+        if ((size_t)nargs > PY_SSIZE_T_MAX/sizeof(Py_ssize_t)/repeat) {
             PyErr_SetString(PyExc_OverflowError, "repeat argument too large");
             return NULL;
         }
     }
     npools = nargs * repeat;
 
-    indices = PyMem_Malloc(npools * sizeof(Py_ssize_t));
+    indices = PyMem_New(Py_ssize_t, npools);
     if (indices == NULL) {
         PyErr_NoMemory();
         goto error;
@@ -2102,11 +2101,7 @@
         goto error;
     }
 
-    if (r > PY_SSIZE_T_MAX/sizeof(Py_ssize_t)) {
-        PyErr_SetString(PyExc_OverflowError, "r is too big");
-        goto error;
-    }
-    indices = PyMem_Malloc(r * sizeof(Py_ssize_t));
+    indices = PyMem_New(Py_ssize_t, r);
     if (indices == NULL) {
         PyErr_NoMemory();
         goto error;
@@ -2355,11 +2350,7 @@
         goto error;
     }
 
-    if (r > PY_SSIZE_T_MAX/sizeof(Py_ssize_t)) {
-        PyErr_SetString(PyExc_OverflowError, "r is too big");
-        goto error;
-    }
-    indices = PyMem_Malloc(r * sizeof(Py_ssize_t));
+    indices = PyMem_New(Py_ssize_t, r);
     if (indices == NULL) {
         PyErr_NoMemory();
         goto error;
@@ -2612,13 +2603,8 @@
         goto error;
     }
 
-    if (n > PY_SSIZE_T_MAX/sizeof(Py_ssize_t) ||
-        r > PY_SSIZE_T_MAX/sizeof(Py_ssize_t)) {
-        PyErr_SetString(PyExc_OverflowError, "parameters too large");
-        goto error;
-    }
-    indices = PyMem_Malloc(n * sizeof(Py_ssize_t));
-    cycles = PyMem_Malloc(r * sizeof(Py_ssize_t));
+    indices = PyMem_New(Py_ssize_t, n);
+    cycles = PyMem_New(Py_ssize_t, r);
     if (indices == NULL || cycles == NULL) {
         PyErr_NoMemory();
         goto error;

-- 
Repository URL: https://hg.python.org/cpython

From solipsis at pitrou.net  Wed Feb  4 09:33:48 2015
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Wed, 04 Feb 2015 09:33:48 +0100
Subject: [Python-checkins] Daily reference leaks (17a8c5f8ca48): sum=3
Message-ID: <E1YIvPA-0007J2-7f@ssl.pitrou.net>

results for 17a8c5f8ca48 on branch "default"
--------------------------------------------

test_functools leaked [0, 0, 3] memory blocks, sum=3


Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogzoUP3t', '-x']

From python-checkins at python.org  Wed Feb  4 10:04:50 2015
From: python-checkins at python.org (gregory.p.smith)
Date: Wed, 04 Feb 2015 09:04:50 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?q?=29=3A_Skip_some_tests_that_require_a_subinterpreter_launched_w?=
 =?utf-8?q?ith_-E_or_-I_when_the?=
Message-ID: <20150204090446.106325.46588@psf.io>

https://hg.python.org/cpython/rev/0bd76ade1c7f
changeset:   94497:0bd76ade1c7f
parent:      94494:17a8c5f8ca48
parent:      94496:ef1d338bec8f
user:        Gregory P. Smith <greg at krypto.org>
date:        Wed Feb 04 01:04:31 2015 -0800
summary:
  Skip some tests that require a subinterpreter launched with -E or -I when the
interpreter under test is being run in an environment that requires the use of
environment variables such as PYTHONHOME in order to function at all.

Adds a test.script_helper.interpreter_requires_environment() function
to be used with @unittest.skipIf on stdlib test methods requiring this.

files:
  Lib/test/script_helper.py      |  35 ++++++++++++++++++
  Lib/test/test_cmd_line.py      |   4 +-
  Lib/test/test_script_helper.py |  41 ++++++++++++++++++++++
  Lib/test/test_tracemalloc.py   |   4 +-
  4 files changed, 82 insertions(+), 2 deletions(-)


diff --git a/Lib/test/script_helper.py b/Lib/test/script_helper.py
--- a/Lib/test/script_helper.py
+++ b/Lib/test/script_helper.py
@@ -15,6 +15,41 @@
 from importlib.util import source_from_cache
 from test.support import make_legacy_pyc, strip_python_stderr, temp_dir
 
+
+# Cached result of the expensive test performed in the function below.
+__cached_interp_requires_environment = None
+
+def interpreter_requires_environment():
+    """
+    Returns True if our sys.executable interpreter requires environment
+    variables in order to be able to run at all.
+
+    This is designed to be used with @unittest.skipIf() to annotate tests
+    that need to use an assert_python*() function to launch an isolated
+    mode (-I) or no environment mode (-E) sub-interpreter process.
+
+    A normal build & test does not run into this situation but it can happen
+    when trying to run the standard library test suite from an interpreter that
+    doesn't have an obvious home with Python's current home finding logic.
+
+    Setting PYTHONHOME is one way to get most of the testsuite to run in that
+    situation.  PYTHONPATH or PYTHONUSERSITE are other common envirnonment
+    variables that might impact whether or not the interpreter can start.
+    """
+    global __cached_interp_requires_environment
+    if __cached_interp_requires_environment is None:
+        # Try running an interpreter with -E to see if it works or not.
+        try:
+            subprocess.check_call([sys.executable, '-E',
+                                   '-c', 'import sys; sys.exit(0)'])
+        except subprocess.CalledProcessError:
+            __cached_interp_requires_environment = True
+        else:
+            __cached_interp_requires_environment = False
+
+    return __cached_interp_requires_environment
+
+
 # Executing the interpreter in a subprocess
 def _assert_python(expected_success, *args, **env_vars):
     if '__isolated' in env_vars:
diff --git a/Lib/test/test_cmd_line.py b/Lib/test/test_cmd_line.py
--- a/Lib/test/test_cmd_line.py
+++ b/Lib/test/test_cmd_line.py
@@ -8,6 +8,7 @@
 import sys
 import subprocess
 import tempfile
+from test import script_helper
 from test.script_helper import (spawn_python, kill_python, assert_python_ok,
     assert_python_failure)
 
@@ -439,7 +440,8 @@
         self.assertEqual(err.splitlines().count(b'Unknown option: -a'), 1)
         self.assertEqual(b'', out)
 
-
+    @unittest.skipIf(script_helper.interpreter_requires_environment(),
+                     'Cannot run -I tests when PYTHON env vars are required.')
     def test_isolatedmode(self):
         self.verify_valid_flag('-I')
         self.verify_valid_flag('-IEs')
diff --git a/Lib/test/test_script_helper.py b/Lib/test/test_script_helper.py
--- a/Lib/test/test_script_helper.py
+++ b/Lib/test/test_script_helper.py
@@ -1,7 +1,10 @@
 """Unittests for test.script_helper.  Who tests the test helper?"""
 
+import subprocess
+import sys
 from test import script_helper
 import unittest
+from unittest import mock
 
 
 class TestScriptHelper(unittest.TestCase):
@@ -31,5 +34,43 @@
                       msg='unexpected command line.')
 
 
+class TestScriptHelperEnvironment(unittest.TestCase):
+    """Code coverage for interpreter_requires_environment()."""
+
+    def setUp(self):
+        self.assertTrue(
+                hasattr(script_helper, '__cached_interp_requires_environment'))
+        # Reset the private cached state.
+        script_helper.__dict__['__cached_interp_requires_environment'] = None
+
+    def tearDown(self):
+        # Reset the private cached state.
+        script_helper.__dict__['__cached_interp_requires_environment'] = None
+
+    @mock.patch('subprocess.check_call')
+    def test_interpreter_requires_environment_true(self, mock_check_call):
+        mock_check_call.side_effect = subprocess.CalledProcessError('', '')
+        self.assertTrue(script_helper.interpreter_requires_environment())
+        self.assertTrue(script_helper.interpreter_requires_environment())
+        self.assertEqual(1, mock_check_call.call_count)
+
+    @mock.patch('subprocess.check_call')
+    def test_interpreter_requires_environment_false(self, mock_check_call):
+        # The mocked subprocess.check_call fakes a no-error process.
+        script_helper.interpreter_requires_environment()
+        self.assertFalse(script_helper.interpreter_requires_environment())
+        self.assertEqual(1, mock_check_call.call_count)
+
+    @mock.patch('subprocess.check_call')
+    def test_interpreter_requires_environment_details(self, mock_check_call):
+        script_helper.interpreter_requires_environment()
+        self.assertFalse(script_helper.interpreter_requires_environment())
+        self.assertFalse(script_helper.interpreter_requires_environment())
+        self.assertEqual(1, mock_check_call.call_count)
+        check_call_command = mock_check_call.call_args[0][0]
+        self.assertEqual(sys.executable, check_call_command[0])
+        self.assertIn('-E', check_call_command)
+
+
 if __name__ == '__main__':
     unittest.main()
diff --git a/Lib/test/test_tracemalloc.py b/Lib/test/test_tracemalloc.py
--- a/Lib/test/test_tracemalloc.py
+++ b/Lib/test/test_tracemalloc.py
@@ -5,7 +5,7 @@
 import unittest
 from unittest.mock import patch
 from test.script_helper import assert_python_ok, assert_python_failure
-from test import support
+from test import script_helper, support
 try:
     import threading
 except ImportError:
@@ -755,6 +755,8 @@
         stdout = stdout.rstrip()
         self.assertEqual(stdout, b'False')
 
+    @unittest.skipIf(script_helper.interpreter_requires_environment(),
+                     'Cannot run -E tests when PYTHON env vars are required.')
     def test_env_var_ignored_with_E(self):
         """PYTHON* environment variables must be ignored when -E is present."""
         code = 'import tracemalloc; print(tracemalloc.is_tracing())'

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Wed Feb  4 10:04:50 2015
From: python-checkins at python.org (gregory.p.smith)
Date: Wed, 04 Feb 2015 09:04:50 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Skip_some_test?=
 =?utf-8?q?s_that_require_a_subinterpreter_launched_with_-E_or_-I_when_the?=
Message-ID: <20150204090446.25871.12134@psf.io>

https://hg.python.org/cpython/rev/ef1d338bec8f
changeset:   94496:ef1d338bec8f
branch:      3.4
parent:      94488:fd445d2dd107
user:        Gregory P. Smith <greg at krypto.org>
date:        Wed Feb 04 00:59:40 2015 -0800
summary:
  Skip some tests that require a subinterpreter launched with -E or -I when the
interpreter under test is being run in an environment that requires the use of
environment variables such as PYTHONHOME in order to function at all.

Adds a private test.script_helper._interpreter_requires_environment() function
to be used with @unittest.skipIf on stdlib test methods requiring this.

files:
  Lib/test/script_helper.py      |  35 ++++++++++++++++++
  Lib/test/test_cmd_line.py      |   4 +-
  Lib/test/test_script_helper.py |  41 ++++++++++++++++++++++
  Lib/test/test_tracemalloc.py   |   4 +-
  4 files changed, 82 insertions(+), 2 deletions(-)


diff --git a/Lib/test/script_helper.py b/Lib/test/script_helper.py
--- a/Lib/test/script_helper.py
+++ b/Lib/test/script_helper.py
@@ -15,6 +15,41 @@
 from importlib.util import source_from_cache
 from test.support import make_legacy_pyc, strip_python_stderr, temp_dir
 
+
+# Cached result of the expensive test performed in the function below.
+__cached_interp_requires_environment = None
+
+def _interpreter_requires_environment():
+    """
+    Returns True if our sys.executable interpreter requires environment
+    variables in order to be able to run at all.
+
+    This is designed to be used with @unittest.skipIf() to annotate tests
+    that need to use an assert_python*() function to launch an isolated
+    mode (-I) or no environment mode (-E) sub-interpreter process.
+
+    A normal build & test does not run into this situation but it can happen
+    when trying to run the standard library test suite from an interpreter that
+    doesn't have an obvious home with Python's current home finding logic.
+
+    Setting PYTHONHOME is one way to get most of the testsuite to run in that
+    situation.  PYTHONPATH or PYTHONUSERSITE are other common envirnonment
+    variables that might impact whether or not the interpreter can start.
+    """
+    global __cached_interp_requires_environment
+    if __cached_interp_requires_environment is None:
+        # Try running an interpreter with -E to see if it works or not.
+        try:
+            subprocess.check_call([sys.executable, '-E',
+                                   '-c', 'import sys; sys.exit(0)'])
+        except subprocess.CalledProcessError:
+            __cached_interp_requires_environment = True
+        else:
+            __cached_interp_requires_environment = False
+
+    return __cached_interp_requires_environment
+
+
 # Executing the interpreter in a subprocess
 def _assert_python(expected_success, *args, **env_vars):
     if '__isolated' in env_vars:
diff --git a/Lib/test/test_cmd_line.py b/Lib/test/test_cmd_line.py
--- a/Lib/test/test_cmd_line.py
+++ b/Lib/test/test_cmd_line.py
@@ -8,6 +8,7 @@
 import sys
 import subprocess
 import tempfile
+from test import script_helper
 from test.script_helper import (spawn_python, kill_python, assert_python_ok,
     assert_python_failure)
 
@@ -439,7 +440,8 @@
         self.assertEqual(err.splitlines().count(b'Unknown option: -a'), 1)
         self.assertEqual(b'', out)
 
-
+    @unittest.skipIf(script_helper._interpreter_requires_environment(),
+                     'Cannot run -I tests when PYTHON env vars are required.')
     def test_isolatedmode(self):
         self.verify_valid_flag('-I')
         self.verify_valid_flag('-IEs')
diff --git a/Lib/test/test_script_helper.py b/Lib/test/test_script_helper.py
--- a/Lib/test/test_script_helper.py
+++ b/Lib/test/test_script_helper.py
@@ -1,7 +1,10 @@
 """Unittests for test.script_helper.  Who tests the test helper?"""
 
+import subprocess
+import sys
 from test import script_helper
 import unittest
+from unittest import mock
 
 
 class TestScriptHelper(unittest.TestCase):
@@ -31,5 +34,43 @@
                       msg='unexpected command line.')
 
 
+class TestScriptHelperEnvironment(unittest.TestCase):
+    """Code coverage for _interpreter_requires_environment()."""
+
+    def setUp(self):
+        self.assertTrue(
+                hasattr(script_helper, '__cached_interp_requires_environment'))
+        # Reset the private cached state.
+        script_helper.__dict__['__cached_interp_requires_environment'] = None
+
+    def tearDown(self):
+        # Reset the private cached state.
+        script_helper.__dict__['__cached_interp_requires_environment'] = None
+
+    @mock.patch('subprocess.check_call')
+    def test_interpreter_requires_environment_true(self, mock_check_call):
+        mock_check_call.side_effect = subprocess.CalledProcessError('', '')
+        self.assertTrue(script_helper._interpreter_requires_environment())
+        self.assertTrue(script_helper._interpreter_requires_environment())
+        self.assertEqual(1, mock_check_call.call_count)
+
+    @mock.patch('subprocess.check_call')
+    def test_interpreter_requires_environment_false(self, mock_check_call):
+        # The mocked subprocess.check_call fakes a no-error process.
+        script_helper._interpreter_requires_environment()
+        self.assertFalse(script_helper._interpreter_requires_environment())
+        self.assertEqual(1, mock_check_call.call_count)
+
+    @mock.patch('subprocess.check_call')
+    def test_interpreter_requires_environment_details(self, mock_check_call):
+        script_helper._interpreter_requires_environment()
+        self.assertFalse(script_helper._interpreter_requires_environment())
+        self.assertFalse(script_helper._interpreter_requires_environment())
+        self.assertEqual(1, mock_check_call.call_count)
+        check_call_command = mock_check_call.call_args[0][0]
+        self.assertEqual(sys.executable, check_call_command[0])
+        self.assertIn('-E', check_call_command)
+
+
 if __name__ == '__main__':
     unittest.main()
diff --git a/Lib/test/test_tracemalloc.py b/Lib/test/test_tracemalloc.py
--- a/Lib/test/test_tracemalloc.py
+++ b/Lib/test/test_tracemalloc.py
@@ -5,7 +5,7 @@
 import unittest
 from unittest.mock import patch
 from test.script_helper import assert_python_ok, assert_python_failure
-from test import support
+from test import script_helper, support
 try:
     import threading
 except ImportError:
@@ -755,6 +755,8 @@
         stdout = stdout.rstrip()
         self.assertEqual(stdout, b'False')
 
+    @unittest.skipIf(script_helper._interpreter_requires_environment(),
+                     'Cannot run -E tests when PYTHON env vars are required.')
     def test_env_var_ignored_with_E(self):
         """PYTHON* environment variables must be ignored when -E is present."""
         code = 'import tracemalloc; print(tracemalloc.is_tracing())'

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Wed Feb  4 11:20:29 2015
From: python-checkins at python.org (gregory.p.smith)
Date: Wed, 04 Feb 2015 10:20:29 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?q?=29=3A_Fixes_issue23390=3A_make_profile-opt_causes_-fprofile-ge?=
 =?utf-8?q?nerate_and_related_flags?=
Message-ID: <20150204102019.34398.92598@psf.io>

https://hg.python.org/cpython/rev/9c46707e5526
changeset:   94499:9c46707e5526
parent:      94497:0bd76ade1c7f
parent:      94498:8957ff9776bd
user:        Gregory P. Smith <greg at krypto.org>
date:        Wed Feb 04 02:16:13 2015 -0800
summary:
  Fixes issue23390: make profile-opt causes -fprofile-generate and related flags
to end up in distutils CFLAGS.

files:
  Makefile.pre.in |  4 ++--
  1 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/Makefile.pre.in b/Makefile.pre.in
--- a/Makefile.pre.in
+++ b/Makefile.pre.in
@@ -488,14 +488,14 @@
 	$(MAKE) build_all_use_profile
 
 build_all_generate_profile:
-	$(MAKE) all CFLAGS="$(CFLAGS) -fprofile-generate" LIBS="$(LIBS) -lgcov"
+	$(MAKE) all CFLAGS_NODIST="$(CFLAGS) -fprofile-generate" LDFLAGS="-fprofile-generate" LIBS="$(LIBS) -lgcov"
 
 run_profile_task:
 	: # FIXME: can't run for a cross build
 	$(RUNSHARED) ./$(BUILDPYTHON) $(PROFILE_TASK)
 
 build_all_use_profile:
-	$(MAKE) all CFLAGS="$(CFLAGS) -fprofile-use -fprofile-correction"
+	$(MAKE) all CFLAGS_NODIST="$(CFLAGS) -fprofile-use -fprofile-correction"
 
 # Compile and run with gcov
 .PHONY=coverage coverage-lcov coverage-report

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Wed Feb  4 11:20:29 2015
From: python-checkins at python.org (gregory.p.smith)
Date: Wed, 04 Feb 2015 10:20:29 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Fixes_issue233?=
 =?utf-8?q?90=3A_make_profile-opt_causes_-fprofile-generate_and_related_fl?=
 =?utf-8?q?ags?=
Message-ID: <20150204102019.106264.95091@psf.io>

https://hg.python.org/cpython/rev/8957ff9776bd
changeset:   94498:8957ff9776bd
branch:      3.4
parent:      94496:ef1d338bec8f
user:        Gregory P. Smith <greg at krypto.org>
date:        Wed Feb 04 02:11:56 2015 -0800
summary:
  Fixes issue23390: make profile-opt causes -fprofile-generate and related flags
to end up in distutils CFLAGS.

files:
  Makefile.pre.in |  4 ++--
  1 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/Makefile.pre.in b/Makefile.pre.in
--- a/Makefile.pre.in
+++ b/Makefile.pre.in
@@ -480,14 +480,14 @@
 	$(MAKE) build_all_use_profile
 
 build_all_generate_profile:
-	$(MAKE) all CFLAGS="$(CFLAGS) -fprofile-generate" LIBS="$(LIBS) -lgcov"
+	$(MAKE) all CFLAGS_NODIST="$(CFLAGS) -fprofile-generate" LDFLAGS="-fprofile-generate" LIBS="$(LIBS) -lgcov"
 
 run_profile_task:
 	: # FIXME: can't run for a cross build
 	$(RUNSHARED) ./$(BUILDPYTHON) $(PROFILE_TASK)
 
 build_all_use_profile:
-	$(MAKE) all CFLAGS="$(CFLAGS) -fprofile-use -fprofile-correction"
+	$(MAKE) all CFLAGS_NODIST="$(CFLAGS) -fprofile-use -fprofile-correction"
 
 # Compile and run with gcov
 .PHONY=coverage coverage-lcov coverage-report

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Wed Feb  4 14:54:22 2015
From: python-checkins at python.org (victor.stinner)
Date: Wed, 04 Feb 2015 13:54:22 +0000
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbzogQmFz?=
 =?utf-8?q?eSelectorEventLoop_uses_directly_the_private_=5Fdebug_attribute?=
Message-ID: <20150204135422.96094.76300@psf.io>

https://hg.python.org/cpython/rev/c02cde905034
changeset:   94500:c02cde905034
branch:      3.4
parent:      94498:8957ff9776bd
user:        Victor Stinner <victor.stinner at gmail.com>
date:        Wed Feb 04 14:50:59 2015 +0100
summary:
  asyncio: BaseSelectorEventLoop uses directly the private _debug attribute

Just try to be consistent: _debug was already used in some places, and always
used in BaseProactorEventLoop.

files:
  Lib/asyncio/selector_events.py |  10 +++++-----
  1 files changed, 5 insertions(+), 5 deletions(-)


diff --git a/Lib/asyncio/selector_events.py b/Lib/asyncio/selector_events.py
--- a/Lib/asyncio/selector_events.py
+++ b/Lib/asyncio/selector_events.py
@@ -214,7 +214,7 @@
 
             # It's now up to the protocol to handle the connection.
         except Exception as exc:
-            if self.get_debug():
+            if self._debug:
                 context = {
                     'message': ('Error on transport creation '
                                 'for incoming connection'),
@@ -312,7 +312,7 @@
 
         This method is a coroutine.
         """
-        if self.get_debug() and sock.gettimeout() != 0:
+        if self._debug and sock.gettimeout() != 0:
             raise ValueError("the socket must be non-blocking")
         fut = futures.Future(loop=self)
         self._sock_recv(fut, False, sock, n)
@@ -350,7 +350,7 @@
 
         This method is a coroutine.
         """
-        if self.get_debug() and sock.gettimeout() != 0:
+        if self._debug and sock.gettimeout() != 0:
             raise ValueError("the socket must be non-blocking")
         fut = futures.Future(loop=self)
         if data:
@@ -393,7 +393,7 @@
 
         This method is a coroutine.
         """
-        if self.get_debug() and sock.gettimeout() != 0:
+        if self._debug and sock.gettimeout() != 0:
             raise ValueError("the socket must be non-blocking")
         fut = futures.Future(loop=self)
         try:
@@ -453,7 +453,7 @@
 
         This method is a coroutine.
         """
-        if self.get_debug() and sock.gettimeout() != 0:
+        if self._debug and sock.gettimeout() != 0:
             raise ValueError("the socket must be non-blocking")
         fut = futures.Future(loop=self)
         self._sock_accept(fut, False, sock)

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Wed Feb  4 14:54:22 2015
From: python-checkins at python.org (victor.stinner)
Date: Wed, 04 Feb 2015 13:54:22 +0000
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbzogT25s?=
 =?utf-8?q?y_call_=5Fcheck=5Fresolved=5Faddress=28=29_in_debug_mode?=
Message-ID: <20150204135422.96080.95954@psf.io>

https://hg.python.org/cpython/rev/8b52ca4d11b8
changeset:   94501:8b52ca4d11b8
branch:      3.4
user:        Victor Stinner <victor.stinner at gmail.com>
date:        Wed Feb 04 14:51:23 2015 +0100
summary:
  asyncio: Only call _check_resolved_address() in debug mode

* _check_resolved_address() is implemented with getaddrinfo() which is slow
* If available, use socket.inet_pton() instead of socket.getaddrinfo(), because
  it is much faster

Microbenchmark (timeit) on Fedora 21 (Python 3.4, Linux 3.17, glibc 2.20) to
validate the IPV4 address "127.0.0.1" or the IPv6 address "::1":

* getaddrinfo() 10.4 usec per loop
* inet_pton(): 0.285 usec per loop

On glibc older than 2.14, getaddrinfo() always requests the list of all local
IP addresses to the kernel (using a NETLINK socket). getaddrinfo() has other
known issues, it's better to avoid it when it is possible.

files:
  Lib/asyncio/base_events.py           |  48 ++++++++++-----
  Lib/asyncio/proactor_events.py       |   3 +-
  Lib/asyncio/selector_events.py       |   3 +-
  Lib/test/test_asyncio/test_events.py |   4 +
  4 files changed, 40 insertions(+), 18 deletions(-)


diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py
--- a/Lib/asyncio/base_events.py
+++ b/Lib/asyncio/base_events.py
@@ -75,7 +75,11 @@
 def _check_resolved_address(sock, address):
     # Ensure that the address is already resolved to avoid the trap of hanging
     # the entire event loop when the address requires doing a DNS lookup.
+    #
+    # getaddrinfo() is slow (around 10 us per call): this function should only
+    # be called in debug mode
     family = sock.family
+
     if family == socket.AF_INET:
         host, port = address
     elif family == socket.AF_INET6:
@@ -83,22 +87,34 @@
     else:
         return
 
-    type_mask = 0
-    if hasattr(socket, 'SOCK_NONBLOCK'):
-        type_mask |= socket.SOCK_NONBLOCK
-    if hasattr(socket, 'SOCK_CLOEXEC'):
-        type_mask |= socket.SOCK_CLOEXEC
-    # Use getaddrinfo(flags=AI_NUMERICHOST) to ensure that the address is
-    # already resolved.
-    try:
-        socket.getaddrinfo(host, port,
-                           family=family,
-                           type=(sock.type & ~type_mask),
-                           proto=sock.proto,
-                           flags=socket.AI_NUMERICHOST)
-    except socket.gaierror as err:
-        raise ValueError("address must be resolved (IP address), got %r: %s"
-                         % (address, err))
+    # On Windows, socket.inet_pton() is only available since Python 3.4
+    if hasattr(socket, 'inet_pton'):
+        # getaddrinfo() is slow and has known issue: prefer inet_pton()
+        # if available
+        try:
+            socket.inet_pton(family, host)
+        except OSError as exc:
+            raise ValueError("address must be resolved (IP address), "
+                             "got host %r: %s"
+                             % (host, exc))
+    else:
+        # Use getaddrinfo(flags=AI_NUMERICHOST) to ensure that the address is
+        # already resolved.
+        type_mask = 0
+        if hasattr(socket, 'SOCK_NONBLOCK'):
+            type_mask |= socket.SOCK_NONBLOCK
+        if hasattr(socket, 'SOCK_CLOEXEC'):
+            type_mask |= socket.SOCK_CLOEXEC
+        try:
+            socket.getaddrinfo(host, port,
+                               family=family,
+                               type=(sock.type & ~type_mask),
+                               proto=sock.proto,
+                               flags=socket.AI_NUMERICHOST)
+        except socket.gaierror as err:
+            raise ValueError("address must be resolved (IP address), "
+                             "got host %r: %s"
+                             % (host, err))
 
 def _raise_stop_error(*args):
     raise _StopError
diff --git a/Lib/asyncio/proactor_events.py b/Lib/asyncio/proactor_events.py
--- a/Lib/asyncio/proactor_events.py
+++ b/Lib/asyncio/proactor_events.py
@@ -437,7 +437,8 @@
 
     def sock_connect(self, sock, address):
         try:
-            base_events._check_resolved_address(sock, address)
+            if self._debug:
+                base_events._check_resolved_address(sock, address)
         except ValueError as err:
             fut = futures.Future(loop=self)
             fut.set_exception(err)
diff --git a/Lib/asyncio/selector_events.py b/Lib/asyncio/selector_events.py
--- a/Lib/asyncio/selector_events.py
+++ b/Lib/asyncio/selector_events.py
@@ -397,7 +397,8 @@
             raise ValueError("the socket must be non-blocking")
         fut = futures.Future(loop=self)
         try:
-            base_events._check_resolved_address(sock, address)
+            if self._debug:
+                base_events._check_resolved_address(sock, address)
         except ValueError as err:
             fut.set_exception(err)
         else:
diff --git a/Lib/test/test_asyncio/test_events.py b/Lib/test/test_asyncio/test_events.py
--- a/Lib/test/test_asyncio/test_events.py
+++ b/Lib/test/test_asyncio/test_events.py
@@ -1437,6 +1437,10 @@
              'selector': self.loop._selector.__class__.__name__})
 
     def test_sock_connect_address(self):
+        # In debug mode, sock_connect() must ensure that the address is already
+        # resolved (call _check_resolved_address())
+        self.loop.set_debug(True)
+
         addresses = [(socket.AF_INET, ('www.python.org', 80))]
         if support.IPV6_ENABLED:
             addresses.extend((

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Wed Feb  4 14:54:24 2015
From: python-checkins at python.org (victor.stinner)
Date: Wed, 04 Feb 2015 13:54:24 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio=29?=
Message-ID: <20150204135422.34410.20500@psf.io>

https://hg.python.org/cpython/rev/cf4894e8f707
changeset:   94502:cf4894e8f707
parent:      94499:9c46707e5526
parent:      94501:8b52ca4d11b8
user:        Victor Stinner <victor.stinner at gmail.com>
date:        Wed Feb 04 14:51:50 2015 +0100
summary:
  Merge 3.4 (asyncio)

files:
  Lib/asyncio/base_events.py           |  48 ++++++++++-----
  Lib/asyncio/proactor_events.py       |   3 +-
  Lib/asyncio/selector_events.py       |  13 ++--
  Lib/test/test_asyncio/test_events.py |   4 +
  4 files changed, 45 insertions(+), 23 deletions(-)


diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py
--- a/Lib/asyncio/base_events.py
+++ b/Lib/asyncio/base_events.py
@@ -75,7 +75,11 @@
 def _check_resolved_address(sock, address):
     # Ensure that the address is already resolved to avoid the trap of hanging
     # the entire event loop when the address requires doing a DNS lookup.
+    #
+    # getaddrinfo() is slow (around 10 us per call): this function should only
+    # be called in debug mode
     family = sock.family
+
     if family == socket.AF_INET:
         host, port = address
     elif family == socket.AF_INET6:
@@ -83,22 +87,34 @@
     else:
         return
 
-    type_mask = 0
-    if hasattr(socket, 'SOCK_NONBLOCK'):
-        type_mask |= socket.SOCK_NONBLOCK
-    if hasattr(socket, 'SOCK_CLOEXEC'):
-        type_mask |= socket.SOCK_CLOEXEC
-    # Use getaddrinfo(flags=AI_NUMERICHOST) to ensure that the address is
-    # already resolved.
-    try:
-        socket.getaddrinfo(host, port,
-                           family=family,
-                           type=(sock.type & ~type_mask),
-                           proto=sock.proto,
-                           flags=socket.AI_NUMERICHOST)
-    except socket.gaierror as err:
-        raise ValueError("address must be resolved (IP address), got %r: %s"
-                         % (address, err))
+    # On Windows, socket.inet_pton() is only available since Python 3.4
+    if hasattr(socket, 'inet_pton'):
+        # getaddrinfo() is slow and has known issue: prefer inet_pton()
+        # if available
+        try:
+            socket.inet_pton(family, host)
+        except OSError as exc:
+            raise ValueError("address must be resolved (IP address), "
+                             "got host %r: %s"
+                             % (host, exc))
+    else:
+        # Use getaddrinfo(flags=AI_NUMERICHOST) to ensure that the address is
+        # already resolved.
+        type_mask = 0
+        if hasattr(socket, 'SOCK_NONBLOCK'):
+            type_mask |= socket.SOCK_NONBLOCK
+        if hasattr(socket, 'SOCK_CLOEXEC'):
+            type_mask |= socket.SOCK_CLOEXEC
+        try:
+            socket.getaddrinfo(host, port,
+                               family=family,
+                               type=(sock.type & ~type_mask),
+                               proto=sock.proto,
+                               flags=socket.AI_NUMERICHOST)
+        except socket.gaierror as err:
+            raise ValueError("address must be resolved (IP address), "
+                             "got host %r: %s"
+                             % (host, err))
 
 def _raise_stop_error(*args):
     raise _StopError
diff --git a/Lib/asyncio/proactor_events.py b/Lib/asyncio/proactor_events.py
--- a/Lib/asyncio/proactor_events.py
+++ b/Lib/asyncio/proactor_events.py
@@ -437,7 +437,8 @@
 
     def sock_connect(self, sock, address):
         try:
-            base_events._check_resolved_address(sock, address)
+            if self._debug:
+                base_events._check_resolved_address(sock, address)
         except ValueError as err:
             fut = futures.Future(loop=self)
             fut.set_exception(err)
diff --git a/Lib/asyncio/selector_events.py b/Lib/asyncio/selector_events.py
--- a/Lib/asyncio/selector_events.py
+++ b/Lib/asyncio/selector_events.py
@@ -214,7 +214,7 @@
 
             # It's now up to the protocol to handle the connection.
         except Exception as exc:
-            if self.get_debug():
+            if self._debug:
                 context = {
                     'message': ('Error on transport creation '
                                 'for incoming connection'),
@@ -312,7 +312,7 @@
 
         This method is a coroutine.
         """
-        if self.get_debug() and sock.gettimeout() != 0:
+        if self._debug and sock.gettimeout() != 0:
             raise ValueError("the socket must be non-blocking")
         fut = futures.Future(loop=self)
         self._sock_recv(fut, False, sock, n)
@@ -350,7 +350,7 @@
 
         This method is a coroutine.
         """
-        if self.get_debug() and sock.gettimeout() != 0:
+        if self._debug and sock.gettimeout() != 0:
             raise ValueError("the socket must be non-blocking")
         fut = futures.Future(loop=self)
         if data:
@@ -393,11 +393,12 @@
 
         This method is a coroutine.
         """
-        if self.get_debug() and sock.gettimeout() != 0:
+        if self._debug and sock.gettimeout() != 0:
             raise ValueError("the socket must be non-blocking")
         fut = futures.Future(loop=self)
         try:
-            base_events._check_resolved_address(sock, address)
+            if self._debug:
+                base_events._check_resolved_address(sock, address)
         except ValueError as err:
             fut.set_exception(err)
         else:
@@ -453,7 +454,7 @@
 
         This method is a coroutine.
         """
-        if self.get_debug() and sock.gettimeout() != 0:
+        if self._debug and sock.gettimeout() != 0:
             raise ValueError("the socket must be non-blocking")
         fut = futures.Future(loop=self)
         self._sock_accept(fut, False, sock)
diff --git a/Lib/test/test_asyncio/test_events.py b/Lib/test/test_asyncio/test_events.py
--- a/Lib/test/test_asyncio/test_events.py
+++ b/Lib/test/test_asyncio/test_events.py
@@ -1437,6 +1437,10 @@
              'selector': self.loop._selector.__class__.__name__})
 
     def test_sock_connect_address(self):
+        # In debug mode, sock_connect() must ensure that the address is already
+        # resolved (call _check_resolved_address())
+        self.loop.set_debug(True)
+
         addresses = [(socket.AF_INET, ('www.python.org', 80))]
         if support.IPV6_ENABLED:
             addresses.extend((

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Wed Feb  4 16:15:14 2015
From: python-checkins at python.org (victor.stinner)
Date: Wed, 04 Feb 2015 15:15:14 +0000
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbyBkb2M6?=
 =?utf-8?q?_explain_how_to_display_ResourceWarning_in_the_debug_mode_secti?=
 =?utf-8?q?on?=
Message-ID: <20150204151505.106346.33250@psf.io>

https://hg.python.org/cpython/rev/8e6c1264e2fe
changeset:   94503:8e6c1264e2fe
branch:      3.4
parent:      94501:8b52ca4d11b8
user:        Victor Stinner <victor.stinner at gmail.com>
date:        Wed Feb 04 16:14:33 2015 +0100
summary:
  asyncio doc: explain how to display ResourceWarning in the debug mode section

files:
  Doc/library/asyncio-dev.rst |  39 +++++++++++++++---------
  1 files changed, 24 insertions(+), 15 deletions(-)


diff --git a/Doc/library/asyncio-dev.rst b/Doc/library/asyncio-dev.rst
--- a/Doc/library/asyncio-dev.rst
+++ b/Doc/library/asyncio-dev.rst
@@ -14,16 +14,22 @@
 Debug mode of asyncio
 ---------------------
 
-To enable the debug mode globally, set the environment variable
-:envvar:`PYTHONASYNCIODEBUG` to ``1``. To see debug traces, set the log level
-of the :ref:`asyncio logger <asyncio-logger>` to :py:data:`logging.DEBUG`.  The
-simplest configuration is::
+The implementation of :mod:`asyncio` module has been written for performances.
+To development with asyncio, it's required to enable the debug checks to ease
+the development of asynchronous code.
 
-   import logging
-   # ...
-   logging.basicConfig(level=logging.DEBUG)
+Setup an application to enable all debug checks:
 
-Examples of effects of the debug mode:
+* Enable the asyncio debug mode globally by setting the environment variable
+  :envvar:`PYTHONASYNCIODEBUG` to ``1``
+* Set the log level of the :ref:`asyncio logger <asyncio-logger>` to
+  :py:data:`logging.DEBUG`. For example, call
+  ``logging.basicConfig(level=logging.DEBUG)`` at startup.
+* Configure the :mod:`warnings` module to display :exc:`ResourceWarning`
+  warnings. For example, use the ``-Wdefault`` command line option of Python to
+  display them.
+
+Examples debug checks:
 
 * Log :ref:`coroutines defined but never "yielded from"
   <asyncio-coroutine-not-scheduled>`
@@ -33,6 +39,8 @@
 * Log callbacks taking more than 100 ms to be executed. The
   :attr:`BaseEventLoop.slow_callback_duration` attribute is the minimum
   duration in seconds of "slow" callbacks.
+* :exc:`ResourceWarning` warnings are emitted when transports and event loops
+  are :ref:`not closed explicitly <asyncio-close-transports>`.
 
 .. seealso::
 
@@ -372,14 +380,15 @@
 
    :ref:`Detect coroutine objects never scheduled <asyncio-coroutine-not-scheduled>`.
 
+.. _asyncio-close-transports:
 
-Close transports
-----------------
+Close transports and event loops
+--------------------------------
 
 When a transport is no more needed, call its ``close()`` method to release
-resources.
+resources. Event loops must also be closed explicitly.
 
-If a transport (or an event loop) is not closed explicitly, a
-:exc:`ResourceWarning` warning will be emitted in its destructor. The
-:exc:`ResourceWarning` warnings are hidden by default: use the ``-Wd`` command
-line option of Python to show them.
+If a transport or an event loop is not closed explicitly, a
+:exc:`ResourceWarning` warning will be emitted in its destructor. By default,
+:exc:`ResourceWarning` warnings are ignored. The :ref:`Debug mode of asyncio
+<asyncio-debug-mode>` section explains how to display them.

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Wed Feb  4 16:15:14 2015
From: python-checkins at python.org (victor.stinner)
Date: Wed, 04 Feb 2015 15:15:14 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio_doc=29?=
Message-ID: <20150204151505.34396.83498@psf.io>

https://hg.python.org/cpython/rev/f0665e8e497d
changeset:   94504:f0665e8e497d
parent:      94502:cf4894e8f707
parent:      94503:8e6c1264e2fe
user:        Victor Stinner <victor.stinner at gmail.com>
date:        Wed Feb 04 16:14:58 2015 +0100
summary:
  Merge 3.4 (asyncio doc)

files:
  Doc/library/asyncio-dev.rst |  39 +++++++++++++++---------
  1 files changed, 24 insertions(+), 15 deletions(-)


diff --git a/Doc/library/asyncio-dev.rst b/Doc/library/asyncio-dev.rst
--- a/Doc/library/asyncio-dev.rst
+++ b/Doc/library/asyncio-dev.rst
@@ -14,16 +14,22 @@
 Debug mode of asyncio
 ---------------------
 
-To enable the debug mode globally, set the environment variable
-:envvar:`PYTHONASYNCIODEBUG` to ``1``. To see debug traces, set the log level
-of the :ref:`asyncio logger <asyncio-logger>` to :py:data:`logging.DEBUG`.  The
-simplest configuration is::
+The implementation of :mod:`asyncio` module has been written for performances.
+To development with asyncio, it's required to enable the debug checks to ease
+the development of asynchronous code.
 
-   import logging
-   # ...
-   logging.basicConfig(level=logging.DEBUG)
+Setup an application to enable all debug checks:
 
-Examples of effects of the debug mode:
+* Enable the asyncio debug mode globally by setting the environment variable
+  :envvar:`PYTHONASYNCIODEBUG` to ``1``
+* Set the log level of the :ref:`asyncio logger <asyncio-logger>` to
+  :py:data:`logging.DEBUG`. For example, call
+  ``logging.basicConfig(level=logging.DEBUG)`` at startup.
+* Configure the :mod:`warnings` module to display :exc:`ResourceWarning`
+  warnings. For example, use the ``-Wdefault`` command line option of Python to
+  display them.
+
+Examples debug checks:
 
 * Log :ref:`coroutines defined but never "yielded from"
   <asyncio-coroutine-not-scheduled>`
@@ -33,6 +39,8 @@
 * Log callbacks taking more than 100 ms to be executed. The
   :attr:`BaseEventLoop.slow_callback_duration` attribute is the minimum
   duration in seconds of "slow" callbacks.
+* :exc:`ResourceWarning` warnings are emitted when transports and event loops
+  are :ref:`not closed explicitly <asyncio-close-transports>`.
 
 .. seealso::
 
@@ -372,14 +380,15 @@
 
    :ref:`Detect coroutine objects never scheduled <asyncio-coroutine-not-scheduled>`.
 
+.. _asyncio-close-transports:
 
-Close transports
-----------------
+Close transports and event loops
+--------------------------------
 
 When a transport is no more needed, call its ``close()`` method to release
-resources.
+resources. Event loops must also be closed explicitly.
 
-If a transport (or an event loop) is not closed explicitly, a
-:exc:`ResourceWarning` warning will be emitted in its destructor. The
-:exc:`ResourceWarning` warnings are hidden by default: use the ``-Wd`` command
-line option of Python to show them.
+If a transport or an event loop is not closed explicitly, a
+:exc:`ResourceWarning` warning will be emitted in its destructor. By default,
+:exc:`ResourceWarning` warnings are ignored. The :ref:`Debug mode of asyncio
+<asyncio-debug-mode>` section explains how to display them.

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Wed Feb  4 17:37:12 2015
From: python-checkins at python.org (raymond.hettinger)
Date: Wed, 04 Feb 2015 16:37:12 +0000
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Minor_code_clean_up=2E?=
Message-ID: <20150204163708.25845.65122@psf.io>

https://hg.python.org/cpython/rev/017e7391ab58
changeset:   94505:017e7391ab58
user:        Raymond Hettinger <python at rcn.com>
date:        Wed Feb 04 08:37:02 2015 -0800
summary:
  Minor code clean up.

files:
  Objects/setobject.c |  4 ++--
  1 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/Objects/setobject.c b/Objects/setobject.c
--- a/Objects/setobject.c
+++ b/Objects/setobject.c
@@ -89,7 +89,7 @@
             freeslot = entry;
 
         if (i + LINEAR_PROBES <= mask) {
-            for (j = 1 ; j <= LINEAR_PROBES ; j++) {
+            for (j = 0 ; j < LINEAR_PROBES ; j++) {
                 entry++;
                 if (entry->key == NULL)
                     goto found_null;
@@ -151,7 +151,7 @@
         if (entry->key == NULL)
             goto found_null;
         if (i + LINEAR_PROBES <= mask) {
-            for (j = 1; j <= LINEAR_PROBES; j++) {
+            for (j = 0; j < LINEAR_PROBES; j++) {
                 entry++;
                 if (entry->key == NULL)
                     goto found_null;

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Thu Feb  5 00:19:09 2015
From: python-checkins at python.org (chris.angelico)
Date: Wed, 04 Feb 2015 23:19:09 +0000
Subject: [Python-checkins] =?utf-8?q?peps=3A_Update_PEP_485_from_Chris_Bar?=
	=?utf-8?q?ker=27s_edits?=
Message-ID: <20150204231853.39274.76411@psf.io>

https://hg.python.org/peps/rev/f77ab680d1e9
changeset:   5692:f77ab680d1e9
user:        Chris Angelico <rosuav at gmail.com>
date:        Thu Feb 05 10:17:42 2015 +1100
summary:
  Update PEP 485 from Chris Barker's edits

files:
  pep-0485.txt |  484 +++++++++++++++++++++++++++++++-------
  1 files changed, 397 insertions(+), 87 deletions(-)


diff --git a/pep-0485.txt b/pep-0485.txt
--- a/pep-0485.txt
+++ b/pep-0485.txt
@@ -15,8 +15,10 @@
 ========
 
 This PEP proposes the addition of a function to the standard library
-that determines whether one value is approximately equal or "close"
-to another value.
+that determines whether one value is approximately equal or "close" to
+another value. It is also proposed that an assertion be added to the
+``unittest.TestCase`` class to provide easy access for those using
+unittest for testing.
 
 
 Rationale
@@ -37,27 +39,35 @@
 Existing Implementations
 ------------------------
 
-The standard library includes the
-``unittest.TestCase.assertAlmostEqual`` method, but it:
+The standard library includes the ``unittest.TestCase.assertAlmostEqual``
+method, but it:
 
 * Is buried in the unittest.TestCase class
 
-* Is an assertion, so you can't use it as a general test (easily)
+* Is an assertion, so you can't use it as a general test at the command
+  line, etc. (easily)
 
-* Uses number of decimal digits or an absolute delta, which are
-  particular use cases that don't provide a general relative error.
+* Is an absolute difference test. Often the measure of difference
+  requires, particularly for floating point numbers, a relative error,
+  i.e "Are these two values within x% of each-other?", rather than an
+  absolute error. Particularly when the magnatude of the values is
+  unknown a priori.
 
-The numpy package has the ``allclose()`` and ``isclose()`` functions.
+The numpy package has the ``allclose()`` and ``isclose()`` functions,
+but they are only available with numpy.
 
 The statistics package tests include an implementation, used for its
 unit tests.
 
 One can also find discussion and sample implementations on Stack
-Overflow, and other help sites.
+Overflow and other help sites.
 
-These existing implementations indicate that this is a common need,
-and not trivial to write oneself, making it a candidate for the
-standard library.
+Many other non-python systems provide such a test, including the Boost C++
+library and the APL language (reference?).
+
+These existing implementations indicate that this is a common need and
+not trivial to write oneself, making it a candidate for the standard
+library.
 
 
 Proposed Implementation
@@ -68,21 +78,22 @@
 
 The new function will have the following signature::
 
-  is_close_to(actual, expected, tol=1e-8, abs_tol=0.0)
+  is_close(a, b, rel_tolerance=1e-9, abs_tolerance=0.0)
 
-``actual``: is the value that has been computed, measured, etc.
+``a`` and ``b``: are the two values to be tested to relative closeness
 
-``expected``: is the "known" value.
+``rel_tolerance``: is the relative tolerance -- it is the amount of
+error allowed, relative to the magnitude a and b. For example, to set
+a tolerance of 5%, pass tol=0.05. The default tolerance is 1e-8, which
+assures that the two values are the same within about 8 decimal
+digits.
 
-``tol``: is the relative tolerance -- it is the amount of error
-allowed, relative to the magnitude of the expected value.
-
-``abs_tol``: is an minimum absolute tolerance level -- useful for
+``abs_tolerance``: is an minimum absolute tolerance level -- useful for
 comparisons near zero.
 
 Modulo error checking, etc, the function will return the result of::
 
-    abs(expected-actual) <= max(tol*expected, abs_tol)
+  abs(a-b) <= max( rel_tolerance * min(abs(a), abs(b), abs_tolerance )
 
 
 Handling of non-finite numbers
@@ -108,7 +119,7 @@
 * ``int``
 
 * ``Fraction``
- 
+
 * ``complex``: for complex, ``abs(z)`` will be used for scaling and
   comparison.
 
@@ -116,35 +127,85 @@
 Behavior near zero
 ------------------
 
-Relative comparison is problematic if either value is zero. In this
-case, the difference is relative to zero, and thus will always be
-smaller than the prescribed tolerance. To handle this case, an
-optional parameter, ``abs_tol`` (default 0.0) can be used to set a
-minimum tolerance to be used in the case of very small relative
-tolerance. That is, the values will be considered close if::
+Relative comparison is problematic if either value is zero. By
+definition, no value is small relative to zero. And computationally,
+if either value is zero, the difference is the absolute value of the
+other value, and the computed absolute tolerance will be rel_tolerance
+times that value. rel-tolerance is always less than one, so the
+difference will never be less than the tolerance.
 
-    abs(a-b) <= abs(tol*expected) or abs(a-b) <= abs_tol
+However, while mathematically correct, there are many use cases where
+a user will need to know if a computed value is "close" to zero. This
+calls for an absolute tolerance test. If the user needs to call this
+function inside a loop or comprehension, where some, but not all, of
+the expected values may be zero, it is important that both a relative
+tolerance and absolute tolerance can be tested for with a single
+function with a single set of parameters.
 
-If the user sets the rel_tol parameter to 0.0, then only the absolute
-tolerance will effect the result, so this function provides an
-absolute tolerance check as well.
+There is a similar issue if the two values to be compared straddle zero:
+if a is approximately equal to -b, then a and b will never be computed
+as "close".
+
+To handle this case, an optional parameter, ``abs_tolerance`` can be
+used to set a minimum tolerance used in the case of very small or zero
+computed absolute tolerance. That is, the values will be always be
+considered close if the difference between them is less than the
+abs_tolerance
+
+The default absolute tolerance value is set to zero because there is
+no value that is appropriate for the general case. It is impossible to
+know an appropriate value without knowing the likely values expected
+for a given use case. If all the values tested are on order of one,
+then a value of about 1e-8 might be appropriate, but that would be far
+too large if expected values are on order of 1e-12 or smaller.
+
+Any non-zero default might result in user's tests passing totally
+inappropriately. If, on the other hand a test against zero fails the
+first time with defaults, a user will be prompted to select an
+appropriate value for the problem at hand in order to get the test to
+pass.
+
+NOTE: that the author of this PEP has resolved to go back over many of
+his tests that use the numpy ``all_close()`` function, which provides
+a default abs_tolerance, and make sure that the default value is
+appropriate.
+
+If the user sets the rel_tolerance parameter to 0.0, then only the
+absolute tolerance will effect the result. While not the goal of the
+function, it does allow it to be used as a purely absolute tolerance
+check as well.
+
+unittest assertion
+-------------------
+
+[need text here]
+
+implementation
+--------------
 
 A sample implementation is available (as of Jan 22, 2015) on gitHub:
 
-https://github.com/PythonCHB/close_pep/blob/master/is_close_to.py
+https://github.com/PythonCHB/close_pep/blob/master
 
+This implementation has a flag that lets the user select which
+relative tolerance test to apply -- this PEP does not suggest that
+that be retained, but rather than the strong test be selected.
 
 Relative Difference
 ===================
 
 There are essentially two ways to think about how close two numbers
-are to each-other: absolute difference: simply ``abs(a-b)``, and
-relative difference: ``abs(a-b)/scale_factor`` [2]_. The absolute
-difference is trivial enough that this proposal focuses on the
-relative difference.
+are to each-other:
+
+Absolute difference: simply ``abs(a-b)``
+
+Relative difference: ``abs(a-b)/scale_factor`` [2]_.
+
+The absolute difference is trivial enough that this proposal focuses
+on the relative difference.
 
 Usually, the scale factor is some function of the values under
-consideration, for instance: 
+consideration, for instance:
 
 1) The absolute value of one of the input values
 
@@ -152,7 +213,106 @@
 
 3) The minimum absolute value of the two.
 
-4) The arithmetic mean of the two
+4) The absolute value of the arithmetic mean of the two
+
+These lead to the following possibilities for determining if two
+values, a and b, are close to each other.
+
+1) ``abs(a-b) <= tol*abs(a)``
+
+2) ``abs(a-b) <= tol * max( abs(a), abs(b) )``
+
+3) ``abs(a-b) <= tol * min( abs(a), abs(b) )``
+
+4) ``abs(a-b) <= tol * (a + b)/2``
+
+NOTE: (2) and (3) can also be written as:
+
+2) ``(abs(a-b) <= tol*abs(a)) or (abs(a-b) <= tol*abs(a))``
+
+3) ``(abs(a-b) <= tol*abs(a)) and (abs(a-b) <= tol*abs(a))``
+
+(Boost refers to these as the "weak" and "strong" formulations [3]_)
+These can be a tiny bit more computationally efficient, and thus are
+used in the example code.
+
+Each of these formulations can lead to slightly different results.
+However, if the tolerance value is small, the differences are quite
+small. In fact, often less than available floating point precision.
+
+How much difference does it make?
+---------------------------------
+
+When selecting a method to determine closeness, one might want to know
+how much  of a difference it could make to use one test or the other
+-- i.e. how many values are there (or what range of values) that will
+pass one test, but not the other.
+
+The largest difference is between options (2) and (3) where the
+allowable absolute difference is scaled by either the larger or
+smaller of the values.
+
+Define ``delta`` to be the difference between the allowable absolute
+tolerance defined by the larger value and that defined by the smaller
+value. That is, the amount that the two input values need to be
+different in order to get a different result from the two tests.
+``tol`` is the relative tolerance value.
+
+Assume that ``a`` is the larger value and that both ``a`` and ``b``
+are positive, to make the analysis a bit easier. ``delta`` is
+therefore::
+
+  delta = tol * (a-b)
+
+
+or::
+
+  delta / tol = (a-b)
+
+
+The largest absolute difference that would pass the test: ``(a-b)``,
+equals the tolerance times the larger value::
+
+  (a-b) = tol * a
+
+
+Substituting into the expression for delta::
+
+  delta / tol = tol * a
+
+
+so::
+
+  delta = tol**2 * a
+
+
+For example, for ``a = 10``, ``b = 9``, ``tol = 0.1`` (10%):
+
+maximum tolerance ``tol * a == 0.1 * 10 == 1.0``
+
+minimum tolerance ``tol * b == 0.1 * 9.0 == 0.9``
+
+delta = ``(1.0 - 0.9) * 0.1 = 0.1`` or  ``tol**2 * a = 0.1**2 * 10 = .01``
+
+The absolute difference between the maximum and minimum tolerance
+tests in this case could be substantial. However, the primary use
+case for the proposed function is testing the results of computations.
+In that case a relative tolerance is likely to be selected of much
+smaller magnitude.
+
+For example, a relative tolerance of ``1e-8`` is about half the
+precision available in a python float. In that case, the difference
+between the two tests is ``1e-8**2 * a`` or ``1e-16 * a``, which is
+close to the limit of precision of a python float. If the relative
+tolerance is set to the proposed default of 1e-9 (or smaller), the
+difference between the two tests will be lost to the limits of
+precision of floating point. That is, each of the four methods will
+yield exactly the same results for all values of a and b.
+
+In addition, in common use, tolerances are defined to 1 significant
+figure -- that is, 1e-8 is specifying about 8 decimal digits of
+accuracy. So the difference between the various possible tests is well
+below the precision to which the tolerance is specified.
 
 
 Symmetry
@@ -161,46 +321,113 @@
 A relative comparison can be either symmetric or non-symmetric. For a
 symmetric algorithm:
 
-``is_close_to(a,b)`` is always equal to ``is_close_to(b,a)``
+``is_close_to(a,b)`` is always the same as ``is_close_to(b,a)``
 
-This is an appealing consistency -- it mirrors the symmetry of
-equality, and is less likely to confuse people. However, often the
-question at hand is:
+If a relative closeness test uses only one of the values (such as (1)
+above), then the result is asymmetric, i.e. is_close_to(a,b) is not
+necessarily the same as is_close_to(b,a).
 
-"Is this computed or measured value within some tolerance of a known
-value?"
+Which approach is most appropriate depends on what question is being
+asked. If the question is: "are these two numbers close to each
+other?", there is no obvious ordering, and a symmetric test is most
+appropriate.
 
-In this case, the user wants the relative tolerance to be specifically
-scaled against the known value. It is also easier for the user to
-reason about.
+However, if the question is: "Is the computed value within x% of this
+known value?", then it is appropriate to scale the tolerance to the
+known value, and an asymmetric test is most appropriate.
 
-This proposal uses this asymmetric test to allow this specific
-definition of relative tolerance.
+From the previous section, it is clear that either approach would
+yield the same or similar results in the common use cases. In that
+case, the goal of this proposal is to provide a function that is least
+likely to produce surprising results.
 
-Example:
+The symmetric approach provide an appealing consistency -- it
+mirrors the symmetry of equality, and is less likely to confuse
+people. A symmetric test also relieves the user of the need to think
+about the order in which to set the arguments.  It was also pointed
+out that there may be some cases where the order of evaluation may not
+be well defined, for instance in the case of comparing a set of values
+all against each other.
 
-For the question: "Is the value of a within 10% of b?", Using b to
-scale the percent error clearly defines the result.
+There may be cases when a user does need to know that a value is
+within a particular range of a known value. In that case, it is easy
+enough to simply write the test directly::
 
-However, as this approach is not symmetric, a may be within 10% of b,
-but b is not within 10% of a. Consider the case::
+  if a-b <= tol*a:
 
-  a =  9.0
-  b = 10.0
+(assuming a > b in this case). There is little need to provide a
+function for this particular case.
 
-The difference between a and b is 1.0. 10% of a is 0.9, so b is not
-within 10% of a. But 10% of b is 1.0, so a is within 10% of b. 
+This proposal uses a symmetric test.
 
-Casual users might reasonably expect that if a is close to b, then b
-would also be close to a. However, in the common cases, the tolerance
-is quite small and often poorly defined, i.e. 1e-8, defined to only
-one significant figure, so the result will be very similar regardless
-of the order of the values. And if the user does care about the
-precise result, s/he can take care to always pass in the two
-parameters in sorted order.
+Which symmetric test?
+---------------------
 
-This proposed implementation uses asymmetric criteria with the scaling
-value clearly identified.
+There are three symmetric tests considered:
+
+The case that uses the arithmetic mean of the two values requires that
+the value be either added together before dividing by 2, which could
+result in extra overflow to inf for very large numbers, or require
+each value to be divided by two before being added together, which
+could result in underflow to -inf for very small numbers. This effect
+would only occur at the very limit of float values, but it was decided
+there as no benefit to the method worth reducing the range of
+functionality.
+
+This leaves the boost "weak" test (2)-- or using the smaller value to
+scale the tolerance, or the Boost "strong" (3) test, which uses the
+smaller of the values to scale the tolerance. For small tolerance,
+they yield the same result, but this proposal uses the boost "strong"
+test case: it is symmetric and provides a slightly stricter criteria
+for tolerance.
+
+
+Defaults
+========
+
+Default values are required for the relative and absolute tolerance.
+
+Relative Tolerance Default
+--------------------------
+
+The relative tolerance required for two values to be considered
+"close" is entirely use-case dependent. Nevertheless, the relative
+tolerance needs to be less than 1.0, and greater than 1e-16
+(approximate precision of a python float). The value of 1e-9 was
+selected because it is the largest relative tolerance for which the
+various possible methods will yield the same result, and it is also
+about half of the precision available to a python float. In the
+general case, a good numerical algorithm is not expected to lose more
+than about half of available digits of accuracy, and if a much larger
+tolerance is acceptable, the user should be considering the proper
+value in that case. Thus 1-e9 is expected to "just work" for many
+cases.
+
+Absolute tolerance default
+--------------------------
+
+The absolute tolerance value will be used primarily for comparing to
+zero. The absolute tolerance required to determine if a value is
+"close" to zero is entirely use-case dependent. There is also
+essentially no bounds to the useful range -- expected values would
+conceivably be anywhere within the limits of a python float.  Thus a
+default of 0.0 is selected.
+
+If, for a given use case, a user needs to compare to zero, the test
+will be guaranteed to fail the first time, and the user can select an
+appropriate value.
+
+It was suggested that comparing to zero is, in fact, a common use case
+(evidence suggest that the numpy functions are often used with zero).
+In this case, it would be desirable to have a "useful" default. Values
+around 1-e8 were suggested, being about half of floating point
+precision for values of around value 1.
+
+However, to quote The Zen: "In the face of ambiguity, refuse the
+temptation to guess." Guessing that users will most often be concerned
+with values close to 1.0 would lead to spurious passing tests when used
+with smaller values -- this is potentially more damaging than
+requiring the user to thoughtfully select an appropriate value.
 
 
 Expected Uses
@@ -208,10 +435,23 @@
 
 The primary expected use case is various forms of testing -- "are the
 results computed near what I expect as a result?" This sort of test
-may or may not be part of a formal unit testing suite.
+may or may not be part of a formal unit testing suite. Such testing
+could be used one-off at the command line, in an iPython notebook,
+part of doctests, or simple assets in an ``if __name__ == "__main__"``
+block.
 
-The function might be used also to determine if a measured value is
-within an expected value.
+The proposed unitest.TestCase assertion would have course be used in
+unit testing.
+
+It would also be an appropriate function to use for the termination
+criteria for a simple iterative solution to an implicit function::
+
+    guess = something
+    while True:
+        new_guess = implicit_function(guess, *args)
+        if is_close(new_guess, guess):
+            break
+        guess = new_guess
 
 
 Inappropriate uses
@@ -238,8 +478,8 @@
 computing the difference, rounding to the given number of decimal
 places (default 7), and comparing to zero.
 
-This method was not selected for this proposal, as the use of decimal
-digits is a specific, not generally useful or flexible test.
+This method is purely an absolute tolerance test, and does not address
+the need for a relative tolerance test.
 
 numpy ``is_close()``
 --------------------
@@ -262,13 +502,16 @@
 In this approach, the absolute and relative tolerance are added
 together, rather than the ``or`` method used in this proposal. This is
 computationally more simple, and if relative tolerance is larger than
-the absolute tolerance, then the addition will have no effect. But if
-the absolute and relative tolerances are of similar magnitude, then
+the absolute tolerance, then the addition will have no effect. However,
+if the absolute and relative tolerances are of similar magnitude, then
 the allowed difference will be about twice as large as expected.
 
-Also, if the value passed in are small compared to the absolute
-tolerance, then the relative tolerance will be completely swamped,
-perhaps unexpectedly.
+This makes the function harder to understand, with no computational
+advantage in this context.
+
+Even more critically, if the values passed in are small compared to
+the absolute  tolerance, then the relative tolerance will be
+completely swamped, perhaps unexpectedly.
 
 This is why, in this proposal, the absolute tolerance defaults to zero
 -- the user will be required to choose a value appropriate for the
@@ -279,25 +522,92 @@
 -------------------------------
 
 The Boost project ( [3]_ ) provides a floating point comparison
-function. Is is a symetric approach, with both "weak" (larger of the
+function. Is is a symmetric approach, with both "weak" (larger of the
 two relative errors) and "strong" (smaller of the two relative errors)
-options.
+options. This proposal uses the Boost "strong" approach. There is no
+need to complicate the API by providing the option to select different
+methods when the results will be similar in most cases, and the user
+is unlikely to know which to select in any case.
 
-It was decided that a method that clearly defined which value was used
-to scale the relative error would be more appropriate for the standard
-library.
+
+Alternate Proposals
+-------------------
+
+
+A Recipe
+'''''''''
+
+The primary alternate proposal was to not provide a standard library
+function at all, but rather, provide a recipe for users to refer to.
+This would have the advantage that the recipe could provide and
+explain the various options, and let the user select that which is
+most appropriate. However, that would require anyone needing such a
+test to, at the very least, copy the function into their code base,
+and select the comparison method to use.
+
+In addition, adding the function to the standard library allows it to
+be used in the ``unittest.TestCase.assertIsClose()`` method, providing
+a substantial convenience to those using unittest.
+
+
+``zero_tol``
+''''''''''''
+
+One possibility was to provide a zero tolerance parameter, rather than
+the absolute tolerance parameter. This would be an absolute tolerance
+that would only be applied in the case of one of the arguments being
+exactly zero. This would have the advantage of retaining the full
+relative tolerance behavior for all non-zero values, while allowing
+tests against zero to work. However, it would also result in the
+potentially surprising result that a small value could be "close" to
+zero, but not "close" to an even smaller value. e.g., 1e-10 is "close"
+to zero, but not "close" to 1e-11.
+
+
+No absolute tolerance
+'''''''''''''''''''''
+
+Given the issues with comparing to zero, another possibility would
+have been to only provide a relative tolerance, and let every
+comparison to zero fail. In this case, the user would need to do a
+simple absolute test: `abs(val) < zero_tol` in the case where the
+comparison involved zero.
+
+However, this would not allow the same call to be used for a sequence
+of values, such as in a loop or comprehension, or in the
+``TestCase.assertClose()`` method. Making the function far less
+useful. It is noted that the default abs_tolerance=0.0 achieves the
+same effect if the default is not overidden.
+
+Other tests
+''''''''''''
+
+The other tests considered are all discussed in the Relative Error
+section above.
+
 
 References
 ==========
 
-.. [1] Python-ideas list discussion thread
-   (https://mail.python.org/pipermail/python-ideas/2015-January/030947.html)
+.. [1] Python-ideas list discussion threads
 
-.. [2] Wikipedaia page on relative difference
-   (http://en.wikipedia.org/wiki/Relative_change_and_difference)
+   https://mail.python.org/pipermail/python-ideas/2015-January/030947.html
+
+   https://mail.python.org/pipermail/python-ideas/2015-January/031124.html
+
+   https://mail.python.org/pipermail/python-ideas/2015-January/031313.html
+
+.. [2] Wikipedia page on relative difference
+
+   http://en.wikipedia.org/wiki/Relative_change_and_difference
 
 .. [3] Boost project floating-point comparison algorithms
-   (http://www.boost.org/doc/libs/1_35_0/libs/test/doc/components/test_tools/floating_point_comparison.html)
+
+   http://www.boost.org/doc/libs/1_35_0/libs/test/doc/components/test_tools/floating_point_comparison.html
+
+.. Bruce Dawson's discussion of floating point.
+
+   https://randomascii.wordpress.com/2012/02/25/comparing-floating-point-numbers-2012-edition/
 
 
 Copyright

-- 
Repository URL: https://hg.python.org/peps

From python-checkins at python.org  Thu Feb  5 02:16:52 2015
From: python-checkins at python.org (gregory.p.smith)
Date: Thu, 05 Feb 2015 01:16:52 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Make_the_stdli?=
 =?utf-8?q?b_test_suite_helper_test=2Escript=5Fhelper=2E=5Fassert=5Fpython?=
 =?utf-8?q?_no_longer?=
Message-ID: <20150205011649.39278.82682@psf.io>

https://hg.python.org/cpython/rev/8d388f6c2ac2
changeset:   94506:8d388f6c2ac2
branch:      3.4
parent:      94503:8e6c1264e2fe
user:        Gregory P. Smith <greg at krypto.org>
date:        Wed Feb 04 17:10:19 2015 -0800
summary:
  Make the stdlib test suite helper test.script_helper._assert_python no longer
pass -I or -E to the child process by default when the environment is required
for the child process interpreter to function properly.

files:
  Lib/test/script_helper.py      |   5 ++-
  Lib/test/test_script_helper.py |  33 ++++++++++++++++++++++
  2 files changed, 36 insertions(+), 2 deletions(-)


diff --git a/Lib/test/script_helper.py b/Lib/test/script_helper.py
--- a/Lib/test/script_helper.py
+++ b/Lib/test/script_helper.py
@@ -52,16 +52,17 @@
 
 # Executing the interpreter in a subprocess
 def _assert_python(expected_success, *args, **env_vars):
+    env_required = _interpreter_requires_environment()
     if '__isolated' in env_vars:
         isolated = env_vars.pop('__isolated')
     else:
-        isolated = not env_vars
+        isolated = not env_vars and not env_required
     cmd_line = [sys.executable, '-X', 'faulthandler']
     if isolated:
         # isolated mode: ignore Python environment variables, ignore user
         # site-packages, and don't add the current directory to sys.path
         cmd_line.append('-I')
-    elif not env_vars:
+    elif not env_vars and not env_required:
         # ignore Python environment variables
         cmd_line.append('-E')
     # Need to preserve the original environment, for in-place testing of
diff --git a/Lib/test/test_script_helper.py b/Lib/test/test_script_helper.py
--- a/Lib/test/test_script_helper.py
+++ b/Lib/test/test_script_helper.py
@@ -33,6 +33,39 @@
         self.assertIn('import sys; sys.exit(0)', error_msg,
                       msg='unexpected command line.')
 
+    @mock.patch('subprocess.Popen')
+    def test_assert_python_isolated_when_env_not_required(self, mock_popen):
+        with mock.patch.object(script_helper,
+                               '_interpreter_requires_environment',
+                               return_value=False) as mock_ire_func:
+            mock_popen.side_effect = RuntimeError('bail out of unittest')
+            try:
+                script_helper._assert_python(True, '-c', 'None')
+            except RuntimeError as err:
+                self.assertEqual('bail out of unittest', err.args[0])
+            self.assertEqual(1, mock_popen.call_count)
+            self.assertEqual(1, mock_ire_func.call_count)
+            popen_command = mock_popen.call_args[0][0]
+            self.assertEqual(sys.executable, popen_command[0])
+            self.assertIn('None', popen_command)
+            self.assertIn('-I', popen_command)
+            self.assertNotIn('-E', popen_command)  # -I overrides this
+
+    @mock.patch('subprocess.Popen')
+    def test_assert_python_not_isolated_when_env_is_required(self, mock_popen):
+        """Ensure that -I is not passed when the environment is required."""
+        with mock.patch.object(script_helper,
+                               '_interpreter_requires_environment',
+                               return_value=True) as mock_ire_func:
+            mock_popen.side_effect = RuntimeError('bail out of unittest')
+            try:
+                script_helper._assert_python(True, '-c', 'None')
+            except RuntimeError as err:
+                self.assertEqual('bail out of unittest', err.args[0])
+            popen_command = mock_popen.call_args[0][0]
+            self.assertNotIn('-I', popen_command)
+            self.assertNotIn('-E', popen_command)
+
 
 class TestScriptHelperEnvironment(unittest.TestCase):
     """Code coverage for _interpreter_requires_environment()."""

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Thu Feb  5 02:16:52 2015
From: python-checkins at python.org (gregory.p.smith)
Date: Thu, 05 Feb 2015 01:16:52 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?q?=29=3A_Make_the_stdlib_test_suite_helper_test=2Escript=5Fhelper?=
 =?utf-8?q?=2E=5Fassert=5Fpython_no_longer?=
Message-ID: <20150205011650.39292.28635@psf.io>

https://hg.python.org/cpython/rev/7dce0cbea210
changeset:   94507:7dce0cbea210
parent:      94505:017e7391ab58
parent:      94506:8d388f6c2ac2
user:        Gregory P. Smith <greg at krypto.org>
date:        Wed Feb 04 17:16:30 2015 -0800
summary:
  Make the stdlib test suite helper test.script_helper._assert_python no longer
pass -I or -E to the child process by default when the environment is required
for the child process interpreter to function properly.

files:
  Lib/test/script_helper.py      |   5 ++-
  Lib/test/test_script_helper.py |  33 ++++++++++++++++++++++
  2 files changed, 36 insertions(+), 2 deletions(-)


diff --git a/Lib/test/script_helper.py b/Lib/test/script_helper.py
--- a/Lib/test/script_helper.py
+++ b/Lib/test/script_helper.py
@@ -52,16 +52,17 @@
 
 # Executing the interpreter in a subprocess
 def _assert_python(expected_success, *args, **env_vars):
+    env_required = interpreter_requires_environment()
     if '__isolated' in env_vars:
         isolated = env_vars.pop('__isolated')
     else:
-        isolated = not env_vars
+        isolated = not env_vars and not env_required
     cmd_line = [sys.executable, '-X', 'faulthandler']
     if isolated:
         # isolated mode: ignore Python environment variables, ignore user
         # site-packages, and don't add the current directory to sys.path
         cmd_line.append('-I')
-    elif not env_vars:
+    elif not env_vars and not env_required:
         # ignore Python environment variables
         cmd_line.append('-E')
     # Need to preserve the original environment, for in-place testing of
diff --git a/Lib/test/test_script_helper.py b/Lib/test/test_script_helper.py
--- a/Lib/test/test_script_helper.py
+++ b/Lib/test/test_script_helper.py
@@ -33,6 +33,39 @@
         self.assertIn('import sys; sys.exit(0)', error_msg,
                       msg='unexpected command line.')
 
+    @mock.patch('subprocess.Popen')
+    def test_assert_python_isolated_when_env_not_required(self, mock_popen):
+        with mock.patch.object(script_helper,
+                               'interpreter_requires_environment',
+                               return_value=False) as mock_ire_func:
+            mock_popen.side_effect = RuntimeError('bail out of unittest')
+            try:
+                script_helper._assert_python(True, '-c', 'None')
+            except RuntimeError as err:
+                self.assertEqual('bail out of unittest', err.args[0])
+            self.assertEqual(1, mock_popen.call_count)
+            self.assertEqual(1, mock_ire_func.call_count)
+            popen_command = mock_popen.call_args[0][0]
+            self.assertEqual(sys.executable, popen_command[0])
+            self.assertIn('None', popen_command)
+            self.assertIn('-I', popen_command)
+            self.assertNotIn('-E', popen_command)  # -I overrides this
+
+    @mock.patch('subprocess.Popen')
+    def test_assert_python_not_isolated_when_env_is_required(self, mock_popen):
+        """Ensure that -I is not passed when the environment is required."""
+        with mock.patch.object(script_helper,
+                               'interpreter_requires_environment',
+                               return_value=True) as mock_ire_func:
+            mock_popen.side_effect = RuntimeError('bail out of unittest')
+            try:
+                script_helper._assert_python(True, '-c', 'None')
+            except RuntimeError as err:
+                self.assertEqual('bail out of unittest', err.args[0])
+            popen_command = mock_popen.call_args[0][0]
+            self.assertNotIn('-I', popen_command)
+            self.assertNotIn('-E', popen_command)
+
 
 class TestScriptHelperEnvironment(unittest.TestCase):
     """Code coverage for interpreter_requires_environment()."""

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Thu Feb  5 04:00:31 2015
From: python-checkins at python.org (donald.stufft)
Date: Thu, 05 Feb 2015 03:00:31 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Update_pip_to_?=
	=?utf-8?b?Ni4wLjg=?=
Message-ID: <20150205030024.34400.90033@psf.io>

https://hg.python.org/cpython/rev/625798e94b14
changeset:   94508:625798e94b14
branch:      3.4
parent:      94506:8d388f6c2ac2
user:        Donald Stufft <donald at stufft.io>
date:        Wed Feb 04 22:00:17 2015 -0500
summary:
  Update pip to 6.0.8

files:
  Lib/ensurepip/__init__.py                             |    2 +-
  Lib/ensurepip/_bundled/pip-6.0.7-py2.py3-none-any.whl |  Bin 
  Lib/ensurepip/_bundled/pip-6.0.8-py2.py3-none-any.whl |  Bin 
  3 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Lib/ensurepip/__init__.py b/Lib/ensurepip/__init__.py
--- a/Lib/ensurepip/__init__.py
+++ b/Lib/ensurepip/__init__.py
@@ -10,7 +10,7 @@
 
 _SETUPTOOLS_VERSION = "12.0.5"
 
-_PIP_VERSION = "6.0.7"
+_PIP_VERSION = "6.0.8"
 
 # pip currently requires ssl support, so we try to provide a nicer
 # error message when that is missing (http://bugs.python.org/issue19744)
diff --git a/Lib/ensurepip/_bundled/pip-6.0.7-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/pip-6.0.7-py2.py3-none-any.whl
deleted file mode 100644
index e01e8c4f44fbf99d9cb286f5a06b6689ff29d4d6..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
GIT binary patch
[stripped]
diff --git a/Lib/ensurepip/_bundled/pip-6.0.8-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/pip-6.0.8-py2.py3-none-any.whl
new file mode 100644
index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..be9238d94c10081b4450aad01851f4c4eda1897c
GIT binary patch
[stripped]

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Thu Feb  5 04:01:10 2015
From: python-checkins at python.org (donald.stufft)
Date: Thu, 05 Feb 2015 03:01:10 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?q?=29=3A_Merge=3A_Update_pip_to_6=2E0=2E8?=
Message-ID: <20150205030103.39294.57209@psf.io>

https://hg.python.org/cpython/rev/a7ddddfe9e36
changeset:   94509:a7ddddfe9e36
parent:      94507:7dce0cbea210
parent:      94508:625798e94b14
user:        Donald Stufft <donald at stufft.io>
date:        Wed Feb 04 22:00:56 2015 -0500
summary:
  Merge: Update pip to 6.0.8

files:
  Lib/ensurepip/__init__.py                             |    2 +-
  Lib/ensurepip/_bundled/pip-6.0.7-py2.py3-none-any.whl |  Bin 
  Lib/ensurepip/_bundled/pip-6.0.8-py2.py3-none-any.whl |  Bin 
  3 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Lib/ensurepip/__init__.py b/Lib/ensurepip/__init__.py
--- a/Lib/ensurepip/__init__.py
+++ b/Lib/ensurepip/__init__.py
@@ -10,7 +10,7 @@
 
 _SETUPTOOLS_VERSION = "12.0.5"
 
-_PIP_VERSION = "6.0.7"
+_PIP_VERSION = "6.0.8"
 
 # pip currently requires ssl support, so we try to provide a nicer
 # error message when that is missing (http://bugs.python.org/issue19744)
diff --git a/Lib/ensurepip/_bundled/pip-6.0.7-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/pip-6.0.7-py2.py3-none-any.whl
deleted file mode 100644
index e01e8c4f44fbf99d9cb286f5a06b6689ff29d4d6..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
GIT binary patch
[stripped]
diff --git a/Lib/ensurepip/_bundled/pip-6.0.8-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/pip-6.0.8-py2.py3-none-any.whl
new file mode 100644
index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..be9238d94c10081b4450aad01851f4c4eda1897c
GIT binary patch
[stripped]

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Thu Feb  5 04:02:28 2015
From: python-checkins at python.org (donald.stufft)
Date: Thu, 05 Feb 2015 03:02:28 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Update_pip_to_?=
	=?utf-8?b?Ni4wLjg=?=
Message-ID: <20150205030217.39286.44606@psf.io>

https://hg.python.org/cpython/rev/b658df02efa3
changeset:   94510:b658df02efa3
branch:      2.7
parent:      94495:887526ebb013
user:        Donald Stufft <donald at stufft.io>
date:        Wed Feb 04 22:02:09 2015 -0500
summary:
  Update pip to 6.0.8

files:
  Lib/ensurepip/__init__.py                             |    2 +-
  Lib/ensurepip/_bundled/pip-6.0.7-py2.py3-none-any.whl |  Bin 
  Lib/ensurepip/_bundled/pip-6.0.8-py2.py3-none-any.whl |  Bin 
  3 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Lib/ensurepip/__init__.py b/Lib/ensurepip/__init__.py
--- a/Lib/ensurepip/__init__.py
+++ b/Lib/ensurepip/__init__.py
@@ -14,7 +14,7 @@
 
 _SETUPTOOLS_VERSION = "12.0.5"
 
-_PIP_VERSION = "6.0.7"
+_PIP_VERSION = "6.0.8"
 
 # pip currently requires ssl support, so we try to provide a nicer
 # error message when that is missing (http://bugs.python.org/issue19744)
diff --git a/Lib/ensurepip/_bundled/pip-6.0.7-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/pip-6.0.7-py2.py3-none-any.whl
deleted file mode 100644
index e01e8c4f44fbf99d9cb286f5a06b6689ff29d4d6..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
GIT binary patch
[stripped]
diff --git a/Lib/ensurepip/_bundled/pip-6.0.8-py2.py3-none-any.whl b/Lib/ensurepip/_bundled/pip-6.0.8-py2.py3-none-any.whl
new file mode 100644
index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..be9238d94c10081b4450aad01851f4c4eda1897c
GIT binary patch
[stripped]

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Thu Feb  5 04:07:09 2015
From: python-checkins at python.org (benjamin.peterson)
Date: Thu, 05 Feb 2015 03:07:09 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_remove_parenth?=
 =?utf-8?q?esis_from_print_statement_=28closes_=2323396=29?=
Message-ID: <20150205030700.106285.25687@psf.io>

https://hg.python.org/cpython/rev/2bb5fa752bfc
changeset:   94511:2bb5fa752bfc
branch:      2.7
user:        Benjamin Peterson <benjamin at python.org>
date:        Wed Feb 04 22:06:55 2015 -0500
summary:
  remove parenthesis from print statement (closes #23396)

files:
  Doc/tutorial/introduction.rst |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Doc/tutorial/introduction.rst b/Doc/tutorial/introduction.rst
--- a/Doc/tutorial/introduction.rst
+++ b/Doc/tutorial/introduction.rst
@@ -170,7 +170,7 @@
    >>> print '"Isn\'t," she said.'
    "Isn't," she said.
    >>> s = 'First line.\nSecond line.'  # \n means newline
-   >>> s  # without print(), \n is included in the output
+   >>> s  # without print, \n is included in the output
    'First line.\nSecond line.'
    >>> print s  # with print, \n produces a new line
    First line.

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Thu Feb  5 07:25:47 2015
From: python-checkins at python.org (ned.deily)
Date: Thu, 05 Feb 2015 06:25:47 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?q?=29=3A_Issue_=2323345=3A_merge_from_3=2E4?=
Message-ID: <20150205062539.96096.73252@psf.io>

https://hg.python.org/cpython/rev/62b322b82f00
changeset:   94514:62b322b82f00
parent:      94509:a7ddddfe9e36
parent:      94513:52932cd7f003
user:        Ned Deily <nad at acm.org>
date:        Thu Feb 05 17:24:00 2015 +1100
summary:
  Issue #23345: merge from 3.4

files:
  Lib/test/test_ssl.py |  2 +-
  Misc/NEWS            |  3 +++
  2 files changed, 4 insertions(+), 1 deletions(-)


diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py
--- a/Lib/test/test_ssl.py
+++ b/Lib/test/test_ssl.py
@@ -306,7 +306,7 @@
         self.assertGreaterEqual(fix, 0)
         self.assertLess(fix, 256)
         self.assertGreaterEqual(patch, 0)
-        self.assertLessEqual(patch, 26)
+        self.assertLessEqual(patch, 63)
         self.assertGreaterEqual(status, 0)
         self.assertLessEqual(status, 15)
         # Version string as returned by {Open,Libre}SSL, the format might change
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -1727,6 +1727,9 @@
 
 - Issue #23211: Workaround test_logging failure on some OS X 10.6 systems.
 
+- Issue #23345: Prevent test_ssl failures with large OpenSSL patch level
+  values (like 0.9.8zc).
+
 Tools/Demos
 -----------
 

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Thu Feb  5 07:25:47 2015
From: python-checkins at python.org (ned.deily)
Date: Thu, 05 Feb 2015 06:25:47 +0000
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIzMzQ1?=
 =?utf-8?q?=3A_Prevent_test=5Fssl_failures_with_large_OpenSSL_patch_level?=
Message-ID: <20150205062539.106389.62903@psf.io>

https://hg.python.org/cpython/rev/52932cd7f003
changeset:   94513:52932cd7f003
branch:      3.4
parent:      94508:625798e94b14
user:        Ned Deily <nad at acm.org>
date:        Thu Feb 05 17:20:13 2015 +1100
summary:
  Issue #23345: Prevent test_ssl failures with large OpenSSL patch level
values (like 0.9.8zc).

files:
  Lib/test/test_ssl.py |  2 +-
  Misc/NEWS            |  3 +++
  2 files changed, 4 insertions(+), 1 deletions(-)


diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py
--- a/Lib/test/test_ssl.py
+++ b/Lib/test/test_ssl.py
@@ -292,7 +292,7 @@
         self.assertGreaterEqual(fix, 0)
         self.assertLess(fix, 256)
         self.assertGreaterEqual(patch, 0)
-        self.assertLessEqual(patch, 26)
+        self.assertLessEqual(patch, 63)
         self.assertGreaterEqual(status, 0)
         self.assertLessEqual(status, 15)
         # Version string as returned by {Open,Libre}SSL, the format might change
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -340,6 +340,9 @@
 
 - Issue #23211: Workaround test_logging failure on some OS X 10.6 systems.
 
+- Issue #23345: Prevent test_ssl failures with large OpenSSL patch level
+  values (like 0.9.8zc).
+
 Build
 -----
 

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Thu Feb  5 07:25:47 2015
From: python-checkins at python.org (ned.deily)
Date: Thu, 05 Feb 2015 06:25:47 +0000
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzIzMzQ1?=
 =?utf-8?q?=3A_Prevent_test=5Fssl_failures_with_large_OpenSSL_patch_level?=
Message-ID: <20150205062539.25841.70311@psf.io>

https://hg.python.org/cpython/rev/49f07942fbd7
changeset:   94512:49f07942fbd7
branch:      2.7
user:        Ned Deily <nad at acm.org>
date:        Thu Feb 05 17:19:11 2015 +1100
summary:
  Issue #23345: Prevent test_ssl failures with large OpenSSL patch level
values (like 0.9.8zc).

files:
  Lib/test/test_ssl.py |  2 +-
  Misc/NEWS            |  3 +++
  2 files changed, 4 insertions(+), 1 deletions(-)


diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py
--- a/Lib/test/test_ssl.py
+++ b/Lib/test/test_ssl.py
@@ -275,7 +275,7 @@
         self.assertGreaterEqual(fix, 0)
         self.assertLess(fix, 256)
         self.assertGreaterEqual(patch, 0)
-        self.assertLessEqual(patch, 26)
+        self.assertLessEqual(patch, 63)
         self.assertGreaterEqual(status, 0)
         self.assertLessEqual(status, 15)
         # Version string as returned by {Open,Libre}SSL, the format might change
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -92,6 +92,9 @@
 - Issue #18905: "pydoc -p 0" now outputs actually used port.  Based on patch by
   Wieland Hoffmann.
 
+- Issue #23345: Prevent test_ssl failures with large OpenSSL patch level
+  values (like 0.9.8zc).
+
 Tests
 -----
 

-- 
Repository URL: https://hg.python.org/cpython

From solipsis at pitrou.net  Thu Feb  5 09:24:14 2015
From: solipsis at pitrou.net (solipsis at pitrou.net)
Date: Thu, 05 Feb 2015 09:24:14 +0100
Subject: [Python-checkins] Daily reference leaks (7dce0cbea210): sum=3
Message-ID: <E1YJHjS-0003rY-4T@ssl.pitrou.net>

results for 7dce0cbea210 on branch "default"
--------------------------------------------

test_functools leaked [0, 0, 3] memory blocks, sum=3


Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/antoine/cpython/refleaks/reflogOdmBU0', '-x']

From python-checkins at python.org  Thu Feb  5 11:47:34 2015
From: python-checkins at python.org (victor.stinner)
Date: Thu, 05 Feb 2015 10:47:34 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?q?=29=3A_Merge_3=2E4_=28asyncio=29?=
Message-ID: <20150205104720.34392.3245@psf.io>

https://hg.python.org/cpython/rev/8ec13c088465
changeset:   94516:8ec13c088465
parent:      94514:62b322b82f00
parent:      94515:2e6972f1104d
user:        Victor Stinner <victor.stinner at gmail.com>
date:        Thu Feb 05 11:46:45 2015 +0100
summary:
  Merge 3.4 (asyncio)

files:
  Lib/asyncio/base_events.py |  12 ++++++------
  1 files changed, 6 insertions(+), 6 deletions(-)


diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py
--- a/Lib/asyncio/base_events.py
+++ b/Lib/asyncio/base_events.py
@@ -188,7 +188,7 @@
         self._internal_fds = 0
         # Identifier of the thread running the event loop, or None if the
         # event loop is not running
-        self._owner = None
+        self._thread_id = None
         self._clock_resolution = time.get_clock_info('monotonic').resolution
         self._exception_handler = None
         self._debug = (not sys.flags.ignore_environment
@@ -269,7 +269,7 @@
         self._check_closed()
         if self.is_running():
             raise RuntimeError('Event loop is running.')
-        self._owner = threading.get_ident()
+        self._thread_id = threading.get_ident()
         try:
             while True:
                 try:
@@ -277,7 +277,7 @@
                 except _StopError:
                     break
         finally:
-            self._owner = None
+            self._thread_id = None
 
     def run_until_complete(self, future):
         """Run until the Future is done.
@@ -362,7 +362,7 @@
 
     def is_running(self):
         """Returns True if the event loop is running."""
-        return (self._owner is not None)
+        return (self._thread_id is not None)
 
     def time(self):
         """Return the time according to the event loop's clock.
@@ -449,10 +449,10 @@
         Should only be called when (self._debug == True).  The caller is
         responsible for checking this condition for performance reasons.
         """
-        if self._owner is None:
+        if self._thread_id is None:
             return
         thread_id = threading.get_ident()
-        if thread_id != self._owner:
+        if thread_id != self._thread_id:
             raise RuntimeError(
                 "Non-thread-safe operation invoked on an event loop other "
                 "than the current one")

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Thu Feb  5 11:47:34 2015
From: python-checkins at python.org (victor.stinner)
Date: Thu, 05 Feb 2015 10:47:34 +0000
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogYXN5bmNpbzogQmFz?=
 =?utf-8?q?eEventLoop=3A_rename_=5Fowner_to_=5Fthread=5Fid?=
Message-ID: <20150205104720.25857.15593@psf.io>

https://hg.python.org/cpython/rev/2e6972f1104d
changeset:   94515:2e6972f1104d
branch:      3.4
parent:      94513:52932cd7f003
user:        Victor Stinner <victor.stinner at gmail.com>
date:        Thu Feb 05 11:45:33 2015 +0100
summary:
  asyncio: BaseEventLoop: rename _owner to _thread_id

files:
  Lib/asyncio/base_events.py |  12 ++++++------
  1 files changed, 6 insertions(+), 6 deletions(-)


diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py
--- a/Lib/asyncio/base_events.py
+++ b/Lib/asyncio/base_events.py
@@ -188,7 +188,7 @@
         self._internal_fds = 0
         # Identifier of the thread running the event loop, or None if the
         # event loop is not running
-        self._owner = None
+        self._thread_id = None
         self._clock_resolution = time.get_clock_info('monotonic').resolution
         self._exception_handler = None
         self._debug = (not sys.flags.ignore_environment
@@ -269,7 +269,7 @@
         self._check_closed()
         if self.is_running():
             raise RuntimeError('Event loop is running.')
-        self._owner = threading.get_ident()
+        self._thread_id = threading.get_ident()
         try:
             while True:
                 try:
@@ -277,7 +277,7 @@
                 except _StopError:
                     break
         finally:
-            self._owner = None
+            self._thread_id = None
 
     def run_until_complete(self, future):
         """Run until the Future is done.
@@ -362,7 +362,7 @@
 
     def is_running(self):
         """Returns True if the event loop is running."""
-        return (self._owner is not None)
+        return (self._thread_id is not None)
 
     def time(self):
         """Return the time according to the event loop's clock.
@@ -449,10 +449,10 @@
         Should only be called when (self._debug == True).  The caller is
         responsible for checking this condition for performance reasons.
         """
-        if self._owner is None:
+        if self._thread_id is None:
             return
         thread_id = threading.get_ident()
-        if thread_id != self._owner:
+        if thread_id != self._thread_id:
             raise RuntimeError(
                 "Non-thread-safe operation invoked on an event loop other "
                 "than the current one")

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Thu Feb  5 14:19:44 2015
From: python-checkins at python.org (serhiy.storchaka)
Date: Thu, 05 Feb 2015 13:19:44 +0000
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE4OTgy?=
 =?utf-8?q?=3A_Add_tests_for_CLI_of_the_calendar_module=2E?=
Message-ID: <20150205131943.34410.49503@psf.io>

https://hg.python.org/cpython/rev/50a5c262b2e6
changeset:   94519:50a5c262b2e6
branch:      2.7
parent:      94512:49f07942fbd7
user:        Serhiy Storchaka <storchaka at gmail.com>
date:        Thu Feb 05 15:18:26 2015 +0200
summary:
  Issue #18982: Add tests for CLI of the calendar module.

files:
  Lib/test/test_calendar.py |  135 +++++++++++++++++++++++++-
  Misc/NEWS                 |    2 +
  2 files changed, 135 insertions(+), 2 deletions(-)


diff --git a/Lib/test/test_calendar.py b/Lib/test/test_calendar.py
--- a/Lib/test/test_calendar.py
+++ b/Lib/test/test_calendar.py
@@ -2,11 +2,22 @@
 import unittest
 
 from test import test_support
+from test.script_helper import assert_python_ok, assert_python_failure
 import locale
 import datetime
+import os
 
+result_2004_01_text = """\
+    January 2004
+Mo Tu We Th Fr Sa Su
+          1  2  3  4
+ 5  6  7  8  9 10 11
+12 13 14 15 16 17 18
+19 20 21 22 23 24 25
+26 27 28 29 30 31
+"""
 
-result_2004_text = """
+result_2004_text = """\
                                   2004
 
       January                   February                   March
@@ -44,7 +55,7 @@
 25 26 27 28 29 30 31      29 30                     27 28 29 30 31
 """
 
-result_2004_html = """
+result_2004_html = """\
 <?xml version="1.0" encoding="ascii"?>
 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
 <html>
@@ -460,6 +471,124 @@
         self.assertEqual(calendar.leapdays(1997,2020), 5)
 
 
+class CommandLineTestCase(unittest.TestCase):
+    def run_ok(self, *args):
+        return assert_python_ok('-m', 'calendar', *args)[1]
+
+    def assertFailure(self, *args):
+        rc, stdout, stderr = assert_python_failure('-m', 'calendar', *args)
+        self.assertIn(b'Usage:', stderr)
+        self.assertEqual(rc, 2)
+
+    def test_help(self):
+        stdout = self.run_ok('-h')
+        self.assertIn(b'Usage:', stdout)
+        self.assertIn(b'calendar.py', stdout)
+        self.assertIn(b'--help', stdout)
+
+    def test_illegal_arguments(self):
+        self.assertFailure('-z')
+        #self.assertFailure('spam')
+        #self.assertFailure('2004', 'spam')
+        self.assertFailure('-t', 'html', '2004', '1')
+
+    def test_output_current_year(self):
+        stdout = self.run_ok()
+        year = datetime.datetime.now().year
+        self.assertIn((' %s' % year).encode(), stdout)
+        self.assertIn(b'January', stdout)
+        self.assertIn(b'Mo Tu We Th Fr Sa Su', stdout)
+
+    def test_output_year(self):
+        stdout = self.run_ok('2004')
+        self.assertEqual(stdout.strip(), result_2004_text.strip())
+
+    def test_output_month(self):
+        stdout = self.run_ok('2004', '1')
+        self.assertEqual(stdout.strip(), result_2004_01_text.strip())
+
+    def test_option_encoding(self):
+        self.assertFailure('-e')
+        self.assertFailure('--encoding')
+        stdout = self.run_ok('--encoding', 'rot-13', '2004')
+        self.assertEqual(stdout.strip(), result_2004_text.encode('rot-13').strip())
+
+    def test_option_locale(self):
+        self.assertFailure('-L')
+        self.assertFailure('--locale')
+        self.assertFailure('-L', 'en')
+        lang, enc = locale.getdefaultlocale()
+        lang = lang or 'C'
+        enc = enc or 'UTF-8'
+        try:
+            oldlocale = locale.getlocale(locale.LC_TIME)
+            try:
+                locale.setlocale(locale.LC_TIME, (lang, enc))
+            finally:
+                locale.setlocale(locale.LC_TIME, oldlocale)
+        except (locale.Error, ValueError):
+            self.skipTest('cannot set the system default locale')
+        stdout = self.run_ok('--locale', lang, '--encoding', enc, '2004')
+        self.assertIn('2004'.encode(enc), stdout)
+
+    def test_option_width(self):
+        self.assertFailure('-w')
+        self.assertFailure('--width')
+        self.assertFailure('-w', 'spam')
+        stdout = self.run_ok('--width', '3', '2004')
+        self.assertIn(b'Mon Tue Wed Thu Fri Sat Sun', stdout)
+
+    def test_option_lines(self):
+        self.assertFailure('-l')
+        self.assertFailure('--lines')
+        self.assertFailure('-l', 'spam')
+        stdout = self.run_ok('--lines', '2', '2004')
+        self.assertIn('December\n\nMo Tu We', stdout)
+
+    def test_option_spacing(self):
+        self.assertFailure('-s')
+        self.assertFailure('--spacing')
+        self.assertFailure('-s', 'spam')
+        stdout = self.run_ok('--spacing', '8', '2004')
+        self.assertIn(b'Su        Mo', stdout)
+
+    def test_option_months(self):
+        self.assertFailure('-m')
+        self.assertFailure('--month')
+        self.assertFailure('-m', 'spam')
+        stdout = self.run_ok('--months', '1', '2004')
+        self.assertIn('\nMo Tu We Th Fr Sa Su\n', stdout)
+
+    def test_option_type(self):
+        self.assertFailure('-t')
+        self.assertFailure('--type')
+        self.assertFailure('-t', 'spam')
+        stdout = self.run_ok('--type', 'text', '2004')
+        self.assertEqual(stdout.strip(), result_2004_text.strip())
+        stdout = self.run_ok('--type', 'html', '2004')
+        self.assertEqual(stdout[:6], b'<?xml ')
+        self.assertIn(b'<title>Calendar for 2004</title>', stdout)
+
+    def test_html_output_current_year(self):
+        stdout = self.run_ok('--type', 'html')
+        year = datetime.datetime.now().year
+        self.assertIn(('<title>Calendar for %s</title>' % year).encode(),
+                      stdout)
+        self.assertIn(b'<tr><th colspan="7" class="month">January</th></tr>',
+                      stdout)
+
+    def test_html_output_year_encoding(self):
+        stdout = self.run_ok('-t', 'html', '--encoding', 'ascii', '2004')
+        self.assertEqual(stdout.strip(), result_2004_html.strip())
+
+    def test_html_output_year_css(self):
+        self.assertFailure('-t', 'html', '-c')
+        self.assertFailure('-t', 'html', '--css')
+        stdout = self.run_ok('-t', 'html', '--css', 'custom.css', '2004')
+        self.assertIn(b'<link rel="stylesheet" type="text/css" '
+                      b'href="custom.css" />', stdout)
+
+
 def test_main():
     test_support.run_unittest(
         OutputTestCase,
@@ -468,8 +597,10 @@
         SundayTestCase,
         MonthRangeTestCase,
         LeapdaysTestCase,
+        CommandLineTestCase,
     )
 
 
 if __name__ == "__main__":
     test_main()
+    unittest.main()
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -98,6 +98,8 @@
 Tests
 -----
 
+- Issue #18982: Add tests for CLI of the calendar module.
+
 - Issue #19949: The test_xpickle test now tests compatibility with installed
   Python 2.7 and reports skipped tests.  Based on patch by Zachary Ware.
 

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Thu Feb  5 14:19:44 2015
From: python-checkins at python.org (serhiy.storchaka)
Date: Thu, 05 Feb 2015 13:19:44 +0000
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzE4OTgy?=
 =?utf-8?q?=3A_Add_tests_for_CLI_of_the_calendar_module=2E?=
Message-ID: <20150205131942.34388.33650@psf.io>

https://hg.python.org/cpython/rev/28a9da0842aa
changeset:   94517:28a9da0842aa
branch:      3.4
parent:      94515:2e6972f1104d
user:        Serhiy Storchaka <storchaka at gmail.com>
date:        Thu Feb 05 15:14:35 2015 +0200
summary:
  Issue #18982: Add tests for CLI of the calendar module.

files:
  Lib/test/test_calendar.py |  155 +++++++++++++++++++++----
  Misc/NEWS                 |    2 +
  2 files changed, 132 insertions(+), 25 deletions(-)


diff --git a/Lib/test/test_calendar.py b/Lib/test/test_calendar.py
--- a/Lib/test/test_calendar.py
+++ b/Lib/test/test_calendar.py
@@ -2,13 +2,14 @@
 import unittest
 
 from test import support
-from test.script_helper import assert_python_ok
+from test.script_helper import assert_python_ok, assert_python_failure
 import time
 import locale
 import sys
 import datetime
+import os
 
-result_2004_01_text = """
+result_2004_01_text = """\
     January 2004
 Mo Tu We Th Fr Sa Su
           1  2  3  4
@@ -18,7 +19,7 @@
 26 27 28 29 30 31
 """
 
-result_2004_text = """
+result_2004_text = """\
                                   2004
 
       January                   February                   March
@@ -56,7 +57,7 @@
 25 26 27 28 29 30 31      29 30                     27 28 29 30 31
 """
 
-result_2004_html = """
+result_2004_html = """\
 <?xml version="1.0" encoding="%(e)s"?>
 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
 <html>
@@ -327,8 +328,8 @@
     def check_htmlcalendar_encoding(self, req, res):
         cal = calendar.HTMLCalendar()
         self.assertEqual(
-            cal.formatyearpage(2004, encoding=req).strip(b' \t\n'),
-            (result_2004_html % {'e': res}).strip(' \t\n').encode(res)
+            cal.formatyearpage(2004, encoding=req),
+            (result_2004_html % {'e': res}).encode(res)
         )
 
     def test_output(self):
@@ -339,8 +340,8 @@
 
     def test_output_textcalendar(self):
         self.assertEqual(
-            calendar.TextCalendar().formatyear(2004).strip(),
-            result_2004_text.strip()
+            calendar.TextCalendar().formatyear(2004),
+            result_2004_text
         )
 
     def test_output_htmlcalendar_encoding_ascii(self):
@@ -383,8 +384,8 @@
 
     def test_formatmonth(self):
         self.assertEqual(
-            calendar.TextCalendar().formatmonth(2004, 1).strip(),
-            result_2004_01_text.strip()
+            calendar.TextCalendar().formatmonth(2004, 1),
+            result_2004_01_text
         )
 
     def test_formatmonthname_with_year(self):
@@ -692,23 +693,127 @@
         self.assertEqual(calendar.leapdays(1997,2020), 5)
 
 
-class ConsoleOutputTestCase(unittest.TestCase):
-    def test_outputs_bytes(self):
-        (return_code, stdout, stderr) = assert_python_ok('-m', 'calendar', '--type=html', '2010')
+def conv(s):
+    return s.replace('\n', os.linesep).encode()
+
+class CommandLineTestCase(unittest.TestCase):
+    def run_ok(self, *args):
+        return assert_python_ok('-m', 'calendar', *args)[1]
+
+    def assertFailure(self, *args):
+        rc, stdout, stderr = assert_python_failure('-m', 'calendar', *args)
+        self.assertIn(b'Usage:', stderr)
+        self.assertEqual(rc, 2)
+
+    def test_help(self):
+        stdout = self.run_ok('-h')
+        self.assertIn(b'Usage:', stdout)
+        self.assertIn(b'calendar.py', stdout)
+        self.assertIn(b'--help', stdout)
+
+    def test_illegal_arguments(self):
+        self.assertFailure('-z')
+        #self.assertFailure('spam')
+        #self.assertFailure('2004', 'spam')
+        self.assertFailure('-t', 'html', '2004', '1')
+
+    def test_output_current_year(self):
+        stdout = self.run_ok()
+        year = datetime.datetime.now().year
+        self.assertIn((' %s' % year).encode(), stdout)
+        self.assertIn(b'January', stdout)
+        self.assertIn(b'Mo Tu We Th Fr Sa Su', stdout)
+
+    def test_output_year(self):
+        stdout = self.run_ok('2004')
+        self.assertEqual(stdout, conv(result_2004_text))
+
+    def test_output_month(self):
+        stdout = self.run_ok('2004', '1')
+        self.assertEqual(stdout, conv(result_2004_01_text))
+
+    def test_option_encoding(self):
+        self.assertFailure('-e')
+        self.assertFailure('--encoding')
+        stdout = self.run_ok('--encoding', 'utf-16-le', '2004')
+        self.assertEqual(stdout, result_2004_text.encode('utf-16-le'))
+
+    def test_option_locale(self):
+        self.assertFailure('-L')
+        self.assertFailure('--locale')
+        self.assertFailure('-L', 'en')
+        lang, enc = locale.getdefaultlocale()
+        lang = lang or 'C'
+        enc = enc or 'UTF-8'
+        try:
+            oldlocale = locale.getlocale(locale.LC_TIME)
+            try:
+                locale.setlocale(locale.LC_TIME, (lang, enc))
+            finally:
+                locale.setlocale(locale.LC_TIME, oldlocale)
+        except (locale.Error, ValueError):
+            self.skipTest('cannot set the system default locale')
+        stdout = self.run_ok('--locale', lang, '--encoding', enc, '2004')
+        self.assertIn('2004'.encode(enc), stdout)
+
+    def test_option_width(self):
+        self.assertFailure('-w')
+        self.assertFailure('--width')
+        self.assertFailure('-w', 'spam')
+        stdout = self.run_ok('--width', '3', '2004')
+        self.assertIn(b'Mon Tue Wed Thu Fri Sat Sun', stdout)
+
+    def test_option_lines(self):
+        self.assertFailure('-l')
+        self.assertFailure('--lines')
+        self.assertFailure('-l', 'spam')
+        stdout = self.run_ok('--lines', '2', '2004')
+        self.assertIn(conv('December\n\nMo Tu We'), stdout)
+
+    def test_option_spacing(self):
+        self.assertFailure('-s')
+        self.assertFailure('--spacing')
+        self.assertFailure('-s', 'spam')
+        stdout = self.run_ok('--spacing', '8', '2004')
+        self.assertIn(b'Su        Mo', stdout)
+
+    def test_option_months(self):
+        self.assertFailure('-m')
+        self.assertFailure('--month')
+        self.assertFailure('-m', 'spam')
+        stdout = self.run_ok('--months', '1', '2004')
+        self.assertIn(conv('\nMo Tu We Th Fr Sa Su\n'), stdout)
+
+    def test_option_type(self):
+        self.assertFailure('-t')
+        self.assertFailure('--type')
+        self.assertFailure('-t', 'spam')
+        stdout = self.run_ok('--type', 'text', '2004')
+        self.assertEqual(stdout, conv(result_2004_text))
+        stdout = self.run_ok('--type', 'html', '2004')
         self.assertEqual(stdout[:6], b'<?xml ')
+        self.assertIn(b'<title>Calendar for 2004</title>', stdout)
 
-def test_main():
-    support.run_unittest(
-        OutputTestCase,
-        CalendarTestCase,
-        MondayTestCase,
-        SundayTestCase,
-        TimegmTestCase,
-        MonthRangeTestCase,
-        LeapdaysTestCase,
-        ConsoleOutputTestCase
-    )
+    def test_html_output_current_year(self):
+        stdout = self.run_ok('--type', 'html')
+        year = datetime.datetime.now().year
+        self.assertIn(('<title>Calendar for %s</title>' % year).encode(),
+                      stdout)
+        self.assertIn(b'<tr><th colspan="7" class="month">January</th></tr>',
+                      stdout)
+
+    def test_html_output_year_encoding(self):
+        stdout = self.run_ok('-t', 'html', '--encoding', 'ascii', '2004')
+        self.assertEqual(stdout,
+                         (result_2004_html % {'e': 'ascii'}).encode('ascii'))
+
+    def test_html_output_year_css(self):
+        self.assertFailure('-t', 'html', '-c')
+        self.assertFailure('-t', 'html', '--css')
+        stdout = self.run_ok('-t', 'html', '--css', 'custom.css', '2004')
+        self.assertIn(b'<link rel="stylesheet" type="text/css" '
+                      b'href="custom.css" />', stdout)
 
 
 if __name__ == "__main__":
-    test_main()
+    unittest.main()
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -323,6 +323,8 @@
 Tests
 -----
 
+- Issue #18982: Add tests for CLI of the calendar module.
+
 - Issue #19548: Added some additional checks to test_codecs to ensure that
   statements in the updated documentation remain accurate. Patch by Martin
   Panter.

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Thu Feb  5 14:19:44 2015
From: python-checkins at python.org (serhiy.storchaka)
Date: Thu, 05 Feb 2015 13:19:44 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?q?=29=3A_Issue_=2318982=3A_Add_tests_for_CLI_of_the_calendar_modu?=
 =?utf-8?b?bGUu?=
Message-ID: <20150205131943.96070.91480@psf.io>

https://hg.python.org/cpython/rev/e057da873673
changeset:   94518:e057da873673
parent:      94516:8ec13c088465
parent:      94517:28a9da0842aa
user:        Serhiy Storchaka <storchaka at gmail.com>
date:        Thu Feb 05 15:17:49 2015 +0200
summary:
  Issue #18982: Add tests for CLI of the calendar module.

files:
  Lib/test/test_calendar.py |  155 +++++++++++++++++++++----
  Misc/NEWS                 |    2 +
  2 files changed, 132 insertions(+), 25 deletions(-)


diff --git a/Lib/test/test_calendar.py b/Lib/test/test_calendar.py
--- a/Lib/test/test_calendar.py
+++ b/Lib/test/test_calendar.py
@@ -2,13 +2,14 @@
 import unittest
 
 from test import support
-from test.script_helper import assert_python_ok
+from test.script_helper import assert_python_ok, assert_python_failure
 import time
 import locale
 import sys
 import datetime
+import os
 
-result_2004_01_text = """
+result_2004_01_text = """\
     January 2004
 Mo Tu We Th Fr Sa Su
           1  2  3  4
@@ -18,7 +19,7 @@
 26 27 28 29 30 31
 """
 
-result_2004_text = """
+result_2004_text = """\
                                   2004
 
       January                   February                   March
@@ -56,7 +57,7 @@
 25 26 27 28 29 30 31      29 30                     27 28 29 30 31
 """
 
-result_2004_html = """
+result_2004_html = """\
 <?xml version="1.0" encoding="%(e)s"?>
 <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
 <html>
@@ -327,8 +328,8 @@
     def check_htmlcalendar_encoding(self, req, res):
         cal = calendar.HTMLCalendar()
         self.assertEqual(
-            cal.formatyearpage(2004, encoding=req).strip(b' \t\n'),
-            (result_2004_html % {'e': res}).strip(' \t\n').encode(res)
+            cal.formatyearpage(2004, encoding=req),
+            (result_2004_html % {'e': res}).encode(res)
         )
 
     def test_output(self):
@@ -339,8 +340,8 @@
 
     def test_output_textcalendar(self):
         self.assertEqual(
-            calendar.TextCalendar().formatyear(2004).strip(),
-            result_2004_text.strip()
+            calendar.TextCalendar().formatyear(2004),
+            result_2004_text
         )
 
     def test_output_htmlcalendar_encoding_ascii(self):
@@ -383,8 +384,8 @@
 
     def test_formatmonth(self):
         self.assertEqual(
-            calendar.TextCalendar().formatmonth(2004, 1).strip(),
-            result_2004_01_text.strip()
+            calendar.TextCalendar().formatmonth(2004, 1),
+            result_2004_01_text
         )
 
     def test_formatmonthname_with_year(self):
@@ -692,23 +693,127 @@
         self.assertEqual(calendar.leapdays(1997,2020), 5)
 
 
-class ConsoleOutputTestCase(unittest.TestCase):
-    def test_outputs_bytes(self):
-        (return_code, stdout, stderr) = assert_python_ok('-m', 'calendar', '--type=html', '2010')
+def conv(s):
+    return s.replace('\n', os.linesep).encode()
+
+class CommandLineTestCase(unittest.TestCase):
+    def run_ok(self, *args):
+        return assert_python_ok('-m', 'calendar', *args)[1]
+
+    def assertFailure(self, *args):
+        rc, stdout, stderr = assert_python_failure('-m', 'calendar', *args)
+        self.assertIn(b'Usage:', stderr)
+        self.assertEqual(rc, 2)
+
+    def test_help(self):
+        stdout = self.run_ok('-h')
+        self.assertIn(b'Usage:', stdout)
+        self.assertIn(b'calendar.py', stdout)
+        self.assertIn(b'--help', stdout)
+
+    def test_illegal_arguments(self):
+        self.assertFailure('-z')
+        #self.assertFailure('spam')
+        #self.assertFailure('2004', 'spam')
+        self.assertFailure('-t', 'html', '2004', '1')
+
+    def test_output_current_year(self):
+        stdout = self.run_ok()
+        year = datetime.datetime.now().year
+        self.assertIn((' %s' % year).encode(), stdout)
+        self.assertIn(b'January', stdout)
+        self.assertIn(b'Mo Tu We Th Fr Sa Su', stdout)
+
+    def test_output_year(self):
+        stdout = self.run_ok('2004')
+        self.assertEqual(stdout, conv(result_2004_text))
+
+    def test_output_month(self):
+        stdout = self.run_ok('2004', '1')
+        self.assertEqual(stdout, conv(result_2004_01_text))
+
+    def test_option_encoding(self):
+        self.assertFailure('-e')
+        self.assertFailure('--encoding')
+        stdout = self.run_ok('--encoding', 'utf-16-le', '2004')
+        self.assertEqual(stdout, result_2004_text.encode('utf-16-le'))
+
+    def test_option_locale(self):
+        self.assertFailure('-L')
+        self.assertFailure('--locale')
+        self.assertFailure('-L', 'en')
+        lang, enc = locale.getdefaultlocale()
+        lang = lang or 'C'
+        enc = enc or 'UTF-8'
+        try:
+            oldlocale = locale.getlocale(locale.LC_TIME)
+            try:
+                locale.setlocale(locale.LC_TIME, (lang, enc))
+            finally:
+                locale.setlocale(locale.LC_TIME, oldlocale)
+        except (locale.Error, ValueError):
+            self.skipTest('cannot set the system default locale')
+        stdout = self.run_ok('--locale', lang, '--encoding', enc, '2004')
+        self.assertIn('2004'.encode(enc), stdout)
+
+    def test_option_width(self):
+        self.assertFailure('-w')
+        self.assertFailure('--width')
+        self.assertFailure('-w', 'spam')
+        stdout = self.run_ok('--width', '3', '2004')
+        self.assertIn(b'Mon Tue Wed Thu Fri Sat Sun', stdout)
+
+    def test_option_lines(self):
+        self.assertFailure('-l')
+        self.assertFailure('--lines')
+        self.assertFailure('-l', 'spam')
+        stdout = self.run_ok('--lines', '2', '2004')
+        self.assertIn(conv('December\n\nMo Tu We'), stdout)
+
+    def test_option_spacing(self):
+        self.assertFailure('-s')
+        self.assertFailure('--spacing')
+        self.assertFailure('-s', 'spam')
+        stdout = self.run_ok('--spacing', '8', '2004')
+        self.assertIn(b'Su        Mo', stdout)
+
+    def test_option_months(self):
+        self.assertFailure('-m')
+        self.assertFailure('--month')
+        self.assertFailure('-m', 'spam')
+        stdout = self.run_ok('--months', '1', '2004')
+        self.assertIn(conv('\nMo Tu We Th Fr Sa Su\n'), stdout)
+
+    def test_option_type(self):
+        self.assertFailure('-t')
+        self.assertFailure('--type')
+        self.assertFailure('-t', 'spam')
+        stdout = self.run_ok('--type', 'text', '2004')
+        self.assertEqual(stdout, conv(result_2004_text))
+        stdout = self.run_ok('--type', 'html', '2004')
         self.assertEqual(stdout[:6], b'<?xml ')
+        self.assertIn(b'<title>Calendar for 2004</title>', stdout)
 
-def test_main():
-    support.run_unittest(
-        OutputTestCase,
-        CalendarTestCase,
-        MondayTestCase,
-        SundayTestCase,
-        TimegmTestCase,
-        MonthRangeTestCase,
-        LeapdaysTestCase,
-        ConsoleOutputTestCase
-    )
+    def test_html_output_current_year(self):
+        stdout = self.run_ok('--type', 'html')
+        year = datetime.datetime.now().year
+        self.assertIn(('<title>Calendar for %s</title>' % year).encode(),
+                      stdout)
+        self.assertIn(b'<tr><th colspan="7" class="month">January</th></tr>',
+                      stdout)
+
+    def test_html_output_year_encoding(self):
+        stdout = self.run_ok('-t', 'html', '--encoding', 'ascii', '2004')
+        self.assertEqual(stdout,
+                         (result_2004_html % {'e': 'ascii'}).encode('ascii'))
+
+    def test_html_output_year_css(self):
+        self.assertFailure('-t', 'html', '-c')
+        self.assertFailure('-t', 'html', '--css')
+        stdout = self.run_ok('-t', 'html', '--css', 'custom.css', '2004')
+        self.assertIn(b'<link rel="stylesheet" type="text/css" '
+                      b'href="custom.css" />', stdout)
 
 
 if __name__ == "__main__":
-    test_main()
+    unittest.main()
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -1629,6 +1629,8 @@
 Tests
 -----
 
+- Issue #18982: Add tests for CLI of the calendar module.
+
 - Issue #19548: Added some additional checks to test_codecs to ensure that
   statements in the updated documentation remain accurate. Patch by Martin
   Panter.

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Thu Feb  5 14:26:13 2015
From: python-checkins at python.org (victor.stinner)
Date: Thu, 05 Feb 2015 13:26:13 +0000
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogdGVzdF9tdWx0aXBy?=
 =?utf-8?q?ocessing=3A_tolerate_a_delta_of_30_ms_because_of_bad_clock_reso?=
 =?utf-8?q?lution?=
Message-ID: <20150205132607.106307.75854@psf.io>

https://hg.python.org/cpython/rev/a436592e60ae
changeset:   94520:a436592e60ae
branch:      3.4
parent:      94517:28a9da0842aa
user:        Victor Stinner <victor.stinner at gmail.com>
date:        Thu Feb 05 14:25:05 2015 +0100
summary:
  test_multiprocessing: tolerate a delta of 30 ms because of bad clock resolution
on Windows

files:
  Lib/test/_test_multiprocessing.py |  6 ++++--
  1 files changed, 4 insertions(+), 2 deletions(-)


diff --git a/Lib/test/_test_multiprocessing.py b/Lib/test/_test_multiprocessing.py
--- a/Lib/test/_test_multiprocessing.py
+++ b/Lib/test/_test_multiprocessing.py
@@ -716,9 +716,11 @@
     def test_timeout(self):
         q = multiprocessing.Queue()
         start = time.time()
-        self.assertRaises(pyqueue.Empty, q.get, True, 0.2)
+        self.assertRaises(pyqueue.Empty, q.get, True, 0.200)
         delta = time.time() - start
-        self.assertGreaterEqual(delta, 0.18)
+        # Tolerate a delta of 30 ms because of the bad clock resolution on
+        # Windows (usually 15.6 ms)
+        self.assertGreaterEqual(delta, 0.170)
 
 #
 #

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Thu Feb  5 14:26:14 2015
From: python-checkins at python.org (victor.stinner)
Date: Thu, 05 Feb 2015 13:26:14 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?q?=29=3A_=28Merge_3=2E4=29_test=5Fmultiprocessing=3A_tolerate_a_d?=
 =?utf-8?q?elta_of_30_ms_because_of_bad?=
Message-ID: <20150205132607.96084.46219@psf.io>

https://hg.python.org/cpython/rev/494376ec13d9
changeset:   94521:494376ec13d9
parent:      94518:e057da873673
parent:      94520:a436592e60ae
user:        Victor Stinner <victor.stinner at gmail.com>
date:        Thu Feb 05 14:25:53 2015 +0100
summary:
  (Merge 3.4) test_multiprocessing: tolerate a delta of 30 ms because of bad
clock resolution on Windows

files:
  Lib/test/_test_multiprocessing.py |  6 ++++--
  1 files changed, 4 insertions(+), 2 deletions(-)


diff --git a/Lib/test/_test_multiprocessing.py b/Lib/test/_test_multiprocessing.py
--- a/Lib/test/_test_multiprocessing.py
+++ b/Lib/test/_test_multiprocessing.py
@@ -716,9 +716,11 @@
     def test_timeout(self):
         q = multiprocessing.Queue()
         start = time.time()
-        self.assertRaises(pyqueue.Empty, q.get, True, 0.2)
+        self.assertRaises(pyqueue.Empty, q.get, True, 0.200)
         delta = time.time() - start
-        self.assertGreaterEqual(delta, 0.18)
+        # Tolerate a delta of 30 ms because of the bad clock resolution on
+        # Windows (usually 15.6 ms)
+        self.assertGreaterEqual(delta, 0.170)
 
 #
 #

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Thu Feb  5 17:04:41 2015
From: python-checkins at python.org (serhiy.storchaka)
Date: Thu, 05 Feb 2015 16:04:41 +0000
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzE4OTgy?=
 =?utf-8?q?=3A_Fixed_newlines_in_calendar_CLI_tests_on_Windows=2E?=
Message-ID: <20150205160410.106379.90846@psf.io>

https://hg.python.org/cpython/rev/f2991b52157e
changeset:   94522:f2991b52157e
branch:      2.7
parent:      94519:50a5c262b2e6
user:        Serhiy Storchaka <storchaka at gmail.com>
date:        Thu Feb 05 18:03:27 2015 +0200
summary:
  Issue #18982: Fixed newlines in calendar CLI tests on Windows.

files:
  Lib/test/test_calendar.py |  17 ++++++++++-------
  1 files changed, 10 insertions(+), 7 deletions(-)


diff --git a/Lib/test/test_calendar.py b/Lib/test/test_calendar.py
--- a/Lib/test/test_calendar.py
+++ b/Lib/test/test_calendar.py
@@ -471,6 +471,9 @@
         self.assertEqual(calendar.leapdays(1997,2020), 5)
 
 
+def conv(s):
+    return s.replace('\n', os.linesep)
+
 class CommandLineTestCase(unittest.TestCase):
     def run_ok(self, *args):
         return assert_python_ok('-m', 'calendar', *args)[1]
@@ -501,17 +504,17 @@
 
     def test_output_year(self):
         stdout = self.run_ok('2004')
-        self.assertEqual(stdout.strip(), result_2004_text.strip())
+        self.assertEqual(stdout.strip(), conv(result_2004_text).strip())
 
     def test_output_month(self):
         stdout = self.run_ok('2004', '1')
-        self.assertEqual(stdout.strip(), result_2004_01_text.strip())
+        self.assertEqual(stdout.strip(), conv(result_2004_01_text).strip())
 
     def test_option_encoding(self):
         self.assertFailure('-e')
         self.assertFailure('--encoding')
         stdout = self.run_ok('--encoding', 'rot-13', '2004')
-        self.assertEqual(stdout.strip(), result_2004_text.encode('rot-13').strip())
+        self.assertEqual(stdout.strip(), conv(result_2004_text.encode('rot-13')).strip())
 
     def test_option_locale(self):
         self.assertFailure('-L')
@@ -543,7 +546,7 @@
         self.assertFailure('--lines')
         self.assertFailure('-l', 'spam')
         stdout = self.run_ok('--lines', '2', '2004')
-        self.assertIn('December\n\nMo Tu We', stdout)
+        self.assertIn(conv('December\n\nMo Tu We'), stdout)
 
     def test_option_spacing(self):
         self.assertFailure('-s')
@@ -557,14 +560,14 @@
         self.assertFailure('--month')
         self.assertFailure('-m', 'spam')
         stdout = self.run_ok('--months', '1', '2004')
-        self.assertIn('\nMo Tu We Th Fr Sa Su\n', stdout)
+        self.assertIn(conv('\nMo Tu We Th Fr Sa Su\n'), stdout)
 
     def test_option_type(self):
         self.assertFailure('-t')
         self.assertFailure('--type')
         self.assertFailure('-t', 'spam')
         stdout = self.run_ok('--type', 'text', '2004')
-        self.assertEqual(stdout.strip(), result_2004_text.strip())
+        self.assertEqual(stdout.strip(), conv(result_2004_text).strip())
         stdout = self.run_ok('--type', 'html', '2004')
         self.assertEqual(stdout[:6], b'<?xml ')
         self.assertIn(b'<title>Calendar for 2004</title>', stdout)
@@ -579,7 +582,7 @@
 
     def test_html_output_year_encoding(self):
         stdout = self.run_ok('-t', 'html', '--encoding', 'ascii', '2004')
-        self.assertEqual(stdout.strip(), result_2004_html.strip())
+        self.assertEqual(stdout.strip(), conv(result_2004_html).strip())
 
     def test_html_output_year_css(self):
         self.assertFailure('-t', 'html', '-c')

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Fri Feb  6 04:17:13 2015
From: python-checkins at python.org (ned.deily)
Date: Fri, 06 Feb 2015 03:17:13 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?q?=29=3A_Issue_=2323212=3A_merge_from_3=2E4?=
Message-ID: <20150206031711.96082.49572@psf.io>

https://hg.python.org/cpython/rev/aeba7274d68a
changeset:   94525:aeba7274d68a
parent:      94521:494376ec13d9
parent:      94524:1ddf68f118c7
user:        Ned Deily <nad at acm.org>
date:        Fri Feb 06 14:16:29 2015 +1100
summary:
  Issue #23212: merge from 3.4

files:
  Mac/BuildScript/build-installer.py |  6 +++---
  Misc/NEWS                          |  2 +-
  2 files changed, 4 insertions(+), 4 deletions(-)


diff --git a/Mac/BuildScript/build-installer.py b/Mac/BuildScript/build-installer.py
--- a/Mac/BuildScript/build-installer.py
+++ b/Mac/BuildScript/build-installer.py
@@ -237,9 +237,9 @@
 
         result.extend([
           dict(
-              name="OpenSSL 1.0.1k",
-              url="https://www.openssl.org/source/openssl-1.0.1k.tar.gz",
-              checksum='d4f002bd22a56881340105028842ae1f',
+              name="OpenSSL 1.0.1l",
+              url="https://www.openssl.org/source/openssl-1.0.1l.tar.gz",
+              checksum='cdb22925fc9bc97ccbf1e007661f2aa6',
               patches=[
                   "openssl_sdk_makedepend.patch",
                    ],
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -1555,7 +1555,7 @@
 
 - Issue #21236: Build _msi.pyd with cabinet.lib instead of fci.lib
 
-- Issue #17128: Use private version of OpenSSL for 2.7.9 OS X 10.5+ installer.
+- Issue #17128: Use private version of OpenSSL for OS X 10.5+ installer.
 
 C API
 -----

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Fri Feb  6 04:17:14 2015
From: python-checkins at python.org (ned.deily)
Date: Fri, 06 Feb 2015 03:17:14 +0000
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMy40KTogSXNzdWUgIzIzMjEy?=
 =?utf-8?q?=3A_Update_OS_X_installer_build_OpenSSL_to_1=2E0=2E1l=2E?=
Message-ID: <20150206031711.39282.14642@psf.io>

https://hg.python.org/cpython/rev/1ddf68f118c7
changeset:   94524:1ddf68f118c7
branch:      3.4
parent:      94520:a436592e60ae
user:        Ned Deily <nad at acm.org>
date:        Fri Feb 06 14:13:30 2015 +1100
summary:
  Issue #23212: Update OS X installer build OpenSSL to 1.0.1l.
(currently only used for builds with <= 10.5 deployment targets)

files:
  Mac/BuildScript/build-installer.py |  6 +++---
  Misc/NEWS                          |  2 +-
  2 files changed, 4 insertions(+), 4 deletions(-)


diff --git a/Mac/BuildScript/build-installer.py b/Mac/BuildScript/build-installer.py
--- a/Mac/BuildScript/build-installer.py
+++ b/Mac/BuildScript/build-installer.py
@@ -237,9 +237,9 @@
 
         result.extend([
           dict(
-              name="OpenSSL 1.0.1k",
-              url="https://www.openssl.org/source/openssl-1.0.1k.tar.gz",
-              checksum='d4f002bd22a56881340105028842ae1f',
+              name="OpenSSL 1.0.1l",
+              url="https://www.openssl.org/source/openssl-1.0.1l.tar.gz",
+              checksum='cdb22925fc9bc97ccbf1e007661f2aa6',
               patches=[
                   "openssl_sdk_makedepend.patch",
                    ],
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -361,7 +361,7 @@
 
 - Issue #17219: Add library build dir for Python extension cross-builds.
 
-- Issue #17128: Use private version of OpenSSL for 2.7.9 OS X 10.5+ installer.
+- Issue #17128: Use private version of OpenSSL for 3.4.3 OS X 10.5+ installer.
 
 C API
 -----

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Fri Feb  6 04:17:13 2015
From: python-checkins at python.org (ned.deily)
Date: Fri, 06 Feb 2015 03:17:13 +0000
Subject: [Python-checkins] =?utf-8?b?Y3B5dGhvbiAoMi43KTogSXNzdWUgIzIzMjEy?=
 =?utf-8?q?=3A_Update_OS_X_installer_build_OpenSSL_to_1=2E0=2E1l=2E?=
Message-ID: <20150206031710.25861.31220@psf.io>

https://hg.python.org/cpython/rev/7b74c65758ca
changeset:   94523:7b74c65758ca
branch:      2.7
user:        Ned Deily <nad at acm.org>
date:        Fri Feb 06 14:11:36 2015 +1100
summary:
  Issue #23212: Update OS X installer build OpenSSL to 1.0.1l.
(currently only used for builds with <= 10.5 deployment targets)

files:
  Mac/BuildScript/build-installer.py |  6 +++---
  Misc/NEWS                          |  2 +-
  2 files changed, 4 insertions(+), 4 deletions(-)


diff --git a/Mac/BuildScript/build-installer.py b/Mac/BuildScript/build-installer.py
--- a/Mac/BuildScript/build-installer.py
+++ b/Mac/BuildScript/build-installer.py
@@ -237,9 +237,9 @@
 
         result.extend([
           dict(
-              name="OpenSSL 1.0.1k",
-              url="https://www.openssl.org/source/openssl-1.0.1k.tar.gz",
-              checksum='d4f002bd22a56881340105028842ae1f',
+              name="OpenSSL 1.0.1l",
+              url="https://www.openssl.org/source/openssl-1.0.1l.tar.gz",
+              checksum='cdb22925fc9bc97ccbf1e007661f2aa6',
               patches=[
                   "openssl_sdk_makedepend.patch",
                    ],
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -130,7 +130,7 @@
 - Issue #23032: Fix installer build failures on OS X 10.4 Tiger
   by disabling assembly code in the OpenSSL build.
 
-- Issue #23212: Update 10.5 OS X installer build to use OpenSSL 1.0.1k.
+- Issue #23212: Update 10.5 OS X installer build to use OpenSSL 1.0.1l.
 
 C API
 -----

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Fri Feb  6 04:31:01 2015
From: python-checkins at python.org (benjamin.peterson)
Date: Fri, 06 Feb 2015 03:31:01 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?b?KTogbWVyZ2UgMy40ICgjMjI3MzUp?=
Message-ID: <20150206033056.25861.97131@psf.io>

https://hg.python.org/cpython/rev/75fd0bd89eef
changeset:   94527:75fd0bd89eef
parent:      94525:aeba7274d68a
parent:      94526:6384c0cd3b2d
user:        Benjamin Peterson <benjamin at python.org>
date:        Thu Feb 05 22:29:46 2015 -0500
summary:
  merge 3.4 (#22735)

files:
  Lib/test/test_descr.py |  227 +++++++++-
  Misc/ACKS              |    1 +
  Misc/NEWS              |    3 +
  Objects/typeobject.c   |  699 +++++++++++++++++-----------
  4 files changed, 649 insertions(+), 281 deletions(-)


diff --git a/Lib/test/test_descr.py b/Lib/test/test_descr.py
--- a/Lib/test/test_descr.py
+++ b/Lib/test/test_descr.py
@@ -5057,11 +5057,236 @@
         self.assertLess(sys.getsizeof(vars(b)), sys.getsizeof({}))
 
 
+class DebugHelperMeta(type):
+    """
+    Sets default __doc__ and simplifies repr() output.
+    """
+    def __new__(mcls, name, bases, attrs):
+        if attrs.get('__doc__') is None:
+            attrs['__doc__'] = name  # helps when debugging with gdb
+        return type.__new__(mcls, name, bases, attrs)
+    def __repr__(cls):
+        return repr(cls.__name__)
+
+
+class MroTest(unittest.TestCase):
+    """
+    Regressions for some bugs revealed through
+    mcsl.mro() customization (typeobject.c: mro_internal()) and
+    cls.__bases__ assignment (typeobject.c: type_set_bases()).
+    """
+
+    def setUp(self):
+        self.step = 0
+        self.ready = False
+
+    def step_until(self, limit):
+        ret = (self.step < limit)
+        if ret:
+            self.step += 1
+        return ret
+
+    def test_incomplete_set_bases_on_self(self):
+        """
+        type_set_bases must be aware that type->tp_mro can be NULL.
+        """
+        class M(DebugHelperMeta):
+            def mro(cls):
+                if self.step_until(1):
+                    assert cls.__mro__ is None
+                    cls.__bases__ += ()
+
+                return type.mro(cls)
+
+        class A(metaclass=M):
+            pass
+
+    def test_reent_set_bases_on_base(self):
+        """
+        Deep reentrancy must not over-decref old_mro.
+        """
+        class M(DebugHelperMeta):
+            def mro(cls):
+                if cls.__mro__ is not None and cls.__name__ == 'B':
+                    # 4-5 steps are usually enough to make it crash somewhere
+                    if self.step_until(10):
+                        A.__bases__ += ()
+
+                return type.mro(cls)
+
+        class A(metaclass=M):
+            pass
+        class B(A):
+            pass
+        B.__bases__ += ()
+
+    def test_reent_set_bases_on_direct_base(self):
+        """
+        Similar to test_reent_set_bases_on_base, but may crash differently.
+        """
+        class M(DebugHelperMeta):
+            def mro(cls):
+                base = cls.__bases__[0]
+                if base is not object:
+                    if self.step_until(5):
+                        base.__bases__ += ()
+
+                return type.mro(cls)
+
+        class A(metaclass=M):
+            pass
+        class B(A):
+            pass
+        class C(B):
+            pass
+
+    def test_reent_set_bases_tp_base_cycle(self):
+        """
+        type_set_bases must check for an inheritance cycle not only through
+        MRO of the type, which may be not yet updated in case of reentrance,
+        but also through tp_base chain, which is assigned before diving into
+        inner calls to mro().
+
+        Otherwise, the following snippet can loop forever:
+            do {
+                // ...
+                type = type->tp_base;
+            } while (type != NULL);
+
+        Functions that rely on tp_base (like solid_base and PyType_IsSubtype)
+        would not be happy in that case, causing a stack overflow.
+        """
+        class M(DebugHelperMeta):
+            def mro(cls):
+                if self.ready:
+                    if cls.__name__ == 'B1':
+                        B2.__bases__ = (B1,)
+                    if cls.__name__ == 'B2':
+                        B1.__bases__ = (B2,)
+                return type.mro(cls)
+
+        class A(metaclass=M):
+            pass
+        class B1(A):
+            pass
+        class B2(A):
+            pass
+
+        self.ready = True
+        with self.assertRaises(TypeError):
+            B1.__bases__ += ()
+
+    def test_tp_subclasses_cycle_in_update_slots(self):
+        """
+        type_set_bases must check for reentrancy upon finishing its job
+        by updating tp_subclasses of old/new bases of the type.
+        Otherwise, an implicit inheritance cycle through tp_subclasses
+        can break functions that recurse on elements of that field
+        (like recurse_down_subclasses and mro_hierarchy) eventually
+        leading to a stack overflow.
+        """
+        class M(DebugHelperMeta):
+            def mro(cls):
+                if self.ready and cls.__name__ == 'C':
+                    self.ready = False
+                    C.__bases__ = (B2,)
+                return type.mro(cls)
+
+        class A(metaclass=M):
+            pass
+        class B1(A):
+            pass
+        class B2(A):
+            pass
+        class C(A):
+            pass
+
+        self.ready = True
+        C.__bases__ = (B1,)
+        B1.__bases__ = (C,)
+
+        self.assertEqual(C.__bases__, (B2,))
+        self.assertEqual(B2.__subclasses__(), [C])
+        self.assertEqual(B1.__subclasses__(), [])
+
+        self.assertEqual(B1.__bases__, (C,))
+        self.assertEqual(C.__subclasses__(), [B1])
+
+    def test_tp_subclasses_cycle_error_return_path(self):
+        """
+        The same as test_tp_subclasses_cycle_in_update_slots, but tests
+        a code path executed on error (goto bail).
+        """
+        class E(Exception):
+            pass
+        class M(DebugHelperMeta):
+            def mro(cls):
+                if self.ready and cls.__name__ == 'C':
+                    if C.__bases__ == (B2,):
+                        self.ready = False
+                    else:
+                        C.__bases__ = (B2,)
+                        raise E
+                return type.mro(cls)
+
+        class A(metaclass=M):
+            pass
+        class B1(A):
+            pass
+        class B2(A):
+            pass
+        class C(A):
+            pass
+
+        self.ready = True
+        with self.assertRaises(E):
+            C.__bases__ = (B1,)
+        B1.__bases__ = (C,)
+
+        self.assertEqual(C.__bases__, (B2,))
+        self.assertEqual(C.__mro__, tuple(type.mro(C)))
+
+    def test_incomplete_extend(self):
+        """
+        Extending an unitialized type with type->tp_mro == NULL must
+        throw a reasonable TypeError exception, instead of failing
+        with PyErr_BadInternalCall.
+        """
+        class M(DebugHelperMeta):
+            def mro(cls):
+                if cls.__mro__ is None and cls.__name__ != 'X':
+                    with self.assertRaises(TypeError):
+                        class X(cls):
+                            pass
+
+                return type.mro(cls)
+
+        class A(metaclass=M):
+            pass
+
+    def test_incomplete_super(self):
+        """
+        Attrubute lookup on a super object must be aware that
+        its target type can be uninitialized (type->tp_mro == NULL).
+        """
+        class M(DebugHelperMeta):
+            def mro(cls):
+                if cls.__mro__ is None:
+                    with self.assertRaises(AttributeError):
+                        super(cls, cls).xxx
+
+                return type.mro(cls)
+
+        class A(metaclass=M):
+            pass
+
+
 def test_main():
     # Run all local test cases, with PTypesLongInitTest first.
     support.run_unittest(PTypesLongInitTest, OperatorsTest,
                          ClassPropertiesAndMethods, DictProxyTests,
-                         MiscTests, PicklingTests, SharedKeyTests)
+                         MiscTests, PicklingTests, SharedKeyTests,
+                         MroTest)
 
 if __name__ == "__main__":
     test_main()
diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -16,6 +16,7 @@
 Rajiv Abraham
 David Abrahams
 Marc Abramowitz
+Eldar Abusalimov
 Ron Adam
 Anton Afanasyev
 Ali Afshar
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -10,6 +10,9 @@
 Core and Builtins
 -----------------
 
+- Issue #22735: Fix many edge cases (including crashes) involving custom mro()
+  implementations.
+
 - Issue #22896: Avoid using PyObject_AsCharBuffer(), PyObject_AsReadBuffer()
   and PyObject_AsWriteBuffer().
 
diff --git a/Objects/typeobject.c b/Objects/typeobject.c
--- a/Objects/typeobject.c
+++ b/Objects/typeobject.c
@@ -564,9 +564,11 @@
 }
 
 static PyTypeObject *best_base(PyObject *);
-static int mro_internal(PyTypeObject *);
+static int mro_internal(PyTypeObject *, PyObject **);
+static int type_is_subtype_base_chain(PyTypeObject *, PyTypeObject *);
 static int compatible_for_assignment(PyTypeObject *, PyTypeObject *, char *);
 static int add_subclass(PyTypeObject*, PyTypeObject*);
+static int add_all_subclasses(PyTypeObject *type, PyObject *bases);
 static void remove_subclass(PyTypeObject *, PyTypeObject *);
 static void remove_all_subclasses(PyTypeObject *type, PyObject *bases);
 static void update_all_slots(PyTypeObject *);
@@ -576,167 +578,194 @@
                              update_callback callback, void *data);
 static int recurse_down_subclasses(PyTypeObject *type, PyObject *name,
                                    update_callback callback, void *data);
+static PyObject *type_subclasses(PyTypeObject *type, PyObject *ignored);
 
 static int
-mro_subclasses(PyTypeObject *type, PyObject* temp)
-{
-    PyTypeObject *subclass;
-    PyObject *ref, *subclasses, *old_mro;
+mro_hierarchy(PyTypeObject *type, PyObject *temp)
+{
+    int res;
+    PyObject *new_mro, *old_mro;
+    PyObject *tuple;
+    PyObject *subclasses;
+    Py_ssize_t i, n;
+
+    res = mro_internal(type, &old_mro);
+    if (res <= 0)
+        /* error / reentrance */
+        return res;
+    new_mro = type->tp_mro;
+
+    if (old_mro != NULL)
+        tuple = PyTuple_Pack(3, type, new_mro, old_mro);
+    else
+        tuple = PyTuple_Pack(2, type, new_mro);
+
+    if (tuple != NULL)
+        res = PyList_Append(temp, tuple);
+    else
+        res = -1;
+    Py_XDECREF(tuple);
+
+    if (res < 0) {
+        type->tp_mro = old_mro;
+        Py_DECREF(new_mro);
+        return -1;
+    }
+    Py_XDECREF(old_mro);
+
+    /* Obtain a copy of subclasses list to iterate over.
+
+       Otherwise type->tp_subclasses might be altered
+       in the middle of the loop, for example, through a custom mro(),
+       by invoking type_set_bases on some subclass of the type
+       which in turn calls remove_subclass/add_subclass on this type.
+
+       Finally, this makes things simple avoiding the need to deal
+       with dictionary iterators and weak references.
+    */
+    subclasses = type_subclasses(type, NULL);
+    if (subclasses == NULL)
+        return -1;
+    n = PyList_GET_SIZE(subclasses);
+    for (i = 0; i < n; i++) {
+        PyTypeObject *subclass;
+        subclass = (PyTypeObject *)PyList_GET_ITEM(subclasses, i);
+        res = mro_hierarchy(subclass, temp);
+        if (res < 0)
+            break;
+    }
+    Py_DECREF(subclasses);
+
+    return res;
+}
+
+static int
+type_set_bases(PyTypeObject *type, PyObject *new_bases, void *context)
+{
+    int res = 0;
+    PyObject *temp;
+    PyObject *old_bases;
+    PyTypeObject *new_base, *old_base;
     Py_ssize_t i;
 
-    subclasses = type->tp_subclasses;
-    if (subclasses == NULL)
-        return 0;
-    assert(PyDict_CheckExact(subclasses));
-    i = 0;
-
-    while (PyDict_Next(subclasses, &i, NULL, &ref)) {
-        assert(PyWeakref_CheckRef(ref));
-        subclass = (PyTypeObject *)PyWeakref_GET_OBJECT(ref);
-        assert(subclass != NULL);
-        if ((PyObject *)subclass == Py_None)
-            continue;
-        assert(PyType_Check(subclass));
-        old_mro = subclass->tp_mro;
-        if (mro_internal(subclass) < 0) {
-            subclass->tp_mro = old_mro;
-            return -1;
-        }
-        else {
-            PyObject* tuple;
-            tuple = PyTuple_Pack(2, subclass, old_mro);
-            Py_DECREF(old_mro);
-            if (!tuple)
-                return -1;
-            if (PyList_Append(temp, tuple) < 0)
-                return -1;
-            Py_DECREF(tuple);
-        }
-        if (mro_subclasses(subclass, temp) < 0)
-            return -1;
-    }
-    return 0;
-}
-
-static int
-type_set_bases(PyTypeObject *type, PyObject *value, void *context)
-{
-    Py_ssize_t i;
-    int r = 0;
-    PyObject *ob, *temp;
-    PyTypeObject *new_base, *old_base;
-    PyObject *old_bases, *old_mro;
-
-    if (!check_set_special_type_attr(type, value, "__bases__"))
+    if (!check_set_special_type_attr(type, new_bases, "__bases__"))
         return -1;
-    if (!PyTuple_Check(value)) {
+    if (!PyTuple_Check(new_bases)) {
         PyErr_Format(PyExc_TypeError,
              "can only assign tuple to %s.__bases__, not %s",
-                 type->tp_name, Py_TYPE(value)->tp_name);
+                 type->tp_name, Py_TYPE(new_bases)->tp_name);
         return -1;
     }
-    if (PyTuple_GET_SIZE(value) == 0) {
+    if (PyTuple_GET_SIZE(new_bases) == 0) {
         PyErr_Format(PyExc_TypeError,
              "can only assign non-empty tuple to %s.__bases__, not ()",
                  type->tp_name);
         return -1;
     }
-    for (i = 0; i < PyTuple_GET_SIZE(value); i++) {
-        ob = PyTuple_GET_ITEM(value, i);
+    for (i = 0; i < PyTuple_GET_SIZE(new_bases); i++) {
+        PyObject *ob;
+        PyTypeObject *base;
+
+        ob = PyTuple_GET_ITEM(new_bases, i);
         if (!PyType_Check(ob)) {
             PyErr_Format(PyExc_TypeError,
                          "%s.__bases__ must be tuple of classes, not '%s'",
                          type->tp_name, Py_TYPE(ob)->tp_name);
             return -1;
         }
-        if (PyType_IsSubtype((PyTypeObject*)ob, type)) {
+
+        base = (PyTypeObject*)ob;
+        if (PyType_IsSubtype(base, type) ||
+            /* In case of reentering here again through a custom mro()
+               the above check is not enough since it relies on
+               base->tp_mro which would gonna be updated inside
+               mro_internal only upon returning from the mro().
+
+               However, base->tp_base has already been assigned (see
+               below), which in turn may cause an inheritance cycle
+               through tp_base chain.  And this is definitely
+               not what you want to ever happen.  */
+            (base->tp_mro != NULL && type_is_subtype_base_chain(base, type))) {
+
             PyErr_SetString(PyExc_TypeError,
                             "a __bases__ item causes an inheritance cycle");
             return -1;
         }
     }
 
-    new_base = best_base(value);
-
-    if (!new_base)
+    new_base = best_base(new_bases);
+    if (new_base == NULL)
         return -1;
 
     if (!compatible_for_assignment(type->tp_base, new_base, "__bases__"))
         return -1;
 
+    Py_INCREF(new_bases);
     Py_INCREF(new_base);
-    Py_INCREF(value);
 
     old_bases = type->tp_bases;
     old_base = type->tp_base;
-    old_mro = type->tp_mro;
-
-    type->tp_bases = value;
+
+    type->tp_bases = new_bases;
     type->tp_base = new_base;
 
-    if (mro_internal(type) < 0) {
+    temp = PyList_New(0);
+    if (temp == NULL)
         goto bail;
-    }
-
-    temp = PyList_New(0);
-    if (!temp)
-        goto bail;
-
-    r = mro_subclasses(type, temp);
-
-    if (r < 0) {
-        for (i = 0; i < PyList_Size(temp); i++) {
-            PyTypeObject* cls;
-            PyObject* mro;
-            PyArg_UnpackTuple(PyList_GET_ITEM(temp, i),
-                             "", 2, 2, &cls, &mro);
-            Py_INCREF(mro);
-            ob = cls->tp_mro;
-            cls->tp_mro = mro;
-            Py_DECREF(ob);
-        }
-        Py_DECREF(temp);
-        goto bail;
-    }
-
+    if (mro_hierarchy(type, temp) < 0)
+        goto undo;
     Py_DECREF(temp);
 
-    /* any base that was in __bases__ but now isn't, we
-       need to remove |type| from its tp_subclasses.
-       conversely, any class now in __bases__ that wasn't
-       needs to have |type| added to its subclasses. */
-
-    /* for now, sod that: just remove from all old_bases,
-       add to all new_bases */
-
-    remove_all_subclasses(type, old_bases);
-
-    for (i = PyTuple_GET_SIZE(value) - 1; i >= 0; i--) {
-        ob = PyTuple_GET_ITEM(value, i);
-        if (PyType_Check(ob)) {
-            if (add_subclass((PyTypeObject*)ob, type) < 0)
-                r = -1;
-        }
-    }
-
-    update_all_slots(type);
+    /* Take no action in case if type->tp_bases has been replaced
+       through reentrance.  */
+    if (type->tp_bases == new_bases) {
+        /* any base that was in __bases__ but now isn't, we
+           need to remove |type| from its tp_subclasses.
+           conversely, any class now in __bases__ that wasn't
+           needs to have |type| added to its subclasses. */
+
+        /* for now, sod that: just remove from all old_bases,
+           add to all new_bases */
+        remove_all_subclasses(type, old_bases);
+        res = add_all_subclasses(type, new_bases);
+        update_all_slots(type);
+    }
 
     Py_DECREF(old_bases);
     Py_DECREF(old_base);
-    Py_DECREF(old_mro);
-
-    return r;
+
+    return res;
+
+  undo:
+    for (i = PyList_GET_SIZE(temp) - 1; i >= 0; i--) {
+        PyTypeObject *cls;
+        PyObject *new_mro, *old_mro = NULL;
+
+        PyArg_UnpackTuple(PyList_GET_ITEM(temp, i),
+                          "", 2, 3, &cls, &new_mro, &old_mro);
+        /* Do not rollback if cls has a newer version of MRO.  */
+        if (cls->tp_mro == new_mro) {
+            Py_XINCREF(old_mro);
+            cls->tp_mro = old_mro;
+            Py_DECREF(new_mro);
+        }
+    }
+    Py_DECREF(temp);
 
   bail:
-    Py_DECREF(type->tp_bases);
-    Py_DECREF(type->tp_base);
-    if (type->tp_mro != old_mro) {
-        Py_DECREF(type->tp_mro);
-    }
-
-    type->tp_bases = old_bases;
-    type->tp_base = old_base;
-    type->tp_mro = old_mro;
+    if (type->tp_bases == new_bases) {
+        assert(type->tp_base == new_base);
+
+        type->tp_bases = old_bases;
+        type->tp_base = old_base;
+
+        Py_DECREF(new_bases);
+        Py_DECREF(new_base);
+    }
+    else {
+        Py_DECREF(old_bases);
+        Py_DECREF(old_base);
+    }
 
     return -1;
 }
@@ -1312,6 +1341,18 @@
 
 /* type test with subclassing support */
 
+Py_LOCAL_INLINE(int)
+type_is_subtype_base_chain(PyTypeObject *a, PyTypeObject *b)
+{
+    do {
+        if (a == b)
+            return 1;
+        a = a->tp_base;
+    } while (a != NULL);
+
+    return (b == &PyBaseObject_Type);
+}
+
 int
 PyType_IsSubtype(PyTypeObject *a, PyTypeObject *b)
 {
@@ -1330,15 +1371,9 @@
         }
         return 0;
     }
-    else {
+    else
         /* a is not completely initilized yet; follow tp_base */
-        do {
-            if (a == b)
-                return 1;
-            a = a->tp_base;
-        } while (a != NULL);
-        return b == &PyBaseObject_Type;
-    }
+        return type_is_subtype_base_chain(a, b);
 }
 
 /* Internal routines to do a method lookup in the type
@@ -1605,10 +1640,11 @@
 }
 
 static int
-pmerge(PyObject *acc, PyObject* to_merge) {
+pmerge(PyObject *acc, PyObject* to_merge)
+{
+    int res = 0;
     Py_ssize_t i, j, to_merge_size, empty_cnt;
     int *remain;
-    int ok;
 
     to_merge_size = PyList_GET_SIZE(to_merge);
 
@@ -1646,15 +1682,13 @@
         candidate = PyList_GET_ITEM(cur_list, remain[i]);
         for (j = 0; j < to_merge_size; j++) {
             PyObject *j_lst = PyList_GET_ITEM(to_merge, j);
-            if (tail_contains(j_lst, remain[j], candidate)) {
+            if (tail_contains(j_lst, remain[j], candidate))
                 goto skip; /* continue outer loop */
-            }
         }
-        ok = PyList_Append(acc, candidate);
-        if (ok < 0) {
-            PyMem_FREE(remain);
-            return -1;
-        }
+        res = PyList_Append(acc, candidate);
+        if (res < 0)
+            goto out;
+
         for (j = 0; j < to_merge_size; j++) {
             PyObject *j_lst = PyList_GET_ITEM(to_merge, j);
             if (remain[j] < PyList_GET_SIZE(j_lst) &&
@@ -1666,22 +1700,25 @@
       skip: ;
     }
 
-    if (empty_cnt == to_merge_size) {
-        PyMem_FREE(remain);
-        return 0;
-    }
-    set_mro_error(to_merge, remain);
+    if (empty_cnt != to_merge_size) {
+        set_mro_error(to_merge, remain);
+        res = -1;
+    }
+
+  out:
     PyMem_FREE(remain);
-    return -1;
+
+    return res;
 }
 
 static PyObject *
 mro_implementation(PyTypeObject *type)
 {
+    PyObject *result = NULL;
+    PyObject *bases;
+    PyObject *to_merge, *bases_aslist;
+    int res;
     Py_ssize_t i, n;
-    int ok;
-    PyObject *bases, *result;
-    PyObject *to_merge, *bases_aslist;
 
     if (type->tp_dict == NULL) {
         if (PyType_Ready(type) < 0)
@@ -1705,42 +1742,44 @@
         return NULL;
 
     for (i = 0; i < n; i++) {
-        PyObject *base = PyTuple_GET_ITEM(bases, i);
-        PyObject *parentMRO;
-        parentMRO = PySequence_List(((PyTypeObject*)base)->tp_mro);
-        if (parentMRO == NULL) {
-            Py_DECREF(to_merge);
-            return NULL;
+        PyTypeObject *base;
+        PyObject *base_mro_aslist;
+
+        base = (PyTypeObject *)PyTuple_GET_ITEM(bases, i);
+        if (base->tp_mro == NULL) {
+            PyErr_Format(PyExc_TypeError,
+                         "Cannot extend an incomplete type '%.100s'",
+                         base->tp_name);
+            goto out;
         }
 
-        PyList_SET_ITEM(to_merge, i, parentMRO);
+        base_mro_aslist = PySequence_List(base->tp_mro);
+        if (base_mro_aslist == NULL)
+            goto out;
+
+        PyList_SET_ITEM(to_merge, i, base_mro_aslist);
     }
 
     bases_aslist = PySequence_List(bases);
-    if (bases_aslist == NULL) {
-        Py_DECREF(to_merge);
-        return NULL;
-    }
+    if (bases_aslist == NULL)
+        goto out;
     /* This is just a basic sanity check. */
     if (check_duplicates(bases_aslist) < 0) {
-        Py_DECREF(to_merge);
         Py_DECREF(bases_aslist);
-        return NULL;
+        goto out;
     }
     PyList_SET_ITEM(to_merge, n, bases_aslist);
 
     result = Py_BuildValue("[O]", (PyObject *)type);
-    if (result == NULL) {
-        Py_DECREF(to_merge);
-        return NULL;
-    }
-
-    ok = pmerge(result, to_merge);
+    if (result == NULL)
+        goto out;
+
+    res = pmerge(result, to_merge);
+    if (res < 0)
+        Py_CLEAR(result);
+
+  out:
     Py_DECREF(to_merge);
-    if (ok < 0) {
-        Py_DECREF(result);
-        return NULL;
-    }
 
     return result;
 }
@@ -1754,59 +1793,133 @@
 }
 
 static int
-mro_internal(PyTypeObject *type)
-{
-    PyObject *mro, *result, *tuple;
-    int checkit = 0;
-
-    if (Py_TYPE(type) == &PyType_Type) {
-        result = mro_implementation(type);
+mro_check(PyTypeObject *type, PyObject *mro)
+{
+    PyTypeObject *solid;
+    Py_ssize_t i, n;
+
+    solid = solid_base(type);
+
+    n = PyTuple_GET_SIZE(mro);
+    for (i = 0; i < n; i++) {
+        PyTypeObject *base;
+        PyObject *tmp;
+
+        tmp = PyTuple_GET_ITEM(mro, i);
+        if (!PyType_Check(tmp)) {
+            PyErr_Format(
+                PyExc_TypeError,
+                "mro() returned a non-class ('%.500s')",
+                Py_TYPE(tmp)->tp_name);
+            return -1;
+        }
+
+        base = (PyTypeObject*)tmp;
+        if (!PyType_IsSubtype(solid, solid_base(base))) {
+            PyErr_Format(
+                PyExc_TypeError,
+                "mro() returned base with unsuitable layout ('%.500s')",
+                base->tp_name);
+            return -1;
+        }
+    }
+
+    return 0;
+}
+
+/* Lookups an mcls.mro method, invokes it and checks the result (if needed,
+   in case of a custom mro() implementation).
+
+   Keep in mind that during execution of this function type->tp_mro
+   can be replaced due to possible reentrance (for example,
+   through type_set_bases):
+
+      - when looking up the mcls.mro attribute (it could be
+        a user-provided descriptor);
+
+      - from inside a custom mro() itself;
+
+      - through a finalizer of the return value of mro().
+*/
+static PyObject *
+mro_invoke(PyTypeObject *type)
+{
+    PyObject *mro_result;
+    PyObject *new_mro;
+    int custom = (Py_TYPE(type) != &PyType_Type);
+
+    if (custom) {
+        _Py_IDENTIFIER(mro);
+        PyObject *mro_meth = lookup_method((PyObject *)type, &PyId_mro);
+        if (mro_meth == NULL)
+            return NULL;
+        mro_result = PyObject_CallObject(mro_meth, NULL);
+        Py_DECREF(mro_meth);
     }
     else {
-        _Py_IDENTIFIER(mro);
-        checkit = 1;
-        mro = lookup_method((PyObject *)type, &PyId_mro);
-        if (mro == NULL)
-            return -1;
-        result = PyObject_CallObject(mro, NULL);
-        Py_DECREF(mro);
-    }
-    if (result == NULL)
+        mro_result = mro_implementation(type);
+    }
+    if (mro_result == NULL)
+        return NULL;
+
+    new_mro = PySequence_Tuple(mro_result);
+    Py_DECREF(mro_result);
+    if (new_mro == NULL)
+        return NULL;
+
+    if (custom && mro_check(type, new_mro) < 0) {
+        Py_DECREF(new_mro);
+        return NULL;
+    }
+
+    return new_mro;
+}
+
+/* Calculates and assigns a new MRO to type->tp_mro.
+   Return values and invariants:
+
+     - Returns 1 if a new MRO value has been set to type->tp_mro due to
+       this call of mro_internal (no tricky reentrancy and no errors).
+
+       In case if p_old_mro argument is not NULL, a previous value
+       of type->tp_mro is put there, and the ownership of this
+       reference is transferred to a caller.
+       Otherwise, the previous value (if any) is decref'ed.
+
+     - Returns 0 in case when type->tp_mro gets changed because of
+       reentering here through a custom mro() (see a comment to mro_invoke).
+
+       In this case, a refcount of an old type->tp_mro is adjusted
+       somewhere deeper in the call stack (by the innermost mro_internal
+       or its caller) and may become zero upon returning from here.
+       This also implies that the whole hierarchy of subclasses of the type
+       has seen the new value and updated their MRO accordingly.
+
+     - Returns -1 in case of an error.
+*/
+static int
+mro_internal(PyTypeObject *type, PyObject **p_old_mro)
+{
+    PyObject *new_mro, *old_mro;
+    int reent;
+
+    /* Keep a reference to be able to do a reentrancy check below.
+       Don't let old_mro be GC'ed and its address be reused for
+       another object, like (suddenly!) a new tp_mro.  */
+    old_mro = type->tp_mro;
+    Py_XINCREF(old_mro);
+    new_mro = mro_invoke(type);  /* might cause reentrance */
+    reent = (type->tp_mro != old_mro);
+    Py_XDECREF(old_mro);
+    if (new_mro == NULL)
         return -1;
-    tuple = PySequence_Tuple(result);
-    Py_DECREF(result);
-    if (tuple == NULL)
-        return -1;
-    if (checkit) {
-        Py_ssize_t i, len;
-        PyObject *cls;
-        PyTypeObject *solid;
-
-        solid = solid_base(type);
-
-        len = PyTuple_GET_SIZE(tuple);
-
-        for (i = 0; i < len; i++) {
-            PyTypeObject *t;
-            cls = PyTuple_GET_ITEM(tuple, i);
-            if (!PyType_Check(cls)) {
-                PyErr_Format(PyExc_TypeError,
-                 "mro() returned a non-class ('%.500s')",
-                                 Py_TYPE(cls)->tp_name);
-                Py_DECREF(tuple);
-                return -1;
-            }
-            t = (PyTypeObject*)cls;
-            if (!PyType_IsSubtype(solid, solid_base(t))) {
-                PyErr_Format(PyExc_TypeError,
-             "mro() returned base with unsuitable layout ('%.500s')",
-                                     t->tp_name);
-                        Py_DECREF(tuple);
-                        return -1;
-            }
-        }
-    }
-    type->tp_mro = tuple;
+
+    if (reent) {
+        Py_DECREF(new_mro);
+        return 0;
+    }
+
+    type->tp_mro = new_mro;
 
     type_mro_modified(type, type->tp_mro);
     /* corner case: the super class might have been hidden
@@ -1815,7 +1928,12 @@
 
     PyType_Modified(type);
 
-    return 0;
+    if (p_old_mro != NULL)
+        *p_old_mro = old_mro;  /* transfer the ownership */
+    else
+        Py_XDECREF(old_mro);
+
+    return 1;
 }
 
 
@@ -4675,9 +4793,8 @@
     }
 
     /* Calculate method resolution order */
-    if (mro_internal(type) < 0) {
+    if (mro_internal(type, NULL) < 0)
         goto error;
-    }
 
     /* Inherit special flags from dominant base */
     if (type->tp_base != NULL)
@@ -4826,6 +4943,24 @@
     return result;
 }
 
+static int
+add_all_subclasses(PyTypeObject *type, PyObject *bases)
+{
+    int res = 0;
+
+    if (bases) {
+        Py_ssize_t i;
+        for (i = 0; i < PyTuple_GET_SIZE(bases); i++) {
+            PyObject *base = PyTuple_GET_ITEM(bases, i);
+            if (PyType_Check(base) &&
+                add_subclass((PyTypeObject*)base, type) < 0)
+                res = -1;
+        }
+    }
+
+    return res;
+}
+
 static void
 remove_subclass(PyTypeObject *base, PyTypeObject *type)
 {
@@ -6797,70 +6932,74 @@
 super_getattro(PyObject *self, PyObject *name)
 {
     superobject *su = (superobject *)self;
-    int skip = su->obj_type == NULL;
-
-    if (!skip) {
-        /* We want __class__ to return the class of the super object
-           (i.e. super, or a subclass), not the class of su->obj. */
-        skip = (PyUnicode_Check(name) &&
-                PyUnicode_GET_LENGTH(name) == 9 &&
-                _PyUnicode_CompareWithId(name, &PyId___class__) == 0);
-    }
-
-    if (!skip) {
-        PyObject *mro, *res, *tmp, *dict;
-        PyTypeObject *starttype;
+    PyTypeObject *starttype;
+    PyObject *mro;
+    Py_ssize_t i, n;
+
+    starttype = su->obj_type;
+    if (starttype == NULL)
+        goto skip;
+
+    /* We want __class__ to return the class of the super object
+       (i.e. super, or a subclass), not the class of su->obj. */
+    if (PyUnicode_Check(name) &&
+        PyUnicode_GET_LENGTH(name) == 9 &&
+        _PyUnicode_CompareWithId(name, &PyId___class__) == 0)
+        goto skip;
+
+    mro = starttype->tp_mro;
+    if (mro == NULL)
+        goto skip;
+
+    assert(PyTuple_Check(mro));
+    n = PyTuple_GET_SIZE(mro);
+
+    /* No need to check the last one: it's gonna be skipped anyway.  */
+    for (i = 0; i+1 < n; i++) {
+        if ((PyObject *)(su->type) == PyTuple_GET_ITEM(mro, i))
+            break;
+    }
+    i++;  /* skip su->type (if any)  */
+    if (i >= n)
+        goto skip;
+
+    /* keep a strong reference to mro because starttype->tp_mro can be
+       replaced during PyDict_GetItem(dict, name)  */
+    Py_INCREF(mro);
+    do {
+        PyObject *res, *tmp, *dict;
         descrgetfunc f;
-        Py_ssize_t i, n;
-
-        starttype = su->obj_type;
-        mro = starttype->tp_mro;
-
-        if (mro == NULL)
-            n = 0;
-        else {
-            assert(PyTuple_Check(mro));
-            n = PyTuple_GET_SIZE(mro);
+
+        tmp = PyTuple_GET_ITEM(mro, i);
+        assert(PyType_Check(tmp));
+
+        dict = ((PyTypeObject *)tmp)->tp_dict;
+        assert(dict != NULL && PyDict_Check(dict));
+
+        res = PyDict_GetItem(dict, name);
+        if (res != NULL) {
+            Py_INCREF(res);
+
+            f = Py_TYPE(res)->tp_descr_get;
+            if (f != NULL) {
+                tmp = f(res,
+                    /* Only pass 'obj' param if this is instance-mode super
+                       (See SF ID #743627)  */
+                    (su->obj == (PyObject *)starttype) ? NULL : su->obj,
+                    (PyObject *)starttype);
+                Py_DECREF(res);
+                res = tmp;
+            }
+
+            Py_DECREF(mro);
+            return res;
         }
-        for (i = 0; i < n; i++) {
-            if ((PyObject *)(su->type) == PyTuple_GET_ITEM(mro, i))
-                break;
-        }
+
         i++;
-        res = NULL;
-        /* keep a strong reference to mro because starttype->tp_mro can be
-           replaced during PyDict_GetItem(dict, name)  */
-        Py_INCREF(mro);
-        for (; i < n; i++) {
-            tmp = PyTuple_GET_ITEM(mro, i);
-            if (PyType_Check(tmp))
-                dict = ((PyTypeObject *)tmp)->tp_dict;
-            else
-                continue;
-            res = PyDict_GetItem(dict, name);
-            if (res != NULL) {
-                Py_INCREF(res);
-                f = Py_TYPE(res)->tp_descr_get;
-                if (f != NULL) {
-                    tmp = f(res,
-                        /* Only pass 'obj' param if
-                           this is instance-mode super
-                           (See SF ID #743627)
-                        */
-                        (su->obj == (PyObject *)
-                                    su->obj_type
-                            ? (PyObject *)NULL
-                            : su->obj),
-                        (PyObject *)starttype);
-                    Py_DECREF(res);
-                    res = tmp;
-                }
-                Py_DECREF(mro);
-                return res;
-            }
-        }
-        Py_DECREF(mro);
-    }
+    } while (i < n);
+    Py_DECREF(mro);
+
+  skip:
     return PyObject_GenericGetAttr(self, name);
 }
 

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Fri Feb  6 04:31:01 2015
From: python-checkins at python.org (benjamin.peterson)
Date: Fri, 06 Feb 2015 03:31:01 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_fix_many_custo?=
 =?utf-8?q?m_mro=28=29_edge_cases_and_improve_code_quality_=28=2322735=29?=
Message-ID: <20150206033056.96084.82237@psf.io>

https://hg.python.org/cpython/rev/6384c0cd3b2d
changeset:   94526:6384c0cd3b2d
branch:      3.4
parent:      94524:1ddf68f118c7
user:        Benjamin Peterson <benjamin at python.org>
date:        Thu Feb 05 22:29:14 2015 -0500
summary:
  fix many custom mro() edge cases and improve code quality (#22735)

Patch by Eldar Abusalimov.

files:
  Lib/test/test_descr.py |  227 +++++++++-
  Misc/ACKS              |    1 +
  Misc/NEWS              |    3 +
  Objects/typeobject.c   |  699 +++++++++++++++++-----------
  4 files changed, 649 insertions(+), 281 deletions(-)


diff --git a/Lib/test/test_descr.py b/Lib/test/test_descr.py
--- a/Lib/test/test_descr.py
+++ b/Lib/test/test_descr.py
@@ -4996,11 +4996,236 @@
         self.assertLess(sys.getsizeof(vars(b)), sys.getsizeof({}))
 
 
+class DebugHelperMeta(type):
+    """
+    Sets default __doc__ and simplifies repr() output.
+    """
+    def __new__(mcls, name, bases, attrs):
+        if attrs.get('__doc__') is None:
+            attrs['__doc__'] = name  # helps when debugging with gdb
+        return type.__new__(mcls, name, bases, attrs)
+    def __repr__(cls):
+        return repr(cls.__name__)
+
+
+class MroTest(unittest.TestCase):
+    """
+    Regressions for some bugs revealed through
+    mcsl.mro() customization (typeobject.c: mro_internal()) and
+    cls.__bases__ assignment (typeobject.c: type_set_bases()).
+    """
+
+    def setUp(self):
+        self.step = 0
+        self.ready = False
+
+    def step_until(self, limit):
+        ret = (self.step < limit)
+        if ret:
+            self.step += 1
+        return ret
+
+    def test_incomplete_set_bases_on_self(self):
+        """
+        type_set_bases must be aware that type->tp_mro can be NULL.
+        """
+        class M(DebugHelperMeta):
+            def mro(cls):
+                if self.step_until(1):
+                    assert cls.__mro__ is None
+                    cls.__bases__ += ()
+
+                return type.mro(cls)
+
+        class A(metaclass=M):
+            pass
+
+    def test_reent_set_bases_on_base(self):
+        """
+        Deep reentrancy must not over-decref old_mro.
+        """
+        class M(DebugHelperMeta):
+            def mro(cls):
+                if cls.__mro__ is not None and cls.__name__ == 'B':
+                    # 4-5 steps are usually enough to make it crash somewhere
+                    if self.step_until(10):
+                        A.__bases__ += ()
+
+                return type.mro(cls)
+
+        class A(metaclass=M):
+            pass
+        class B(A):
+            pass
+        B.__bases__ += ()
+
+    def test_reent_set_bases_on_direct_base(self):
+        """
+        Similar to test_reent_set_bases_on_base, but may crash differently.
+        """
+        class M(DebugHelperMeta):
+            def mro(cls):
+                base = cls.__bases__[0]
+                if base is not object:
+                    if self.step_until(5):
+                        base.__bases__ += ()
+
+                return type.mro(cls)
+
+        class A(metaclass=M):
+            pass
+        class B(A):
+            pass
+        class C(B):
+            pass
+
+    def test_reent_set_bases_tp_base_cycle(self):
+        """
+        type_set_bases must check for an inheritance cycle not only through
+        MRO of the type, which may be not yet updated in case of reentrance,
+        but also through tp_base chain, which is assigned before diving into
+        inner calls to mro().
+
+        Otherwise, the following snippet can loop forever:
+            do {
+                // ...
+                type = type->tp_base;
+            } while (type != NULL);
+
+        Functions that rely on tp_base (like solid_base and PyType_IsSubtype)
+        would not be happy in that case, causing a stack overflow.
+        """
+        class M(DebugHelperMeta):
+            def mro(cls):
+                if self.ready:
+                    if cls.__name__ == 'B1':
+                        B2.__bases__ = (B1,)
+                    if cls.__name__ == 'B2':
+                        B1.__bases__ = (B2,)
+                return type.mro(cls)
+
+        class A(metaclass=M):
+            pass
+        class B1(A):
+            pass
+        class B2(A):
+            pass
+
+        self.ready = True
+        with self.assertRaises(TypeError):
+            B1.__bases__ += ()
+
+    def test_tp_subclasses_cycle_in_update_slots(self):
+        """
+        type_set_bases must check for reentrancy upon finishing its job
+        by updating tp_subclasses of old/new bases of the type.
+        Otherwise, an implicit inheritance cycle through tp_subclasses
+        can break functions that recurse on elements of that field
+        (like recurse_down_subclasses and mro_hierarchy) eventually
+        leading to a stack overflow.
+        """
+        class M(DebugHelperMeta):
+            def mro(cls):
+                if self.ready and cls.__name__ == 'C':
+                    self.ready = False
+                    C.__bases__ = (B2,)
+                return type.mro(cls)
+
+        class A(metaclass=M):
+            pass
+        class B1(A):
+            pass
+        class B2(A):
+            pass
+        class C(A):
+            pass
+
+        self.ready = True
+        C.__bases__ = (B1,)
+        B1.__bases__ = (C,)
+
+        self.assertEqual(C.__bases__, (B2,))
+        self.assertEqual(B2.__subclasses__(), [C])
+        self.assertEqual(B1.__subclasses__(), [])
+
+        self.assertEqual(B1.__bases__, (C,))
+        self.assertEqual(C.__subclasses__(), [B1])
+
+    def test_tp_subclasses_cycle_error_return_path(self):
+        """
+        The same as test_tp_subclasses_cycle_in_update_slots, but tests
+        a code path executed on error (goto bail).
+        """
+        class E(Exception):
+            pass
+        class M(DebugHelperMeta):
+            def mro(cls):
+                if self.ready and cls.__name__ == 'C':
+                    if C.__bases__ == (B2,):
+                        self.ready = False
+                    else:
+                        C.__bases__ = (B2,)
+                        raise E
+                return type.mro(cls)
+
+        class A(metaclass=M):
+            pass
+        class B1(A):
+            pass
+        class B2(A):
+            pass
+        class C(A):
+            pass
+
+        self.ready = True
+        with self.assertRaises(E):
+            C.__bases__ = (B1,)
+        B1.__bases__ = (C,)
+
+        self.assertEqual(C.__bases__, (B2,))
+        self.assertEqual(C.__mro__, tuple(type.mro(C)))
+
+    def test_incomplete_extend(self):
+        """
+        Extending an unitialized type with type->tp_mro == NULL must
+        throw a reasonable TypeError exception, instead of failing
+        with PyErr_BadInternalCall.
+        """
+        class M(DebugHelperMeta):
+            def mro(cls):
+                if cls.__mro__ is None and cls.__name__ != 'X':
+                    with self.assertRaises(TypeError):
+                        class X(cls):
+                            pass
+
+                return type.mro(cls)
+
+        class A(metaclass=M):
+            pass
+
+    def test_incomplete_super(self):
+        """
+        Attrubute lookup on a super object must be aware that
+        its target type can be uninitialized (type->tp_mro == NULL).
+        """
+        class M(DebugHelperMeta):
+            def mro(cls):
+                if cls.__mro__ is None:
+                    with self.assertRaises(AttributeError):
+                        super(cls, cls).xxx
+
+                return type.mro(cls)
+
+        class A(metaclass=M):
+            pass
+
+
 def test_main():
     # Run all local test cases, with PTypesLongInitTest first.
     support.run_unittest(PTypesLongInitTest, OperatorsTest,
                          ClassPropertiesAndMethods, DictProxyTests,
-                         MiscTests, PicklingTests, SharedKeyTests)
+                         MiscTests, PicklingTests, SharedKeyTests,
+                         MroTest)
 
 if __name__ == "__main__":
     test_main()
diff --git a/Misc/ACKS b/Misc/ACKS
--- a/Misc/ACKS
+++ b/Misc/ACKS
@@ -16,6 +16,7 @@
 Rajiv Abraham
 David Abrahams
 Marc Abramowitz
+Eldar Abusalimov
 Ron Adam
 Anton Afanasyev
 Ali Afshar
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -11,6 +11,9 @@
 Core and Builtins
 -----------------
 
+- Issue #22735: Fix many edge cases (including crashes) involving custom mro()
+  implementations.
+
 - Issue #22896: Avoid using PyObject_AsCharBuffer(), PyObject_AsReadBuffer()
   and PyObject_AsWriteBuffer().
 
diff --git a/Objects/typeobject.c b/Objects/typeobject.c
--- a/Objects/typeobject.c
+++ b/Objects/typeobject.c
@@ -539,9 +539,11 @@
 }
 
 static PyTypeObject *best_base(PyObject *);
-static int mro_internal(PyTypeObject *);
+static int mro_internal(PyTypeObject *, PyObject **);
+static int type_is_subtype_base_chain(PyTypeObject *, PyTypeObject *);
 static int compatible_for_assignment(PyTypeObject *, PyTypeObject *, char *);
 static int add_subclass(PyTypeObject*, PyTypeObject*);
+static int add_all_subclasses(PyTypeObject *type, PyObject *bases);
 static void remove_subclass(PyTypeObject *, PyTypeObject *);
 static void remove_all_subclasses(PyTypeObject *type, PyObject *bases);
 static void update_all_slots(PyTypeObject *);
@@ -551,167 +553,194 @@
                              update_callback callback, void *data);
 static int recurse_down_subclasses(PyTypeObject *type, PyObject *name,
                                    update_callback callback, void *data);
+static PyObject *type_subclasses(PyTypeObject *type, PyObject *ignored);
 
 static int
-mro_subclasses(PyTypeObject *type, PyObject* temp)
-{
-    PyTypeObject *subclass;
-    PyObject *ref, *subclasses, *old_mro;
+mro_hierarchy(PyTypeObject *type, PyObject *temp)
+{
+    int res;
+    PyObject *new_mro, *old_mro;
+    PyObject *tuple;
+    PyObject *subclasses;
+    Py_ssize_t i, n;
+
+    res = mro_internal(type, &old_mro);
+    if (res <= 0)
+        /* error / reentrance */
+        return res;
+    new_mro = type->tp_mro;
+
+    if (old_mro != NULL)
+        tuple = PyTuple_Pack(3, type, new_mro, old_mro);
+    else
+        tuple = PyTuple_Pack(2, type, new_mro);
+
+    if (tuple != NULL)
+        res = PyList_Append(temp, tuple);
+    else
+        res = -1;
+    Py_XDECREF(tuple);
+
+    if (res < 0) {
+        type->tp_mro = old_mro;
+        Py_DECREF(new_mro);
+        return -1;
+    }
+    Py_XDECREF(old_mro);
+
+    /* Obtain a copy of subclasses list to iterate over.
+
+       Otherwise type->tp_subclasses might be altered
+       in the middle of the loop, for example, through a custom mro(),
+       by invoking type_set_bases on some subclass of the type
+       which in turn calls remove_subclass/add_subclass on this type.
+
+       Finally, this makes things simple avoiding the need to deal
+       with dictionary iterators and weak references.
+    */
+    subclasses = type_subclasses(type, NULL);
+    if (subclasses == NULL)
+        return -1;
+    n = PyList_GET_SIZE(subclasses);
+    for (i = 0; i < n; i++) {
+        PyTypeObject *subclass;
+        subclass = (PyTypeObject *)PyList_GET_ITEM(subclasses, i);
+        res = mro_hierarchy(subclass, temp);
+        if (res < 0)
+            break;
+    }
+    Py_DECREF(subclasses);
+
+    return res;
+}
+
+static int
+type_set_bases(PyTypeObject *type, PyObject *new_bases, void *context)
+{
+    int res = 0;
+    PyObject *temp;
+    PyObject *old_bases;
+    PyTypeObject *new_base, *old_base;
     Py_ssize_t i;
 
-    subclasses = type->tp_subclasses;
-    if (subclasses == NULL)
-        return 0;
-    assert(PyDict_CheckExact(subclasses));
-    i = 0;
-
-    while (PyDict_Next(subclasses, &i, NULL, &ref)) {
-        assert(PyWeakref_CheckRef(ref));
-        subclass = (PyTypeObject *)PyWeakref_GET_OBJECT(ref);
-        assert(subclass != NULL);
-        if ((PyObject *)subclass == Py_None)
-            continue;
-        assert(PyType_Check(subclass));
-        old_mro = subclass->tp_mro;
-        if (mro_internal(subclass) < 0) {
-            subclass->tp_mro = old_mro;
-            return -1;
-        }
-        else {
-            PyObject* tuple;
-            tuple = PyTuple_Pack(2, subclass, old_mro);
-            Py_DECREF(old_mro);
-            if (!tuple)
-                return -1;
-            if (PyList_Append(temp, tuple) < 0)
-                return -1;
-            Py_DECREF(tuple);
-        }
-        if (mro_subclasses(subclass, temp) < 0)
-            return -1;
-    }
-    return 0;
-}
-
-static int
-type_set_bases(PyTypeObject *type, PyObject *value, void *context)
-{
-    Py_ssize_t i;
-    int r = 0;
-    PyObject *ob, *temp;
-    PyTypeObject *new_base, *old_base;
-    PyObject *old_bases, *old_mro;
-
-    if (!check_set_special_type_attr(type, value, "__bases__"))
+    if (!check_set_special_type_attr(type, new_bases, "__bases__"))
         return -1;
-    if (!PyTuple_Check(value)) {
+    if (!PyTuple_Check(new_bases)) {
         PyErr_Format(PyExc_TypeError,
              "can only assign tuple to %s.__bases__, not %s",
-                 type->tp_name, Py_TYPE(value)->tp_name);
+                 type->tp_name, Py_TYPE(new_bases)->tp_name);
         return -1;
     }
-    if (PyTuple_GET_SIZE(value) == 0) {
+    if (PyTuple_GET_SIZE(new_bases) == 0) {
         PyErr_Format(PyExc_TypeError,
              "can only assign non-empty tuple to %s.__bases__, not ()",
                  type->tp_name);
         return -1;
     }
-    for (i = 0; i < PyTuple_GET_SIZE(value); i++) {
-        ob = PyTuple_GET_ITEM(value, i);
+    for (i = 0; i < PyTuple_GET_SIZE(new_bases); i++) {
+        PyObject *ob;
+        PyTypeObject *base;
+
+        ob = PyTuple_GET_ITEM(new_bases, i);
         if (!PyType_Check(ob)) {
             PyErr_Format(PyExc_TypeError,
                          "%s.__bases__ must be tuple of classes, not '%s'",
                          type->tp_name, Py_TYPE(ob)->tp_name);
             return -1;
         }
-        if (PyType_IsSubtype((PyTypeObject*)ob, type)) {
+
+        base = (PyTypeObject*)ob;
+        if (PyType_IsSubtype(base, type) ||
+            /* In case of reentering here again through a custom mro()
+               the above check is not enough since it relies on
+               base->tp_mro which would gonna be updated inside
+               mro_internal only upon returning from the mro().
+
+               However, base->tp_base has already been assigned (see
+               below), which in turn may cause an inheritance cycle
+               through tp_base chain.  And this is definitely
+               not what you want to ever happen.  */
+            (base->tp_mro != NULL && type_is_subtype_base_chain(base, type))) {
+
             PyErr_SetString(PyExc_TypeError,
                             "a __bases__ item causes an inheritance cycle");
             return -1;
         }
     }
 
-    new_base = best_base(value);
-
-    if (!new_base)
+    new_base = best_base(new_bases);
+    if (new_base == NULL)
         return -1;
 
     if (!compatible_for_assignment(type->tp_base, new_base, "__bases__"))
         return -1;
 
+    Py_INCREF(new_bases);
     Py_INCREF(new_base);
-    Py_INCREF(value);
 
     old_bases = type->tp_bases;
     old_base = type->tp_base;
-    old_mro = type->tp_mro;
-
-    type->tp_bases = value;
+
+    type->tp_bases = new_bases;
     type->tp_base = new_base;
 
-    if (mro_internal(type) < 0) {
+    temp = PyList_New(0);
+    if (temp == NULL)
         goto bail;
-    }
-
-    temp = PyList_New(0);
-    if (!temp)
-        goto bail;
-
-    r = mro_subclasses(type, temp);
-
-    if (r < 0) {
-        for (i = 0; i < PyList_Size(temp); i++) {
-            PyTypeObject* cls;
-            PyObject* mro;
-            PyArg_UnpackTuple(PyList_GET_ITEM(temp, i),
-                             "", 2, 2, &cls, &mro);
-            Py_INCREF(mro);
-            ob = cls->tp_mro;
-            cls->tp_mro = mro;
-            Py_DECREF(ob);
-        }
-        Py_DECREF(temp);
-        goto bail;
-    }
-
+    if (mro_hierarchy(type, temp) < 0)
+        goto undo;
     Py_DECREF(temp);
 
-    /* any base that was in __bases__ but now isn't, we
-       need to remove |type| from its tp_subclasses.
-       conversely, any class now in __bases__ that wasn't
-       needs to have |type| added to its subclasses. */
-
-    /* for now, sod that: just remove from all old_bases,
-       add to all new_bases */
-
-    remove_all_subclasses(type, old_bases);
-
-    for (i = PyTuple_GET_SIZE(value) - 1; i >= 0; i--) {
-        ob = PyTuple_GET_ITEM(value, i);
-        if (PyType_Check(ob)) {
-            if (add_subclass((PyTypeObject*)ob, type) < 0)
-                r = -1;
-        }
-    }
-
-    update_all_slots(type);
+    /* Take no action in case if type->tp_bases has been replaced
+       through reentrance.  */
+    if (type->tp_bases == new_bases) {
+        /* any base that was in __bases__ but now isn't, we
+           need to remove |type| from its tp_subclasses.
+           conversely, any class now in __bases__ that wasn't
+           needs to have |type| added to its subclasses. */
+
+        /* for now, sod that: just remove from all old_bases,
+           add to all new_bases */
+        remove_all_subclasses(type, old_bases);
+        res = add_all_subclasses(type, new_bases);
+        update_all_slots(type);
+    }
 
     Py_DECREF(old_bases);
     Py_DECREF(old_base);
-    Py_DECREF(old_mro);
-
-    return r;
+
+    return res;
+
+  undo:
+    for (i = PyList_GET_SIZE(temp) - 1; i >= 0; i--) {
+        PyTypeObject *cls;
+        PyObject *new_mro, *old_mro = NULL;
+
+        PyArg_UnpackTuple(PyList_GET_ITEM(temp, i),
+                          "", 2, 3, &cls, &new_mro, &old_mro);
+        /* Do not rollback if cls has a newer version of MRO.  */
+        if (cls->tp_mro == new_mro) {
+            Py_XINCREF(old_mro);
+            cls->tp_mro = old_mro;
+            Py_DECREF(new_mro);
+        }
+    }
+    Py_DECREF(temp);
 
   bail:
-    Py_DECREF(type->tp_bases);
-    Py_DECREF(type->tp_base);
-    if (type->tp_mro != old_mro) {
-        Py_DECREF(type->tp_mro);
-    }
-
-    type->tp_bases = old_bases;
-    type->tp_base = old_base;
-    type->tp_mro = old_mro;
+    if (type->tp_bases == new_bases) {
+        assert(type->tp_base == new_base);
+
+        type->tp_bases = old_bases;
+        type->tp_base = old_base;
+
+        Py_DECREF(new_bases);
+        Py_DECREF(new_base);
+    }
+    else {
+        Py_DECREF(old_bases);
+        Py_DECREF(old_base);
+    }
 
     return -1;
 }
@@ -1284,6 +1313,18 @@
 
 /* type test with subclassing support */
 
+Py_LOCAL_INLINE(int)
+type_is_subtype_base_chain(PyTypeObject *a, PyTypeObject *b)
+{
+    do {
+        if (a == b)
+            return 1;
+        a = a->tp_base;
+    } while (a != NULL);
+
+    return (b == &PyBaseObject_Type);
+}
+
 int
 PyType_IsSubtype(PyTypeObject *a, PyTypeObject *b)
 {
@@ -1302,15 +1343,9 @@
         }
         return 0;
     }
-    else {
+    else
         /* a is not completely initilized yet; follow tp_base */
-        do {
-            if (a == b)
-                return 1;
-            a = a->tp_base;
-        } while (a != NULL);
-        return b == &PyBaseObject_Type;
-    }
+        return type_is_subtype_base_chain(a, b);
 }
 
 /* Internal routines to do a method lookup in the type
@@ -1577,10 +1612,11 @@
 }
 
 static int
-pmerge(PyObject *acc, PyObject* to_merge) {
+pmerge(PyObject *acc, PyObject* to_merge)
+{
+    int res = 0;
     Py_ssize_t i, j, to_merge_size, empty_cnt;
     int *remain;
-    int ok;
 
     to_merge_size = PyList_GET_SIZE(to_merge);
 
@@ -1618,15 +1654,13 @@
         candidate = PyList_GET_ITEM(cur_list, remain[i]);
         for (j = 0; j < to_merge_size; j++) {
             PyObject *j_lst = PyList_GET_ITEM(to_merge, j);
-            if (tail_contains(j_lst, remain[j], candidate)) {
+            if (tail_contains(j_lst, remain[j], candidate))
                 goto skip; /* continue outer loop */
-            }
         }
-        ok = PyList_Append(acc, candidate);
-        if (ok < 0) {
-            PyMem_FREE(remain);
-            return -1;
-        }
+        res = PyList_Append(acc, candidate);
+        if (res < 0)
+            goto out;
+
         for (j = 0; j < to_merge_size; j++) {
             PyObject *j_lst = PyList_GET_ITEM(to_merge, j);
             if (remain[j] < PyList_GET_SIZE(j_lst) &&
@@ -1638,22 +1672,25 @@
       skip: ;
     }
 
-    if (empty_cnt == to_merge_size) {
-        PyMem_FREE(remain);
-        return 0;
-    }
-    set_mro_error(to_merge, remain);
+    if (empty_cnt != to_merge_size) {
+        set_mro_error(to_merge, remain);
+        res = -1;
+    }
+
+  out:
     PyMem_FREE(remain);
-    return -1;
+
+    return res;
 }
 
 static PyObject *
 mro_implementation(PyTypeObject *type)
 {
+    PyObject *result = NULL;
+    PyObject *bases;
+    PyObject *to_merge, *bases_aslist;
+    int res;
     Py_ssize_t i, n;
-    int ok;
-    PyObject *bases, *result;
-    PyObject *to_merge, *bases_aslist;
 
     if (type->tp_dict == NULL) {
         if (PyType_Ready(type) < 0)
@@ -1677,42 +1714,44 @@
         return NULL;
 
     for (i = 0; i < n; i++) {
-        PyObject *base = PyTuple_GET_ITEM(bases, i);
-        PyObject *parentMRO;
-        parentMRO = PySequence_List(((PyTypeObject*)base)->tp_mro);
-        if (parentMRO == NULL) {
-            Py_DECREF(to_merge);
-            return NULL;
+        PyTypeObject *base;
+        PyObject *base_mro_aslist;
+
+        base = (PyTypeObject *)PyTuple_GET_ITEM(bases, i);
+        if (base->tp_mro == NULL) {
+            PyErr_Format(PyExc_TypeError,
+                         "Cannot extend an incomplete type '%.100s'",
+                         base->tp_name);
+            goto out;
         }
 
-        PyList_SET_ITEM(to_merge, i, parentMRO);
+        base_mro_aslist = PySequence_List(base->tp_mro);
+        if (base_mro_aslist == NULL)
+            goto out;
+
+        PyList_SET_ITEM(to_merge, i, base_mro_aslist);
     }
 
     bases_aslist = PySequence_List(bases);
-    if (bases_aslist == NULL) {
-        Py_DECREF(to_merge);
-        return NULL;
-    }
+    if (bases_aslist == NULL)
+        goto out;
     /* This is just a basic sanity check. */
     if (check_duplicates(bases_aslist) < 0) {
-        Py_DECREF(to_merge);
         Py_DECREF(bases_aslist);
-        return NULL;
+        goto out;
     }
     PyList_SET_ITEM(to_merge, n, bases_aslist);
 
     result = Py_BuildValue("[O]", (PyObject *)type);
-    if (result == NULL) {
-        Py_DECREF(to_merge);
-        return NULL;
-    }
-
-    ok = pmerge(result, to_merge);
+    if (result == NULL)
+        goto out;
+
+    res = pmerge(result, to_merge);
+    if (res < 0)
+        Py_CLEAR(result);
+
+  out:
     Py_DECREF(to_merge);
-    if (ok < 0) {
-        Py_DECREF(result);
-        return NULL;
-    }
 
     return result;
 }
@@ -1726,59 +1765,133 @@
 }
 
 static int
-mro_internal(PyTypeObject *type)
-{
-    PyObject *mro, *result, *tuple;
-    int checkit = 0;
-
-    if (Py_TYPE(type) == &PyType_Type) {
-        result = mro_implementation(type);
+mro_check(PyTypeObject *type, PyObject *mro)
+{
+    PyTypeObject *solid;
+    Py_ssize_t i, n;
+
+    solid = solid_base(type);
+
+    n = PyTuple_GET_SIZE(mro);
+    for (i = 0; i < n; i++) {
+        PyTypeObject *base;
+        PyObject *tmp;
+
+        tmp = PyTuple_GET_ITEM(mro, i);
+        if (!PyType_Check(tmp)) {
+            PyErr_Format(
+                PyExc_TypeError,
+                "mro() returned a non-class ('%.500s')",
+                Py_TYPE(tmp)->tp_name);
+            return -1;
+        }
+
+        base = (PyTypeObject*)tmp;
+        if (!PyType_IsSubtype(solid, solid_base(base))) {
+            PyErr_Format(
+                PyExc_TypeError,
+                "mro() returned base with unsuitable layout ('%.500s')",
+                base->tp_name);
+            return -1;
+        }
+    }
+
+    return 0;
+}
+
+/* Lookups an mcls.mro method, invokes it and checks the result (if needed,
+   in case of a custom mro() implementation).
+
+   Keep in mind that during execution of this function type->tp_mro
+   can be replaced due to possible reentrance (for example,
+   through type_set_bases):
+
+      - when looking up the mcls.mro attribute (it could be
+        a user-provided descriptor);
+
+      - from inside a custom mro() itself;
+
+      - through a finalizer of the return value of mro().
+*/
+static PyObject *
+mro_invoke(PyTypeObject *type)
+{
+    PyObject *mro_result;
+    PyObject *new_mro;
+    int custom = (Py_TYPE(type) != &PyType_Type);
+
+    if (custom) {
+        _Py_IDENTIFIER(mro);
+        PyObject *mro_meth = lookup_method((PyObject *)type, &PyId_mro);
+        if (mro_meth == NULL)
+            return NULL;
+        mro_result = PyObject_CallObject(mro_meth, NULL);
+        Py_DECREF(mro_meth);
     }
     else {
-        _Py_IDENTIFIER(mro);
-        checkit = 1;
-        mro = lookup_method((PyObject *)type, &PyId_mro);
-        if (mro == NULL)
-            return -1;
-        result = PyObject_CallObject(mro, NULL);
-        Py_DECREF(mro);
-    }
-    if (result == NULL)
+        mro_result = mro_implementation(type);
+    }
+    if (mro_result == NULL)
+        return NULL;
+
+    new_mro = PySequence_Tuple(mro_result);
+    Py_DECREF(mro_result);
+    if (new_mro == NULL)
+        return NULL;
+
+    if (custom && mro_check(type, new_mro) < 0) {
+        Py_DECREF(new_mro);
+        return NULL;
+    }
+
+    return new_mro;
+}
+
+/* Calculates and assigns a new MRO to type->tp_mro.
+   Return values and invariants:
+
+     - Returns 1 if a new MRO value has been set to type->tp_mro due to
+       this call of mro_internal (no tricky reentrancy and no errors).
+
+       In case if p_old_mro argument is not NULL, a previous value
+       of type->tp_mro is put there, and the ownership of this
+       reference is transferred to a caller.
+       Otherwise, the previous value (if any) is decref'ed.
+
+     - Returns 0 in case when type->tp_mro gets changed because of
+       reentering here through a custom mro() (see a comment to mro_invoke).
+
+       In this case, a refcount of an old type->tp_mro is adjusted
+       somewhere deeper in the call stack (by the innermost mro_internal
+       or its caller) and may become zero upon returning from here.
+       This also implies that the whole hierarchy of subclasses of the type
+       has seen the new value and updated their MRO accordingly.
+
+     - Returns -1 in case of an error.
+*/
+static int
+mro_internal(PyTypeObject *type, PyObject **p_old_mro)
+{
+    PyObject *new_mro, *old_mro;
+    int reent;
+
+    /* Keep a reference to be able to do a reentrancy check below.
+       Don't let old_mro be GC'ed and its address be reused for
+       another object, like (suddenly!) a new tp_mro.  */
+    old_mro = type->tp_mro;
+    Py_XINCREF(old_mro);
+    new_mro = mro_invoke(type);  /* might cause reentrance */
+    reent = (type->tp_mro != old_mro);
+    Py_XDECREF(old_mro);
+    if (new_mro == NULL)
         return -1;
-    tuple = PySequence_Tuple(result);
-    Py_DECREF(result);
-    if (tuple == NULL)
-        return -1;
-    if (checkit) {
-        Py_ssize_t i, len;
-        PyObject *cls;
-        PyTypeObject *solid;
-
-        solid = solid_base(type);
-
-        len = PyTuple_GET_SIZE(tuple);
-
-        for (i = 0; i < len; i++) {
-            PyTypeObject *t;
-            cls = PyTuple_GET_ITEM(tuple, i);
-            if (!PyType_Check(cls)) {
-                PyErr_Format(PyExc_TypeError,
-                 "mro() returned a non-class ('%.500s')",
-                                 Py_TYPE(cls)->tp_name);
-                Py_DECREF(tuple);
-                return -1;
-            }
-            t = (PyTypeObject*)cls;
-            if (!PyType_IsSubtype(solid, solid_base(t))) {
-                PyErr_Format(PyExc_TypeError,
-             "mro() returned base with unsuitable layout ('%.500s')",
-                                     t->tp_name);
-                        Py_DECREF(tuple);
-                        return -1;
-            }
-        }
-    }
-    type->tp_mro = tuple;
+
+    if (reent) {
+        Py_DECREF(new_mro);
+        return 0;
+    }
+
+    type->tp_mro = new_mro;
 
     type_mro_modified(type, type->tp_mro);
     /* corner case: the super class might have been hidden
@@ -1787,7 +1900,12 @@
 
     PyType_Modified(type);
 
-    return 0;
+    if (p_old_mro != NULL)
+        *p_old_mro = old_mro;  /* transfer the ownership */
+    else
+        Py_XDECREF(old_mro);
+
+    return 1;
 }
 
 
@@ -4661,9 +4779,8 @@
     }
 
     /* Calculate method resolution order */
-    if (mro_internal(type) < 0) {
+    if (mro_internal(type, NULL) < 0)
         goto error;
-    }
 
     /* Inherit special flags from dominant base */
     if (type->tp_base != NULL)
@@ -4812,6 +4929,24 @@
     return result;
 }
 
+static int
+add_all_subclasses(PyTypeObject *type, PyObject *bases)
+{
+    int res = 0;
+
+    if (bases) {
+        Py_ssize_t i;
+        for (i = 0; i < PyTuple_GET_SIZE(bases); i++) {
+            PyObject *base = PyTuple_GET_ITEM(bases, i);
+            if (PyType_Check(base) &&
+                add_subclass((PyTypeObject*)base, type) < 0)
+                res = -1;
+        }
+    }
+
+    return res;
+}
+
 static void
 remove_subclass(PyTypeObject *base, PyTypeObject *type)
 {
@@ -6765,70 +6900,74 @@
 super_getattro(PyObject *self, PyObject *name)
 {
     superobject *su = (superobject *)self;
-    int skip = su->obj_type == NULL;
-
-    if (!skip) {
-        /* We want __class__ to return the class of the super object
-           (i.e. super, or a subclass), not the class of su->obj. */
-        skip = (PyUnicode_Check(name) &&
-                PyUnicode_GET_LENGTH(name) == 9 &&
-                _PyUnicode_CompareWithId(name, &PyId___class__) == 0);
-    }
-
-    if (!skip) {
-        PyObject *mro, *res, *tmp, *dict;
-        PyTypeObject *starttype;
+    PyTypeObject *starttype;
+    PyObject *mro;
+    Py_ssize_t i, n;
+
+    starttype = su->obj_type;
+    if (starttype == NULL)
+        goto skip;
+
+    /* We want __class__ to return the class of the super object
+       (i.e. super, or a subclass), not the class of su->obj. */
+    if (PyUnicode_Check(name) &&
+        PyUnicode_GET_LENGTH(name) == 9 &&
+        _PyUnicode_CompareWithId(name, &PyId___class__) == 0)
+        goto skip;
+
+    mro = starttype->tp_mro;
+    if (mro == NULL)
+        goto skip;
+
+    assert(PyTuple_Check(mro));
+    n = PyTuple_GET_SIZE(mro);
+
+    /* No need to check the last one: it's gonna be skipped anyway.  */
+    for (i = 0; i+1 < n; i++) {
+        if ((PyObject *)(su->type) == PyTuple_GET_ITEM(mro, i))
+            break;
+    }
+    i++;  /* skip su->type (if any)  */
+    if (i >= n)
+        goto skip;
+
+    /* keep a strong reference to mro because starttype->tp_mro can be
+       replaced during PyDict_GetItem(dict, name)  */
+    Py_INCREF(mro);
+    do {
+        PyObject *res, *tmp, *dict;
         descrgetfunc f;
-        Py_ssize_t i, n;
-
-        starttype = su->obj_type;
-        mro = starttype->tp_mro;
-
-        if (mro == NULL)
-            n = 0;
-        else {
-            assert(PyTuple_Check(mro));
-            n = PyTuple_GET_SIZE(mro);
+
+        tmp = PyTuple_GET_ITEM(mro, i);
+        assert(PyType_Check(tmp));
+
+        dict = ((PyTypeObject *)tmp)->tp_dict;
+        assert(dict != NULL && PyDict_Check(dict));
+
+        res = PyDict_GetItem(dict, name);
+        if (res != NULL) {
+            Py_INCREF(res);
+
+            f = Py_TYPE(res)->tp_descr_get;
+            if (f != NULL) {
+                tmp = f(res,
+                    /* Only pass 'obj' param if this is instance-mode super
+                       (See SF ID #743627)  */
+                    (su->obj == (PyObject *)starttype) ? NULL : su->obj,
+                    (PyObject *)starttype);
+                Py_DECREF(res);
+                res = tmp;
+            }
+
+            Py_DECREF(mro);
+            return res;
         }
-        for (i = 0; i < n; i++) {
-            if ((PyObject *)(su->type) == PyTuple_GET_ITEM(mro, i))
-                break;
-        }
+
         i++;
-        res = NULL;
-        /* keep a strong reference to mro because starttype->tp_mro can be
-           replaced during PyDict_GetItem(dict, name)  */
-        Py_INCREF(mro);
-        for (; i < n; i++) {
-            tmp = PyTuple_GET_ITEM(mro, i);
-            if (PyType_Check(tmp))
-                dict = ((PyTypeObject *)tmp)->tp_dict;
-            else
-                continue;
-            res = PyDict_GetItem(dict, name);
-            if (res != NULL) {
-                Py_INCREF(res);
-                f = Py_TYPE(res)->tp_descr_get;
-                if (f != NULL) {
-                    tmp = f(res,
-                        /* Only pass 'obj' param if
-                           this is instance-mode super
-                           (See SF ID #743627)
-                        */
-                        (su->obj == (PyObject *)
-                                    su->obj_type
-                            ? (PyObject *)NULL
-                            : su->obj),
-                        (PyObject *)starttype);
-                    Py_DECREF(res);
-                    res = tmp;
-                }
-                Py_DECREF(mro);
-                return res;
-            }
-        }
-        Py_DECREF(mro);
-    }
+    } while (i < n);
+    Py_DECREF(mro);
+
+  skip:
     return PyObject_GenericGetAttr(self, name);
 }
 

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Fri Feb  6 05:43:37 2015
From: python-checkins at python.org (ned.deily)
Date: Fri, 06 Feb 2015 04:43:37 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=283=2E4=29=3A_Fix_missing_?=
 =?utf-8?q?=3Aref=3A_for_idle_in_doc_build=2E?=
Message-ID: <20150206044337.25859.1280@psf.io>

https://hg.python.org/cpython/rev/b8078220a6b8
changeset:   94529:b8078220a6b8
branch:      3.4
parent:      94526:6384c0cd3b2d
user:        Ned Deily <nad at acm.org>
date:        Fri Feb 06 15:42:06 2015 +1100
summary:
  Fix missing :ref: for idle in doc build.

files:
  Doc/library/idle.rst |  6 +++---
  1 files changed, 3 insertions(+), 3 deletions(-)


diff --git a/Doc/library/idle.rst b/Doc/library/idle.rst
--- a/Doc/library/idle.rst
+++ b/Doc/library/idle.rst
@@ -1,13 +1,13 @@
 .. _idle:
 
+IDLE
+====
+
 .. index::
    single: IDLE
    single: Python Editor
    single: Integrated Development Environment
 
-IDLE
-====
-
 .. moduleauthor:: Guido van Rossum <guido at Python.org>
 
 IDLE is the Python IDE built with the :mod:`tkinter` GUI toolkit.

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Fri Feb  6 05:43:37 2015
From: python-checkins at python.org (ned.deily)
Date: Fri, 06 Feb 2015 04:43:37 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=282=2E7=29=3A_Fix_missing_?=
 =?utf-8?q?=3Aref=3A_for_idle_in_doc_build=2E?=
Message-ID: <20150206044336.34402.21545@psf.io>

https://hg.python.org/cpython/rev/b3df5bb1cd14
changeset:   94528:b3df5bb1cd14
branch:      2.7
parent:      94523:7b74c65758ca
user:        Ned Deily <nad at acm.org>
date:        Fri Feb 06 15:41:27 2015 +1100
summary:
  Fix missing :ref: for idle in doc build.

files:
  Doc/library/idle.rst |  6 +++---
  1 files changed, 3 insertions(+), 3 deletions(-)


diff --git a/Doc/library/idle.rst b/Doc/library/idle.rst
--- a/Doc/library/idle.rst
+++ b/Doc/library/idle.rst
@@ -1,13 +1,13 @@
 .. _idle:
 
+IDLE
+====
+
 .. index::
    single: IDLE
    single: Python Editor
    single: Integrated Development Environment
 
-IDLE
-====
-
 .. moduleauthor:: Guido van Rossum <guido at Python.org>
 
 IDLE is the Python IDE built with the :mod:`tkinter` GUI toolkit.

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Fri Feb  6 05:43:37 2015
From: python-checkins at python.org (ned.deily)
Date: Fri, 06 Feb 2015 04:43:37 +0000
Subject: [Python-checkins] =?utf-8?q?cpython_=28merge_3=2E4_-=3E_default?=
 =?utf-8?q?=29=3A_Fix_missing_=3Aref=3A_for_idle_in_doc_build=2E?=
Message-ID: <20150206044337.39290.67069@psf.io>

https://hg.python.org/cpython/rev/f1a82e949fb8
changeset:   94530:f1a82e949fb8
parent:      94527:75fd0bd89eef
parent:      94529:b8078220a6b8
user:        Ned Deily <nad at acm.org>
date:        Fri Feb 06 15:43:01 2015 +1100
summary:
  Fix missing :ref: for idle in doc build.

files:
  Doc/library/idle.rst |  6 +++---
  1 files changed, 3 insertions(+), 3 deletions(-)


diff --git a/Doc/library/idle.rst b/Doc/library/idle.rst
--- a/Doc/library/idle.rst
+++ b/Doc/library/idle.rst
@@ -1,13 +1,13 @@
 .. _idle:
 
+IDLE
+====
+
 .. index::
    single: IDLE
    single: Python Editor
    single: Integrated Development Environment
 
-IDLE
-====
-
 .. moduleauthor:: Guido van Rossum <guido at Python.org>
 
 IDLE is the Python IDE built with the :mod:`tkinter` GUI toolkit.

-- 
Repository URL: https://hg.python.org/cpython

From python-checkins at python.org  Fri Feb  6 07:12:43 2015
From: python-checkins at python.org (steve.dower)
Date: Fri, 06 Feb 2015 06:12:43 +0000
Subject: [Python-checkins] =?utf-8?q?cpython=3A_Issue_=2323260=3A_Update_W?=
	=?utf-8?q?indows_installer?=
Message-ID: <20150206061243.25855.57060@psf.io>

https://hg.python.org/cpython/rev/e7dbef447157
changeset:   94531:e7dbef447157
user:        Steve Dower <steve.dower at microsoft.com>
date:        Thu Feb 05 22:08:48 2015 -0800
summary:
  Issue #23260: Update Windows installer

files:
  .gitignore                                                   |     1 +
  .hgignore                                                    |     1 +
  Doc/make.bat                                                 |    15 +-
  Doc/using/win_installer.png                                  |   Bin 
  Doc/using/windows.rst                                        |   505 +-
  Misc/NEWS                                                    |     2 +
  PC/launcher.c                                                |    20 +-
  PCbuild/_freeze_importlib.vcxproj                            |     9 +-
  PCbuild/libeay.vcxproj                                       |     6 +-
  PCbuild/pylauncher.vcxproj                                   |     1 +
  PCbuild/pyproject.props                                      |    15 +
  PCbuild/python.props                                         |     8 +-
  PCbuild/pywlauncher.vcxproj                                  |     1 +
  PCbuild/ssleay.vcxproj                                       |     6 +-
  Tools/buildbot/buildmsi.bat                                  |    20 +-
  Tools/msi/README.txt                                         |   501 +-
  Tools/msi/build.bat                                          |    46 +
  Tools/msi/buildrelease.bat                                   |   131 +
  Tools/msi/bundle/Default.thm                                 |   123 +
  Tools/msi/bundle/Default.wxl                                 |   103 +
  Tools/msi/bundle/SideBar.png                                 |   Bin 
  Tools/msi/bundle/bootstrap/LICENSE.txt                       |    25 +
  Tools/msi/bundle/bootstrap/PythonBootstrapperApplication.cpp |  2645 ++++++++++
  Tools/msi/bundle/bootstrap/pch.cpp                           |     1 +
  Tools/msi/bundle/bootstrap/pch.h                             |    59 +
  Tools/msi/bundle/bootstrap/pythonba.cpp                      |    76 +
  Tools/msi/bundle/bootstrap/pythonba.def                      |    18 +
  Tools/msi/bundle/bootstrap/pythonba.sln                      |    22 +
  Tools/msi/bundle/bootstrap/pythonba.vcxproj                  |    69 +
  Tools/msi/bundle/bootstrap/resource.h                        |    25 +
  Tools/msi/bundle/bundle.ico                                  |   Bin 
  Tools/msi/bundle/bundle.targets                              |    99 +
  Tools/msi/bundle/bundle.wxs                                  |    80 +
  Tools/msi/bundle/full.wixproj                                |    21 +
  Tools/msi/bundle/packagegroups/core.wxs                      |    56 +
  Tools/msi/bundle/packagegroups/crt.wxs                       |    25 +
  Tools/msi/bundle/packagegroups/dev.wxs                       |    40 +
  Tools/msi/bundle/packagegroups/doc.wxs                       |    24 +
  Tools/msi/bundle/packagegroups/exe.wxs                       |    56 +
  Tools/msi/bundle/packagegroups/launcher.wxs                  |    23 +
  Tools/msi/bundle/packagegroups/lib.wxs                       |    56 +
  Tools/msi/bundle/packagegroups/postinstall.wxs               |    62 +
  Tools/msi/bundle/packagegroups/tcltk.wxs                     |    62 +
  Tools/msi/bundle/packagegroups/test.wxs                      |    56 +
  Tools/msi/bundle/packagegroups/tools.wxs                     |    24 +
  Tools/msi/bundle/postinstall_en-US.wxl_template              |     4 +
  Tools/msi/bundle/releaselocal.wixproj                        |    21 +
  Tools/msi/bundle/releaseweb.wixproj                          |    21 +
  Tools/msi/bundle/snapshot.wixproj                            |    21 +
  Tools/msi/common.wxs                                         |   101 +
  Tools/msi/common_en-US.wxl_template                          |    17 +
  Tools/msi/core/core.props                                    |    12 +
  Tools/msi/core/core.wixproj                                  |    11 +
  Tools/msi/core/core.wxs                                      |    25 +
  Tools/msi/core/core_d.wixproj                                |    11 +
  Tools/msi/core/core_en-US.wxl                                |     5 +
  Tools/msi/core/core_files.wxs                                |    31 +
  Tools/msi/core/core_pdb.wixproj                              |    11 +
  Tools/msi/crt/crt.wixproj                                    |    20 +
  Tools/msi/crt/crt.wxs                                        |    26 +
  Tools/msi/crt/crt_en-US.wxl                                  |     5 +
  Tools/msi/crt/crt_files.12.0.wxs                             |    20 +
  Tools/msi/crt/crt_files.14.0.wxs                             |    40 +
  Tools/msi/crt/crtlicense.txt                                 |    47 +
  Tools/msi/crtlicense.txt                                     |    44 -
  Tools/msi/csv_to_wxs.py                                      |   127 +
  Tools/msi/dev/dev.props                                      |    42 +
  Tools/msi/dev/dev.wixproj                                    |    11 +
  Tools/msi/dev/dev.wxs                                        |    25 +
  Tools/msi/dev/dev_d.wixproj                                  |    11 +
  Tools/msi/dev/dev_en-US.wxl                                  |     5 +
  Tools/msi/dev/dev_files.wxs                                  |    42 +
  Tools/msi/doc/doc.wixproj                                    |    30 +
  Tools/msi/doc/doc.wxs                                        |    26 +
  Tools/msi/doc/doc_en-US.wxl_template                         |     7 +
  Tools/msi/doc/doc_files.wxs                                  |    12 +
  Tools/msi/doc/doc_no_files.wxs                               |    17 +
  Tools/msi/exe/crtlicense.txt                                 |    44 +
  Tools/msi/exe/exe.props                                      |    36 +
  Tools/msi/exe/exe.wixproj                                    |    11 +
  Tools/msi/exe/exe.wxs                                        |    40 +
  Tools/msi/exe/exe_d.wixproj                                  |    11 +
  Tools/msi/exe/exe_en-US.wxl_template                         |     9 +
  Tools/msi/exe/exe_files.wxs                                  |    68 +
  Tools/msi/exe/exe_pdb.wixproj                                |    11 +
  Tools/msi/get_wix.py                                         |    49 +
  Tools/msi/launcher/launcher.props                            |    12 +
  Tools/msi/launcher/launcher.wixproj                          |    11 +
  Tools/msi/launcher/launcher.wxs                              |    38 +
  Tools/msi/launcher/launcher_en-US.wxl                        |     8 +
  Tools/msi/launcher/launcher_files.wxs                        |    35 +
  Tools/msi/launcher/launcher_pdb.wixproj                      |    11 +
  Tools/msi/launcher/launcher_reg.wxs                          |    32 +
  Tools/msi/lib/lib.props                                      |    27 +
  Tools/msi/lib/lib.wixproj                                    |    11 +
  Tools/msi/lib/lib.wxs                                        |    28 +
  Tools/msi/lib/lib_d.wixproj                                  |    11 +
  Tools/msi/lib/lib_en-US.wxl                                  |     5 +
  Tools/msi/lib/lib_files.wxs                                  |    72 +
  Tools/msi/lib/lib_pdb.wixproj                                |    11 +
  Tools/msi/msi.props                                          |   161 +
  Tools/msi/msi.py                                             |  1454 -----
  Tools/msi/msi.targets                                        |    62 +
  Tools/msi/msilib.py                                          |   679 --
  Tools/msi/msisupport.c                                       |    93 -
  Tools/msi/msisupport.mak                                     |     9 -
  Tools/msi/path/path.wixproj                                  |    19 +
  Tools/msi/path/path.wxs                                      |    35 +
  Tools/msi/path/path_en-US.wxl                                |     6 +
  Tools/msi/pip/pip.wixproj                                    |    19 +
  Tools/msi/pip/pip.wxs                                        |    41 +
  Tools/msi/pip/pip_en-US.wxl                                  |     6 +
  Tools/msi/schema.py                                          |  1007 ---
  Tools/msi/sequence.py                                        |   126 -
  Tools/msi/tcltk/tcltk.props                                  |    49 +
  Tools/msi/tcltk/tcltk.wixproj                                |    11 +
  Tools/msi/tcltk/tcltk.wxs                                    |    53 +
  Tools/msi/tcltk/tcltk_d.wixproj                              |    11 +
  Tools/msi/tcltk/tcltk_en-US.wxl_template                     |     8 +
  Tools/msi/tcltk/tcltk_files.wxs                              |    35 +
  Tools/msi/tcltk/tcltk_pdb.wixproj                            |    11 +
  Tools/msi/test/test.props                                    |    22 +
  Tools/msi/test/test.wixproj                                  |    11 +
  Tools/msi/test/test.wxs                                      |    27 +
  Tools/msi/test/test_d.wixproj                                |    11 +
  Tools/msi/test/test_en-US.wxl                                |     7 +
  Tools/msi/test/test_files.wxs                                |    65 +
  Tools/msi/test/test_pdb.wixproj                              |    11 +
  Tools/msi/tools/tools.wixproj                                |    42 +
  Tools/msi/tools/tools.wxs                                    |    14 +
  Tools/msi/tools/tools_en-US.wxl                              |     5 +
  Tools/msi/tools/tools_files.wxs                              |    16 +
  Tools/msi/uisample.py                                        |  1400 -----
  Tools/msi/wix.props                                          |    12 +
  134 files changed, 7080 insertions(+), 5035 deletions(-)


diff --git a/.gitignore b/.gitignore
--- a/.gitignore
+++ b/.gitignore
@@ -82,5 +82,6 @@
 coverage/
 externals/
 htmlcov/
+Tools/msi/obj
 Tools/ssl/amd64
 Tools/ssl/win32
diff --git a/.hgignore b/.hgignore
--- a/.hgignore
+++ b/.hgignore
@@ -90,5 +90,6 @@
 *.gcno
 *.gcov
 coverage.info
+Tools/msi/obj
 Tools/ssl/amd64
 Tools/ssl/win32
diff --git a/Doc/make.bat b/Doc/make.bat
--- a/Doc/make.bat
+++ b/Doc/make.bat
@@ -8,11 +8,15 @@
 if "%SPHINXBUILD%" EQU "" set SPHINXBUILD=sphinx-build
 if "%PYTHON%" EQU "" set PYTHON=py
 
-if DEFINED ProgramFiles(x86) set _PRGMFLS=%ProgramFiles(x86)%
-if NOT DEFINED ProgramFiles(x86) set _PRGMFLS=%ProgramFiles%
-if "%HTMLHELP%" EQU "" set HTMLHELP=%_PRGMFLS%\HTML Help Workshop\hhc.exe
+if "%HTMLHELP%" EQU "" (
+    where hhc 2>nul >"%TEMP%\hhc.loc"
+    if errorlevel 1 dir "..\externals\hhc.exe" /s/b > "%TEMP%\hhc.loc"
+    if errorlevel 1 echo Cannot find HHC on PATH or in externals & exit /B 1
+    set /P HTMLHELP= < "%TEMP%\hhc.loc"
+    del "%TEMP%\hhc.loc"
+)
 
-if "%DISTVERSION%" EQU "" for /f "usebackq" %%v in (`%PYTHON% tools/patchlevel.py`) do set DISTVERSION=%%v
+if "%DISTVERSION%" EQU "" for /f "usebackq" %%v in (`%PYTHON% tools/extensions/patchlevel.py`) do set DISTVERSION=%%v
 
 if "%BUILDDIR%" EQU "" set BUILDDIR=build
 
@@ -36,7 +40,8 @@
     echo.
     echo.If you don't have Sphinx installed, grab it from
     echo.http://sphinx-doc.org/
-    goto end
+    popd
+    exit /B 1
 )
 
 rem Targets that do require sphinx-build and have their own label
diff --git a/Doc/using/win_installer.png b/Doc/using/win_installer.png
new file mode 100644
index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..4696bb2a400eb1c7ab18b48e9eda9a004e08749f
GIT binary patch
[stripped]
diff --git a/Doc/using/windows.rst b/Doc/using/windows.rst
--- a/Doc/using/windows.rst
+++ b/Doc/using/windows.rst
@@ -7,6 +7,7 @@
 *************************
 
 .. sectionauthor:: Robert Lehmann <lehmannro at gmail.com>
+.. sectionauthor:: Steve Dower <steve.dower at microsoft.com>
 
 This document aims to give an overview of Windows-specific behaviour you should
 know about when using Python on Microsoft Windows.
@@ -19,10 +20,172 @@
 Installing Python
 =================
 
-Unlike most Unix systems and services, Windows does not require Python natively
-and thus does not pre-install a version of Python.  However, the CPython team
+Unlike most Unix systems and services, Windows does not include a system
+supported installation of Python. To make Python available, the CPython team
 has compiled Windows installers (MSI packages) with every `release
-<https://www.python.org/download/releases/>`_ for many years.
+<https://www.python.org/download/releases/>`_ for many years. These installers
+are primarily intended to add a system-wide installation of Python, with the
+core interpreter and library being shared by all application. Non-shared
+layouts of the Python interpreter may also be created with the same installer,
+however, the released installer is not intended for embedding in other
+installers.
+
+Installation Steps
+------------------
+
+Four Python 3.5 installers are available for download - two each for the 32-bit
+and 64-bit versions of the interpreter. The *web installer* is a small initial
+download, and it will automatically download the required components as
+necessary. The *offline installer* includes the components necessary for a
+default installation and only requires an internet connection for optional
+features. See :ref:`install-layout-option` for other ways to avoid downloading
+during installation.
+
+After starting the installer, one of three options may be selected:
+
+.. image:: win_installer.png
+
+If you select "Install for All Users":
+
+* You may be required to provide administrative credentials or approval
+* Python will be installed into your Program Files directory
+* The :ref:`launcher` will be installed into your Windows directory
+* The standard library, test suite, launcher and pip will be installed
+* After installation, the standard library will be pre-compiled to bytecode
+* If selected, the install directory will be added to :envvar:`PATH`
+
+If you select "Install Just for Me":
+
+* You will *not* need to be an administrator
+* Python will be installed into your user directory
+* The :ref:`launcher` will *also* be installed into your user directory
+* The standard library, test suite, launcher and pip will be installed
+* If selected, the install directory will be added to :envvar:`PATH`
+
+Selecting "Customize installation" will allow you to select the features to
+install, the installation location and other options or post-install actions.
+To install debugging symbols or binaries, you will need to use this option.
+
+.. _install-quiet-option:
+
+Installing Without UI
+---------------------
+
+All of the options available in the installer UI can also be specified from the
+command line, allowing scripted installers to replicate an installation on many
+machines without user interaction.  These options may also be set without
+suppressing the UI in order to change some of the defaults.
+
+To completely hide the installer UI and install Python silently, pass the
+``/quiet`` (``/q``) option. To skip past the user interaction but still display
+progress and errors, pass the ``/passive`` (``/p``) option. The ``/uninstall``
+option may be passed to immediately begin removing Python - no prompt will be
+displayed.
+
+All other options are passed as ``name=value``, where the value is usually
+``0`` to disable a feature, ``1`` to enable a feature, or a path. The full list
+of available options is shown below.
+
++---------------------------+--------------------------------------+--------------------------+
+| Name                      | Description                          | Default                  |
++===========================+======================================+==========================+
+| InstallAllUsers           | Perform a system-wide installation.  | 1                        |
++---------------------------+--------------------------------------+--------------------------+
+| TargetDir                 | The installation directory           | Selected based on        |
+|                           |                                      | InstallAllUsers          |
++---------------------------+--------------------------------------+--------------------------+
+| DefaultAllUsersTargetDir  | The default installation directory   | :file:`%ProgramFiles%\\\ |
+|                           | for all-user installs                | Python X.Y` or :file:`\  |
+|                           |                                      | %ProgramFiles(x86)%\\\   |
+|                           |                                      | Python X.Y`              |
++---------------------------+--------------------------------------+--------------------------+
+| DefaultJustForMeTargetDir | The default install directory for    | :file:`%LocalAppData%\\\ |
+|                           | just-for-me installs                 | Programs\\PythonXY` or   |
+|                           |                                      | :file:`%LocalAppData%\\\ |
+|                           |                                      | Programs\\PythonXY-32`   |
++---------------------------+--------------------------------------+--------------------------+
+| DefaultCustomTargetDir    | The default custom install directory | (empty)                  |
+|                           | displayed in the UI                  |                          |
++---------------------------+--------------------------------------+--------------------------+
+| AssociateFiles            | Create file associations if the      | 1                        |
+|                           | launcher is also installed.          |                          |
++---------------------------+--------------------------------------+--------------------------+
+| CompileAll                | Compile all ``.py`` files to         | 0                        |
+|                           | ``.pyc`` and ``.pyo``.               |                          |
++---------------------------+--------------------------------------+--------------------------+
+| PrependPath               | Add install and Scripts directories  | 0                        |
+|                           | tho :envvar:`PATH` and ``.PY`` to    |                          |
+|                           | :envvar:`PATHEXT`                    |                          |
++---------------------------+--------------------------------------+--------------------------+
+| Include_doc               | Install Python manual                | 1                        |
++---------------------------+--------------------------------------+--------------------------+
+| Include_debug             | Install debug binaries               | 0                        |
++---------------------------+--------------------------------------+--------------------------+
+| Include_dev               | Install developer headers and        | 1                        |
+|                           | libraries                            |                          |
++---------------------------+--------------------------------------+--------------------------+
+| Include_exe               | Install :file:`python.exe` and       | 1                        |
+|                           | related files                        |                          |
++---------------------------+--------------------------------------+--------------------------+
+| Include_launcher          | Install :ref:`launcher`.             | 1                        |
++---------------------------+--------------------------------------+--------------------------+
+| Include_lib               | Install standard library and         | 1                        |
+|                           | extension modules                    |                          |
++---------------------------+--------------------------------------+--------------------------+
+| Include_pip               | Install bundled pip and setuptools   | 1                        |
++---------------------------+--------------------------------------+--------------------------+
+| Include_symbols           | Install debugging symbols (`*`.pdb)  | 0                        |
++---------------------------+--------------------------------------+--------------------------+
+| Include_tcltk             | Install Tcl/Tk support and IDLE      | 1                        |
++---------------------------+--------------------------------------+--------------------------+
+| Include_test              | Install standard library test suite  | 1                        |
++---------------------------+--------------------------------------+--------------------------+
+| Include_tools             | Install utility scripts              | 1                        |
++---------------------------+--------------------------------------+--------------------------+
+| SimpleInstall             | Disable most install UI              | 0                        |
++---------------------------+--------------------------------------+--------------------------+
+
+For example, to silently install a default, system-wide Python installation,
+you could use the following command (from an elevated command prompt)::
+
+    python-3.5.0.exe /quiet InstallAllUsers=1 PrependPath=1 Include_test=0
+
+To allow users to easily install a personal copy of Python without the test
+suite, you could provide a shortcut with the following command::
+
+    python-3.5.0.exe /passive InstallAllUsers=0 Include_launcher=0 Include_test=0 SimpleInstall=1
+
+(Note that omitting the launcher also omits file associations, and is only
+recommended for per-user installs when there is also a system-wide installation
+that included the launcher.)
+
+.. _install-layout-option:
+
+Installing Without Downloading
+------------------------------
+
+As some features of Python are not included in the initial installer download,
+selecting those features may require an internet connection.  To avoid this
+need, all possible components may be downloaded on-demand to create a complete
+*layout* that will no longer require an internet connection regardless of the
+selected features. Note that this download may be bigger than required, but
+where a large number of installations are going to be performed it is very
+useful to have a locally cached copy.
+
+Execute the following command from Command Prompt to download all possible
+required files.  Remember to substitute ``python-3.5.0.exe`` for the actual
+name of your installer, and to create layouts in their own directories to
+avoid collisions between files with the same name.
+
+::
+
+    python-3.5.0.exe /layout [optional target directory]
+
+You may also specify the ``/quiet`` option to hide the progress display.
+
+
+Other Platforms
+---------------
 
 With ongoing development of Python, some platforms that used to be supported
 earlier are no longer supported (due to the lack of users or developers).
@@ -66,19 +229,31 @@
 `ActivePython <http://www.activestate.com/activepython/>`_
     Installer with multi-platform compatibility, documentation, PyWin32
 
-`Enthought Python Distribution <https://www.enthought.com/products/epd/>`_
-    Popular modules (such as PyWin32) with their respective documentation, tool
-    suite for building extensible Python applications
+`Anaconda <http://www.continuum.io/downloads/>`_
+    Popular scientific modules (such as numpy, scipy and pandas) and the
+    ``conda`` package manager.
 
-Notice that these packages are likely to install *older* versions of Python.
+`Canopy <https://www.enthought.com/products/canopy/>`_
+    A "comprehensive Python analysis environment" with editors and other
+    development tools.
+
+`WinPython <https://winpython.github.io/>`_
+    Windows-specific distribution with prebuilt scientific packages and
+    tools for building packages.
+
+Note that these packages may not include the latest versions of Python or
+other libraries, and are not maintained or supported by the core Python team.
 
 
 
 Configuring Python
 ==================
 
-In order to run Python flawlessly, you might have to change certain environment
-settings in Windows.
+To run Python conveniently from a command prompt, you might consider changing
+some default environment variables in Windows.  While the installer provides an
+option to configure the PATH and PATHEXT variables for you, this is only
+reliable for a single, system-wide installation.  If you regularly use multiple
+versions of Python, consider using the :ref:`launcher`.
 
 
 .. _setting-envvars:
@@ -86,163 +261,86 @@
 Excursus: Setting environment variables
 ---------------------------------------
 
-Windows has a built-in dialog for changing environment variables (following
-guide applies to XP classical view): Right-click the icon for your machine
-(usually located on your Desktop and called "My Computer") and choose
-:menuselection:`Properties` there.  Then, open the :guilabel:`Advanced` tab
-and click the :guilabel:`Environment Variables` button.
+Windows allows environment variables to be configured permanently at both the
+User level and the System level, or temporarily in a command prompt.
 
-In short, your path is:
+To temporarily set environment variables, open Command Prompt and use the
+:command:`set` command::
 
-    :menuselection:`My Computer
-    --> Properties
-    --> Advanced
-    --> Environment Variables`
+    C:\>set PATH=C:\Program Files\Python 3.5;%PATH%
+    C:\>set PYTHONPATH=%PYTHONPATH%;C:\My_python_lib
+    C:\>python
 
+These changes will apply to any further commands executed in that console, and
+will be inherited by any applications started from the console.
+
+Including the variable name within percent signs will expand to the existing
+value, allowing you to add your new value at either the start or the end.
+Modifying :envvar:`PATH` by adding the directory containing
+:program:`python.exe` to the start is a common way to ensure the correct version
+of Python is launched.
+
+To permanently modify the default environment variables, click Start and search
+for 'edit environment variables', or open System properties, :guilabel:`Advanced
+system settings` and click the :guilabel:`Environment Variables` button.
 In this dialog, you can add or modify User and System variables. To change
 System variables, you need non-restricted access to your machine
 (i.e. Administrator rights).
 
-Another way of adding variables to your environment is using the :command:`set`
-command::
+.. note::
 
-    set PYTHONPATH=%PYTHONPATH%;C:\My_python_lib
+    Windows will concatenate User variables *after* System variables, which may
+    cause unexpected results when modifying :envvar:`PATH`.
 
-To make this setting permanent, you could add the corresponding command line to
-your :file:`autoexec.bat`. :program:`msconfig` is a graphical interface to this
-file.
-
-Viewing environment variables can also be done more straight-forward: The
-command prompt will expand strings wrapped into percent signs automatically::
-
-    echo %PATH%
-
-Consult :command:`set /?` for details on this behaviour.
+    The :envvar:`PYTHONPATH` variable is used by all versions of Python 2 and
+    Python 3, so you should not permanently configure this variable unless it
+    only includes code that is compatible all of your installed Python
+    versions.
 
 .. seealso::
 
-   http://support.microsoft.com/kb/100843
+    http://support.microsoft.com/kb/100843
       Environment variables in Windows NT
 
-   http://support.microsoft.com/kb/310519
+    http://technet.microsoft.com/en-us/library/cc754250.aspx
+      The SET command, for temporarily modifying environment variables
+
+    http://technet.microsoft.com/en-us/library/cc755104.aspx
+      The SETX command, for permanently modifying environment variables
+
+    http://support.microsoft.com/kb/310519
       How To Manage Environment Variables in Windows XP
 
-   http://www.chem.gla.ac.uk/~louis/software/faq/q1.html
+    http://www.chem.gla.ac.uk/~louis/software/faq/q1.html
       Setting Environment variables, Louis J. Farrugia
 
-
 .. _windows-path-mod:
 
 Finding the Python executable
 -----------------------------
 
-.. versionchanged:: 3.3
+.. versionchanged:: 3.5
 
 Besides using the automatically created start menu entry for the Python
-interpreter, you might want to start Python in the command prompt. As of
-Python 3.3, the installer has an option to set that up for you.
+interpreter, you might want to start Python in the command prompt. The
+installer for Python 3.5 and later has an option to set that up for you.
 
-At the "Customize Python 3.3" screen, an option called
-"Add python.exe to search path" can be enabled to have the installer place
-your installation into the :envvar:`%PATH%`. This allows you to type
-:command:`python` to run the interpreter. Thus, you can also execute your
+On the first page of the installer, an option labelled "Add Python 3.5 to
+PATH" can be selected to have the installer add the install location into the
+:envvar:`PATH`.  The location of the :file:`Scripts\\` folder is also added.
+This allows you to type :command:`python` to run the interpreter, and
+:command:`pip` or . Thus, you can also execute your
 scripts with command line options, see :ref:`using-on-cmdline` documentation.
 
 If you don't enable this option at install time, you can always re-run the
-installer to choose it.
+installer, select Modify, and enable it.  Alternatively, you can manually
+modify the :envvar:`PATH` using the directions in :ref:`setting-envvars`.  You
+need to set your :envvar:`PATH` environment variable to include the directory
+of your Python installation, delimited by a semicolon from other entries.  An
+example variable could look like this (assuming the first two entries already
+existed)::
 
-The alternative is manually modifying the :envvar:`%PATH%` using the
-directions in :ref:`setting-envvars`. You need to set your :envvar:`%PATH%`
-environment variable to include the directory of your Python distribution,
-delimited by a semicolon from other entries. An example variable could look
-like this (assuming the first two entries are Windows' default)::
-
-    C:\WINDOWS\system32;C:\WINDOWS;C:\Python33
-
-
-Finding modules
----------------
-
-Python usually stores its library (and thereby your site-packages folder) in the
-installation directory.  So, if you had installed Python to
-:file:`C:\\Python\\`, the default library would reside in
-:file:`C:\\Python\\Lib\\` and third-party modules should be stored in
-:file:`C:\\Python\\Lib\\site-packages\\`.
-
-This is how :data:`sys.path` is populated on Windows:
-
-* An empty entry is added at the start, which corresponds to the current
-  directory.
-
-* If the environment variable :envvar:`PYTHONPATH` exists, as described in
-  :ref:`using-on-envvars`, its entries are added next.  Note that on Windows,
-  paths in this variable must be separated by semicolons, to distinguish them
-  from the colon used in drive identifiers (``C:\`` etc.).
-
-* Additional "application paths" can be added in the registry as subkeys of
-  :samp:`\\SOFTWARE\\Python\\PythonCore\\{version}\\PythonPath` under both the
-  ``HKEY_CURRENT_USER`` and ``HKEY_LOCAL_MACHINE`` hives.  Subkeys which have
-  semicolon-delimited path strings as their default value will cause each path
-  to be added to :data:`sys.path`.  (Note that all known installers only use
-  HKLM, so HKCU is typically empty.)
-
-* If the environment variable :envvar:`PYTHONHOME` is set, it is assumed as
-  "Python Home".  Otherwise, the path of the main Python executable is used to
-  locate a "landmark file" (``Lib\os.py``) to deduce the "Python Home".  If a
-  Python home is found, the relevant sub-directories added to :data:`sys.path`
-  (``Lib``, ``plat-win``, etc) are based on that folder.  Otherwise, the core
-  Python path is constructed from the PythonPath stored in the registry.
-
-* If the Python Home cannot be located, no :envvar:`PYTHONPATH` is specified in
-  the environment, and no registry entries can be found, a default path with
-  relative entries is used (e.g. ``.\Lib;.\plat-win``, etc).
-
-The end result of all this is:
-
-* When running :file:`python.exe`, or any other .exe in the main Python
-  directory (either an installed version, or directly from the PCbuild
-  directory), the core path is deduced, and the core paths in the registry are
-  ignored.  Other "application paths" in the registry are always read.
-
-* When Python is hosted in another .exe (different directory, embedded via COM,
-  etc), the "Python Home" will not be deduced, so the core path from the
-  registry is used.  Other "application paths" in the registry are always read.
-
-* If Python can't find its home and there is no registry (eg, frozen .exe, some
-  very strange installation setup) you get a path with some default, but
-  relative, paths.
-
-
-Executing scripts
------------------
-
-As of Python 3.3, Python includes a launcher which facilitates running Python
-scripts. See :ref:`launcher` for more information.
-
-Executing scripts without the Python launcher
----------------------------------------------
-
-Without the Python launcher installed, Python scripts (files with the extension
-``.py``) will be executed by :program:`python.exe` by default.  This executable
-opens a terminal, which stays open even if the program uses a GUI.  If you do
-not want this to happen, use the extension ``.pyw`` which will cause the script
-to be executed by :program:`pythonw.exe` by default (both executables are
-located in the top-level of your Python installation directory).  This
-suppresses the terminal window on startup.
-
-You can also make all ``.py`` scripts execute with :program:`pythonw.exe`,
-setting this through the usual facilities, for example (might require
-administrative rights):
-
-#. Launch a command prompt.
-#. Associate the correct file group with ``.py`` scripts::
-
-      assoc .py=Python.File
-
-#. Redirect all Python files to the new executable::
-
-      ftype Python.File=C:\Path\to\pythonw.exe "%1" %*
-
+    C:\WINDOWS\system32;C:\WINDOWS;C:\Program Files\Python 3.5
 
 .. _launcher:
 
@@ -251,21 +349,26 @@
 
 .. versionadded:: 3.3
 
-The Python launcher for Windows is a utility which aids in the location and
-execution of different Python versions.  It allows scripts (or the
+The Python launcher for Windows is a utility which aids in locating and
+executing of different Python versions.  It allows scripts (or the
 command-line) to indicate a preference for a specific Python version, and
 will locate and execute that version.
 
+Unlike the :envvar:`PATH` variable, the launcher will correctly select the most
+appropriate version of Python. It will prefer per-user installations over
+system-wide ones, and orders by language version rather than using the most
+recently installed version.
+
 Getting started
 ---------------
 
 From the command-line
 ^^^^^^^^^^^^^^^^^^^^^
 
-You should ensure the launcher is on your PATH - depending on how it was
-installed it may already be there, but check just in case it is not.
-
-From a command-prompt, execute the following command:
+System-wide installations of Python 3.3 and later will put the launcher on your
+:envvar:`PATH`. The launcher is compatible with all available versions of
+Python, so it does not matter which version is installed. To check that the
+launcher is available, execute the following command in Command Prompt:
 
 ::
 
@@ -291,6 +394,16 @@
 
 You should find the latest version of Python 3.x starts.
 
+If you see the following error, you do not have the launcher installed:
+
+::
+
+  'py' is not recognized as an internal or external command,
+  operable program or batch file.
+
+Per-user installations of Python do not add the launcher to :envvar:`PATH`
+unless the option was selected on installation.
+
 From a script
 ^^^^^^^^^^^^^
 
@@ -383,17 +496,16 @@
 Customization via INI files
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-    Two .ini files will be searched by the launcher - ``py.ini`` in the
-    current user's "application data" directory (i.e. the directory returned
-    by calling the Windows function SHGetFolderPath with CSIDL_LOCAL_APPDATA)
-    and ``py.ini`` in the same directory as the launcher.  The same .ini
-    files are used for both the 'console' version of the launcher (i.e.
-    py.exe) and for the 'windows' version (i.e. pyw.exe)
+Two .ini files will be searched by the launcher - ``py.ini`` in the current
+user's "application data" directory (i.e. the directory returned by calling the
+Windows function SHGetFolderPath with CSIDL_LOCAL_APPDATA) and ``py.ini`` in the
+same directory as the launcher. The same .ini files are used for both the
+'console' version of the launcher (i.e. py.exe) and for the 'windows' version
+(i.e. pyw.exe)
 
-    Customization specified in the "application directory" will have
-    precedence over the one next to the executable, so a user, who may not
-    have write access to the .ini file next to the launcher, can override
-    commands in that global .ini file)
+Customization specified in the "application directory" will have precedence over
+the one next to the executable, so a user, who may not have write access to the
+.ini file next to the launcher, can override commands in that global .ini file)
 
 Customizing default Python versions
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -488,6 +600,78 @@
 target Python.
 
 
+
+Finding modules
+===============
+
+Python usually stores its library (and thereby your site-packages folder) in the
+installation directory.  So, if you had installed Python to
+:file:`C:\\Python\\`, the default library would reside in
+:file:`C:\\Python\\Lib\\` and third-party modules should be stored in
+:file:`C:\\Python\\Lib\\site-packages\\`.
+
+This is how :data:`sys.path` is populated on Windows:
+
+* An empty entry is added at the start, which corresponds to the current
+  directory.
+
+* If the environment variable :envvar:`PYTHONPATH` exists, as described in
+  :ref:`using-on-envvars`, its entries are added next.  Note that on Windows,
+  paths in this variable must be separated by semicolons, to distinguish them
+  from the colon used in drive identifiers (``C:\`` etc.).
+
+* Additional "application paths" can be added in the registry as subkeys of
+  :samp:`\\SOFTWARE\\Python\\PythonCore\\{version}\\PythonPath` under both the
+  ``HKEY_CURRENT_USER`` and ``HKEY_LOCAL_MACHINE`` hives.  Subkeys which have
+  semicolon-delimited path strings as their default value will cause each path
+  to be added to :data:`sys.path`.  (Note that all known installers only use
+  HKLM, so HKCU is typically empty.)
+
+* If the environment variable :envvar:`PYTHONHOME` is set, it is assumed as
+  "Python Home".  Otherwise, the path of the main Python executable is used to
+  locate a "landmark file" (``Lib\os.py``) to deduce the "Python Home".  If a
+  Python home is found, the relevant sub-directories added to :data:`sys.path`
+  (``Lib``, ``plat-win``, etc) are based on that folder.  Otherwise, the core
+  Python path is constructed from the PythonPath stored in the registry.
+
+* If the Python Home cannot be located, no :envvar:`PYTHONPATH` is specified in
+  the environment, and no registry entries can be found, a default path with
+  relative entries is used (e.g. ``.\Lib;.\plat-win``, etc).
+
+The end result of all this is:
+
+* When running :file:`python.exe`, or any other .exe in the main Python
+  directory (either an installed version, or directly from the PCbuild
+  directory), the core path is deduced, and the core paths in the registry are
+  ignored.  Other "application paths" in the registry are always read.
+
+* When Python is hosted in another .exe (different directory, embedded via COM,
+  etc), the "Python Home" will not be deduced, so the core path from the
+  registry is used.  Other "application paths" in the registry are always read.
+
+* If Python can't find its home and there is no registry (eg, frozen .exe, some
+  very strange installation setup) you get a path with some default, but
+  relative, paths.
+
+For those who want to bundle Python into their application or distribution, the
+following advice will prevent conflicts with other installations:
+
+* If you are loading :file:`python3.dll` or :file:`python35.dll` in your own
+  executable, explicitly call :c:func:`Py_SetPath` or (at least)
+  :c:func:`Py_SetProgramName` before :c:func:`Py_Initialize`.
+
+* Clear and/or overwrite :envvar:`PYTHONPATH` and set :envvar:`PYTHONHOME`
+  before launching :file:`python.exe` from your application.
+
+* If you cannot use the previous suggestions (for example, you are a
+  distribution that allows people to run :file:`python.exe` directly), ensure
+  that the landmark file (:file:`Lib\\os.py`) exists in your bundled library.
+  (Note that it will not be detected inside a ZIP file.)
+
+These will ensure that the files in a system-wide installation will not take
+precedence over the copy of the standard library bundled with your application.
+Otherwise, your users may experience problems using your application.
+
 Additional modules
 ==================
 
@@ -498,7 +682,6 @@
 The Windows-specific standard modules are documented in
 :ref:`mswin-specific-services`.
 
-
 PyWin32
 -------
 
@@ -557,20 +740,8 @@
 <https://docs.python.org/devguide/setup.html#getting-the-source-code>`_.
 
 The source tree contains a build solution and project files for Microsoft
-Visual C++, which is the compiler used to build the official Python releases.
-View the :file:`readme.txt` in their respective directories:
-
-+--------------------+--------------+-----------------------+
-| Directory          | MSVC version | Visual Studio version |
-+====================+==============+=======================+
-| :file:`PC/VS9.0/`  | 9.0          | 2008                  |
-+--------------------+--------------+-----------------------+
-| :file:`PCbuild/`   | 10.0         | 2010                  |
-+--------------------+--------------+-----------------------+
-
-Note that any build directories within the :file:`PC` directory are not
-necessarily fully supported.  The :file:`PCbuild` directory contains the files
-for the compiler used to build the official release.
+Visual Studio 2015, which is the compiler used to build the official Python
+releases. These files are in the :file:`PCbuild` directory.
 
 Check :file:`PCbuild/readme.txt` for general information on the build process.
 
diff --git a/Misc/NEWS b/Misc/NEWS
--- a/Misc/NEWS
+++ b/Misc/NEWS
@@ -1778,6 +1778,8 @@
 Windows
 -------
 
+- Issue #23260: Update Windows installer
+
 - The bundled version of Tcl/Tk has been updated to 8.6.3.  The most visible
   result of this change is the addition of new native file dialogs when
   running on Windows Vista or newer.  See Tcl/Tk's TIP 432 for more
diff --git a/PC/launcher.c b/PC/launcher.c
--- a/PC/launcher.c
+++ b/PC/launcher.c
@@ -157,14 +157,19 @@
 
 static size_t num_installed_pythons = 0;
 
-/* to hold SOFTWARE\Python\PythonCore\X.Y\InstallPath */
+/*
+ * To hold SOFTWARE\Python\PythonCore\X.Y...\InstallPath
+ * The version name can be longer than MAX_VERSION_SIZE, but will be
+ * truncated to just X.Y for comparisons.
+ */
 #define IP_BASE_SIZE 40
-#define IP_SIZE (IP_BASE_SIZE + MAX_VERSION_SIZE)
+#define IP_VERSION_SIZE 8
+#define IP_SIZE (IP_BASE_SIZE + IP_VERSION_SIZE)
 #define CORE_PATH L"SOFTWARE\\Python\\PythonCore"
 
 static wchar_t * location_checks[] = {
     L"\\",
-    L"\\PCBuild\\",
+    L"\\PCBuild\\win32\\",
     L"\\PCBuild\\amd64\\",
     NULL
 };
@@ -196,6 +201,7 @@
     BOOL ok;
     DWORD type, data_size, attrs;
     INSTALLED_PYTHON * ip, * pip;
+    wchar_t ip_version[IP_VERSION_SIZE];
     wchar_t ip_path[IP_SIZE];
     wchar_t * check;
     wchar_t ** checkp;
@@ -207,19 +213,21 @@
     else {
         ip = &installed_pythons[num_installed_pythons];
         for (i = 0; num_installed_pythons < MAX_INSTALLED_PYTHONS; i++) {
-            status = RegEnumKeyW(core_root, i, ip->version, MAX_VERSION_SIZE);
+            status = RegEnumKeyW(core_root, i, ip_version, IP_VERSION_SIZE);
             if (status != ERROR_SUCCESS) {
                 if (status != ERROR_NO_MORE_ITEMS) {
                     /* unexpected error */
                     winerror(status, message, MSGSIZE);
                     debug(L"Can't enumerate registry key for version %ls: %ls\n",
-                          ip->version, message);
+                          ip_version, message);
                 }
                 break;
             }
             else {
+                wcsncpy_s(ip->version, MAX_VERSION_SIZE, ip_version,
+                          MAX_VERSION_SIZE-1);
                 _snwprintf_s(ip_path, IP_SIZE, _TRUNCATE,
-                             L"%ls\\%ls\\InstallPath", CORE_PATH, ip->version);
+                             L"%ls\\%ls\\InstallPath", CORE_PATH, ip_version);
                 status = RegOpenKeyExW(root, ip_path, 0, flags, &ip_key);
                 if (status != ERROR_SUCCESS) {
                     winerror(status, message, MSGSIZE);
diff --git a/PCbuild/_freeze_importlib.vcxproj b/PCbuild/_freeze_importlib.vcxproj
--- a/PCbuild/_freeze_importlib.vcxproj
+++ b/PCbuild/_freeze_importlib.vcxproj
@@ -81,7 +81,7 @@
   <Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
   <ImportGroup Label="ExtensionTargets">
   </ImportGroup>
-  <Target Name="RebuildImportLib" AfterTargets="AfterBuild">
+  <Target Name="RebuildImportLib" AfterTargets="AfterBuild" Condition="$(Configuration) == 'Debug' or $(Configuration) == 'Release'">
     <Exec Command='"$(TargetPath)" "$(PySourcePath)Lib\importlib\_bootstrap.py" "$(IntDir)importlib.g.h"' />
     
     <PropertyGroup>
@@ -93,9 +93,10 @@
           DestinationFiles="$(PySourcePath)Python\importlib.h"
           Condition="Exists('$(IntDir)importlib.g.h') and '$(_OldContent)' != '$(_NewContent)'" />
     
-    <Message Text="importlib.h has been updated. You will need to rebuild pythoncore to see the changes."
-             Importance="high"
-             Condition="Exists('$(IntDir)importlib.g.h') and '$(_OldContent)' != '$(_NewContent)'" />
+    <Warning Text="importlib.h has been updated. You will need to rebuild pythoncore to see the changes."
+             Condition="Exists('$(IntDir)importlib.g.h') and '$(_OldContent)' != '$(_NewContent)' and $(Configuration) == 'Debug'" />
+    <Error Text="importlib.h has been updated. You will need to rebuild pythoncore to see the changes."
+           Condition="Exists('$(IntDir)importlib.g.h') and '$(_OldContent)' != '$(_NewContent)' and $(Configuration) == 'Release'" />
   </Target>
   <Target Name="_CleanImportLib" BeforeTargets="CoreClean">
     <ItemGroup>
diff --git a/PCbuild/libeay.vcxproj b/PCbuild/libeay.vcxproj
--- a/PCbuild/libeay.vcxproj
+++ b/PCbuild/libeay.vcxproj
@@ -41,8 +41,12 @@
 
   <Import Project="python.props" />
   <Import Project="$(VCTargetsPath)\Microsoft.Cpp.Default.props" />
+
+  <PropertyGroup Label="Configuration">
+    <ConfigurationType>StaticLibrary</ConfigurationType>
+  </PropertyGroup>
+   
   <Import Project="openssl.props" />
-  
   <Import Project="$(VCTargetsPath)\Microsoft.Cpp.props" />
 
   <Target Name="CreateBuildinfH" Inputs="$(MSBuildProjectFullPath)" Outputs="$(IntDir)\buildinf.h" AfterTargets="PrepareForBuild">
diff --git a/PCbuild/pylauncher.vcxproj b/PCbuild/pylauncher.vcxproj
--- a/PCbuild/pylauncher.vcxproj
+++ b/PCbuild/pylauncher.vcxproj
@@ -61,6 +61,7 @@
   <ItemDefinitionGroup>
     <ClCompile>
       <PreprocessorDefinitions>_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
+      <RuntimeLibrary>MultiThreaded</RuntimeLibrary>
     </ClCompile>
     <Link>
       <AdditionalDependencies>version.lib;%(AdditionalDependencies)</AdditionalDependencies>
diff --git a/PCbuild/pyproject.props b/PCbuild/pyproject.props
--- a/PCbuild/pyproject.props
+++ b/PCbuild/pyproject.props
@@ -11,6 +11,9 @@
     <GenerateManifest>false</GenerateManifest>
     <EmbedManifest>false</EmbedManifest>
     <SupportPGO Condition="'$(SupportPGO)' == ''">true</SupportPGO>
+    <SupportSigning Condition="'$(SupportSigning)' == ''">true</SupportSigning>
+    <SupportSigning Condition="'$(Configuration)' == 'Debug'">false</SupportSigning>
+    <SupportSigning Condition="'$(ConfigurationType)' == 'StaticLibrary'">false</SupportSigning>
   </PropertyGroup>
 
   <PropertyGroup>
@@ -138,4 +141,16 @@
              Condition="Exists(%(FullPath))"
              Targets="CleanAll" />
   </Target>
+
+  <PropertyGroup Condition="'$(SigningCertificate)' != '' and $(SupportSigning)">
+    <SignToolPath Condition="'$(SignToolPath)' == '' or !Exists($(SignToolPath))">$(registry:HKEY_LOCAL_MACHINE\Software\Microsoft\Windows Kits\Installed Roots at KitsRoot81)\bin\x86\signtool.exe</SignToolPath>
+    <SignToolPath Condition="!Exists($(SignToolPath))">$(registry:HKEY_LOCAL_MACHINE\Software\Microsoft\Windows Kits\Installed Roots at KitsRoot)\bin\x86\signtool.exe</SignToolPath>
+    <SignToolPath Condition="!Exists($(SignToolPath))">$(registry:HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SDKs\Windows\v7.1A at InstallationFolder)\Bin\signtool.exe</SignToolPath>
+    <_SignCommand Condition="Exists($(SignToolPath))">"$(SignToolPath)" sign /q /n "$(SigningCertificate)" /t http://timestamp.verisign.com/scripts/timestamp.dll /d "Python $(PythonVersion)"</_SignCommand>
+  </PropertyGroup>
+  
+  <Target Name="_SignBuild" AfterTargets="AfterBuild" Condition="'$(SigningCertificate)' != '' and $(SupportSigning)">
+    <Error Text="Unable to locate signtool.exe. Set /p:SignToolPath and rebuild" Condition="'$(_SignCommand)' == ''" />
+    <Exec Command='$(_SignCommand) "$(TargetPath)" || $(_SignCommand) "$(TargetPath)" || $(_SignCommand) "$(TargetPath)"' ContinueOnError="false" />
+  </Target>
 </Project>
\ No newline at end of file
diff --git a/PCbuild/python.props b/PCbuild/python.props
--- a/PCbuild/python.props
+++ b/PCbuild/python.props
@@ -1,7 +1,7 @@
 <?xml version="1.0" encoding="utf-8"?>
-<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003" TreatAsLocalProperty="Platform">
+<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
   <PropertyGroup>
-    <Platform Condition="'$(Platform)' == '' or '$(Platform)' == 'x86'">Win32</Platform>
+    <Platform Condition="'$(Platform)' == ''">Win32</Platform>
     <Configuration Condition="'$(Configuration)' == ''">Release</Configuration>
     <!--
     Use the latest available version of Visual Studio to build. To override
@@ -98,12 +98,12 @@
     <PyDllName>python$(MajorVersionNumber)$(MinorVersionNumber)$(PyDebugExt)</PyDllName>
 
     <!-- The version and platform tag to include in .pyd filenames -->
-    <PydTag Condition="$(Platform) == 'Win32'">.cp$(MajorVersionNumber)$(MinorVersionNumber)-win32</PydTag>
+    <PydTag Condition="$(Platform) == 'Win32' or $(Platform) == 'x86'">.cp$(MajorVersionNumber)$(MinorVersionNumber)-win32</PydTag>
     <PydTag Condition="$(Platform) == 'x64'">.cp$(MajorVersionNumber)$(MinorVersionNumber)-win_amd64</PydTag>
     
     <!-- The version number for sys.winver -->
     <SysWinVer>$(MajorVersionNumber).$(MinorVersionNumber)</SysWinVer>
-    <SysWinVer Condition="$(Platform) == 'Win32'">$(SysWinVer)-32</SysWinVer>
+    <SysWinVer Condition="$(Platform) == 'Win32' or $(Platform) == 'x86'">$(SysWinVer)-32</SysWinVer>
   </PropertyGroup>
   
   <!-- Displays the calculated version info -->
diff --git a/PCbuild/pywlauncher.vcxproj b/PCbuild/pywlauncher.vcxproj
--- a/PCbuild/pywlauncher.vcxproj
+++ b/PCbuild/pywlauncher.vcxproj
@@ -61,6 +61,7 @@
   <ItemDefinitionGroup>
     <ClCompile>
       <PreprocessorDefinitions>_WINDOWS;%(PreprocessorDefinitions)</PreprocessorDefinitions>
+      <RuntimeLibrary>MultiThreaded</RuntimeLibrary>
     </ClCompile>
     <Link>
       <AdditionalDependencies>version.lib;%(AdditionalDependencies)</AdditionalDependencies>
diff --git a/PCbuild/ssleay.vcxproj b/PCbuild/ssleay.vcxproj
--- a/PCbuild/ssleay.vcxproj
+++ b/PCbuild/ssleay.vcxproj
@@ -41,8 +41,12 @@
 
   <Import Project="python.props" />
   <Import Project="$(VCTargetsPath)\Microsoft.Cpp.Default.props" />
+
+  <PropertyGroup Label="Configuration">
+    <ConfigurationType>StaticLibrary</ConfigurationType>
+  </PropertyGroup>
+
   <Import Project="openssl.props" />
-  
   <Import Project="$(VCTargetsPath)\Microsoft.Cpp.props" />
 
   <ItemGroup>
diff --git a/Tools/buildbot/buildmsi.bat b/Tools/buildbot/buildmsi.bat
--- a/Tools/buildbot/buildmsi.bat
+++ b/Tools/buildbot/buildmsi.bat
@@ -1,21 +1,9 @@
 @rem Used by the buildbot "buildmsi" step.
 setlocal
 
-set cwd=%CD%
+pushd
 
- at rem build release versions of things
-call "%~dp0build.bat" -c Release
+ at rem build both snapshot MSIs
+call "%~dp0..\msi\build.bat" -x86 -x64
 
- at rem build the documentation
-call "%~dp0..\..\Doc\make.bat" htmlhelp
-
- at rem build the MSI file
-call "%~dp0..\..\PCBuild\env.bat" x86
-cd "%~dp0..\..\PC"
-nmake /f icons.mak
-cd ..\Tools\msi
-del *.msi
-nmake /f msisupport.mak
-%HOST_PYTHON% msi.py
-
-cd "%cwd%"
+popd
\ No newline at end of file
diff --git a/Tools/msi/README.txt b/Tools/msi/README.txt
--- a/Tools/msi/README.txt
+++ b/Tools/msi/README.txt
@@ -1,25 +1,488 @@
-Packaging Python as a Microsoft Installer Package (MSI)
-=======================================================
+Quick Build Info
+================
 
-Using this library, Python can be packaged as a MS-Windows
-MSI file. To generate an installer package, you need
-a build tree. By default, the build tree root directory
-is assumed to be in "../..". This location can be changed
-by adding a file config.py; see the beginning of msi.py
-for additional customization options.
+For testing, the installer should be built with the Tools/msi/build.bat
+script:
 
-The packaging process assumes that binaries have been 
-generated according to the instructions in PCBuild/README.txt,
-and that you have either Visual Studio or the Platform SDK
-installed. In addition, you need the Python COM extensions,
-either from PythonWin, or from ActivePython.
+    build.bat [-x86] [-x64] [--doc]
 
-To invoke the script, open a cmd.exe window which has 
-cabarc.exe in its PATH (e.g. "Visual Studio .NET 2003
-Command Prompt"). Then invoke
+For an official release, the installer should be built with the
+Tools/msi/buildrelease.bat script and environment variables:
 
-<path-to-python.exe> msi.py
+    set PYTHON=<path to Python 2.7>
+    set SPHINXBUILD=<path to sphinx-build.exe>
+    set PATH=<path to Mercurial (hg.exe)>;
+             <path to HTML Help Compiler (hhc.exe)>;%PATH%
 
-If everything succeeds, pythonX.Y.Z.msi is generated
-in the current directory.
+    buildrelease.bat [-x86] [-x64] [-D] [-B]
+        [-o <output directory>] [-c <certificate name>]
 
+See the Building the Installer section for more information.
+
+Overview
+========
+
+Python is distributed on Windows as an installer that will configure the
+user's system. This allows users to have a functioning copy of Python
+without having to build it themselves.
+
+The main tasks of the installer are:
+
+* copy required files into the expected layout
+* configure system settings so the installation can be located by
+  other programs
+* add entry points for modifying, repairing and uninstalling Python
+* make it easy to launch Python, its documentation, and IDLE
+
+Each of these is discussed in a later section of this document.
+
+Structure of the Installer
+==========================
+
+The installer is structured as a 'layout', which consists of a number of
+CAB and MSI files and a single EXE.
+
+The EXE is the main entry point into the installer. It contains the UI
+and command-line logic, as well as the ability to locate and optionally
+download other parts of the layout.
+
+Each MSI contains the logic required to install a component or feature
+of Python. These MSIs should not be launched directly by users. MSIs can
+be embedded into the EXE or automatically downloaded as needed.
+
+Each CAB contains the files making up a Python installation. CABs are
+embedded into their associated MSI and are never seen by users.
+
+MSIs are only required when the related feature or component is being
+installed. When components are not selected for installation, the
+associated MSI is not downloaded. This allows the installer to offer
+options to install debugging symbols and binaries without increasing
+the initial download size by separating them into their own MSIs.
+
+Building the Installer
+======================
+
+For testing, the installer should be built with the Tools/msi/build.bat
+script:
+
+    build.bat [-x86] [-x64] [--doc]
+
+This script will build the required configurations of Python and
+generate an installer layout in PCBuild/(win32|amd64)/en-us.
+
+Specify -x86 and/or -x64 to build for each platform. If neither is
+specified, both platforms will be built. Currently, both the debug and
+release versions of Python are required for the installer.
+
+Specify --doc to build the documentation (.chm) file. If the file is not
+available, it will simply be excluded from the installer. Ensure
+%PYTHON% and %SPHINXBUILD% are set when passing this option. You may
+also set %HTMLHELP% to the Html Help Compiler (hhc.exe), or put HHC on
+your PATH or in externals/.
+
+If WiX is not found on your system, it will be automatically downloaded
+and extracted to the externals/ directory.
+
+
+For an official release, the installer should be built with the
+Tools/msi/buildrelease.bat script:
+
+    set PYTHON=<path to Python 2.7>
+    set SPHINXBUILD=<path to sphinx-build.exe>
+    set PATH=<path to Mercurial (hg.exe)>;
+             <path to HTML Help Compiler (hhc.exe)>;%PATH%
+
+    buildrelease.bat [-x86] [-x64] [-D] [-B]
+        [-o <output directory>] [-c <certificate name>]
+
+Specify -x86 and/or -x64 to build for each platform. If neither is
+specified, both platforms will be built. Currently, both the debug and
+release versions of Python are required for the installer.
+
+Specify -D to skip rebuilding the documentation. The documentation is
+required for a release and the build will fail if it is not available.
+
+Specify -B to skip rebuilding Python. This is useful to only rebuild the
+installer layout after a previous call to buildrelease.bat.
+
+Specify -o to set an output directory. The installer layouts will be
+copied to platform-specific subdirectories of this path.
+
+Specify -c to choose a code-signing certificate to be used for all the
+signable binaries in Python as well as each file making up the
+installer. Official releases of Python must be signed.
+
+Ensure %PYTHON% and %SPHINXBUILD% are set when passing this option. You
+may also set %HTMLHELP% to the Html Help Compiler (hhc.exe), or put HHC
+on your PATH or in externals/. You will also need Mercurial (hg.exe) on
+your PATH.
+
+If WiX is not found on your system, it will be automatically downloaded
+and extracted to the externals/ directory.
+
+To manually build layouts of the installer, build one of the projects in
+the bundle folder.
+
+    msbuild bundle\snapshot.wixproj
+    msbuild bundle\releaseweb.wixproj
+    msbuild bundle\releaselocal.wixproj
+    msbuild bundle\full.wixproj
+
+snapshot.wixproj produces a test installer versioned based on the date.
+
+releaseweb.wixproj produces a release installer that does not embed any
+of the layout.
+
+releaselocal.wixproj produces a release installer that embeds the files
+required for a default installation.
+
+full.wixproj produces a test installer that embeds the entire layout.
+
+The following properties may be passed when building these projects.
+
+  /p:BuildForRelease=(true|false)
+    When true, adds extra verification to ensure a complete installer is
+    produced. For example, binutils is required when building for a release
+    to generate MinGW-compatible libraries, and the build will be aborted if
+    this fails. Defaults to false.
+
+  /p:ReleaseUri=(any URI)
+    Used to generate unique IDs for the installers to allow side-by-side
+    installation. Forks of Python can use the same installer infrastructure
+    by providing a unique URI for this property. It does not need to be an
+    active internet address. Defaults to $(ComputerName).
+
+    Official releases use http://www.python.org/(architecture name)
+
+  /p:DownloadUrlBase=(any URI)
+    Specifies the base of a URL where missing parts of the installer layout
+    can be downloaded from. The build version and architecture will be
+    appended to create the full address. If omitted, missing components will
+    not be automatically downloaded.
+
+  /p:DownloadUrl=(any URI)
+    Specifies the full URL where missing parts of the installer layout can
+    be downloaded from. Should normally include '{2}', which will be
+    substituted for the filename. If omitted, missing components will not be
+    automatically downloaded. If specified, this value overrides
+    DownloadUrlBase.
+
+  /p:SigningCertificate=(certificate name)
+    Specifies the certificate to sign the installer layout with. If omitted,
+    the layout will not be signed.
+
+  /p:RebuildAll=(true|false)
+    When true, rebuilds all of the MSIs making up the layout. Defaults to
+    true.
+
+Modifying the Installer
+=======================
+
+The code for the installer is divided into three main groups: packages,
+the bundle and the bootstrap application.
+
+Packages
+--------
+
+Packages appear as subdirectories of Tools/msi (other than the bundle/
+directory). The project file is a .wixproj and the build output is a
+single MSI. Packages are built with the WiX Toolset. Some project files
+share source files and use preprocessor directives to enable particular
+features. These are typically used to keep the sources close when the
+files are related, but produce multiple independent packages.
+
+A package is the smallest element that may be independently installed or
+uninstalled (as used in this installer). For example, the test suite has
+its own package, as users can choose to add or remove it after the
+initial installation.
+
+All the files installed by a single package should be related, though
+some packages may not install any files. For example, the pip package
+executes the ensurepip package, but does not add or remove any of its
+own files. (It is represented as a package because of its
+installed/uninstalled nature, as opposed to the "precompile standard
+library" option, for example.) Dependencies between packages are handled
+by the bundle, but packages should detect when dependencies are missing
+and raise an error.
+
+Packages that include a lot of files may use an InstallFiles element in
+the .wixproj file to generate sources. See lib/lib.wixproj for an
+example, and msi.targets and csv_to_wxs.py for the implementation. This
+element is also responsible for generating the code for cleaning up and
+removing __pycache__ folders in any directory containing .py files.
+
+All packages are built with the Tools/msi/common.wxs file, and so any
+directory or property in this file may be referenced. Of particular
+interest:
+
+  REGISTRYKEY (property)
+    The registry key for the current installation.
+
+  InstallDirectory (directory)
+    The root install directory for the current installation. Subdirectories
+    are also specified in this file (DLLs, Lib, etc.)
+
+  MenuDir (directory)
+    The Start Menu folder for the current installation.
+
+  UpgradeTable (property)
+    Every package should reference this property to include upgrade
+    information.
+
+The .wxl_template file is specially handled by the build system for this
+project to perform {{substitutions}} as defined in msi.targets. They
+should be included in projects as <WxlTemplate> items, where .wxl files
+are normally included as <EmbeddedResource> items.
+
+Bundle
+------
+
+The bundle is compiled to the main EXE entry point that for most users
+will represent the Python installer. It is built from Tools/msi/bundle
+with packages references in Tools/msi/bundle/packagegroups.
+
+Build logic for the bundle is in bundle.targets, but should be invoked
+through one of the .wixproj files as described in Building the
+Installer.
+
+The UI is separated between Default.thm (UI layout), Default.wxl
+(strings), bundle.wxs (properties) and the bootstrap application.
+Bundle.wxs also contains the chain, which is the list of packages to
+install and the order they should be installed in. These refer to named
+package groups in bundle/packagegroups.
+
+Each package group specifies one or more packages to install. Most
+packages require two separate entries to support both per-user and
+all-users installations. Because these reuse the same package, it does
+not increase the overall size of the package.
+
+Package groups refer to payload groups, which allow better control over
+embedding and downloading files than the default settings. Whether files
+are embedded and where they are downloaded from depends on settings
+created by the project files.
+
+Package references can include install conditions that determine when to
+install the package. When a package is a dependency for others, the
+condition should be crafted to ensure it is installed.
+
+MSI packages are installed or uninstalled based on their current state
+and the install condition. This makes them most suitable for features
+that are clearly present or absent from the user's machine.
+
+EXE packages are executed based on a customisable condition that can be
+omitted. This makes them suitable for pre- or post-install tasks that
+need to run regardless of whether features have been added or removed.
+
+Bootstrap Application
+---------------------
+
+The bootstrap application is a C++ application that controls the UI and
+installation. While it does not directly compile into the main EXE of
+the installer, it forms the main active component. Most of the
+installation functionality is provided by WiX, and so the bootstrap
+application is predominantly responsible for the code behind the UI that
+is defined in the Default.thm file. The bootstrap application code is in
+bundle/bootstrap and is built automatically when building the bundle.
+
+Installation Layout
+===================
+
+There are two installation layouts for Python on Windows, with the only
+differences being supporting files. A layout is selected implicitly
+based on whether the install is for all users of the machine or just for
+the user performing the installation.
+
+The default installation location when installing for all users is
+"%ProgramFiles%\Python 3.X" for the 64-bit interpreter and
+"%ProgramFiles(x86)%\Python 3.X" for the 32-bit interpreter. (Note that
+the latter path is equivalent to "%ProgramFiles%\Python 3.X" when
+running a 32-bit version of Windows.) This location requires
+administrative privileges to install or later modify the installation.
+
+The default installation location when installing for the current user
+is "%LocalAppData%\Programs\Python\Python3X" for the 64-bit interpreter
+and "%LocalAppData%\Programs\Python\Python3X-32" for the 32-bit
+interpreter. Only the current user can access this location. This
+provides a suitable level of protection against malicious modification
+of Python's files.
+
+Within this install directory is the following approximate layout:
+
+.\python[w].exe The core executable files
+.\DLLs          Stdlib extensions (*.pyd) and dependencies
+.\Doc           Documentation (*.chm)
+.\include       Development headers (*.h)
+.\Lib           Standard library
+.\Lib\test      Test suite
+.\libs          Development libraries (*.lib)
+.\Scripts       Launcher scripts (*.exe, *.py)
+.\tcl           Tcl dependencies (*.dll, *.tcl and others)
+.\Tools         Tool scripts (*.py)
+
+When installed for all users, the following files are installed to
+either "%SystemRoot%\System32" or "%SystemRoot%\SysWOW64" as
+appropriate. For the current user, they are installed in the Python
+install directory.
+
+.\python3x.dll      The core interpreter
+.\python3.dll       The stable ABI reference
+.\appcrt140.dll     Microsoft Visual C Runtime
+.\desktopcrt140.dll Microsoft Visual C Runtime
+.\vcruntime140.dll  Microsoft Visual C Runtime
+
+When installed for all users, the following files are installed to
+"%SystemRoot%" (typically "C:\Windows") to ensure they are always
+available on PATH. (See Launching Python below.) For the current user,
+they are installed in the Python install directory.
+
+.\py[w].exe         PEP 397 launcher
+
+System Settings
+===============
+
+On installation, registry keys are created so that other applications
+can locate and identify installations of Python. The locations of these
+keys vary based on the install type.
+
+For 64-bit interpreters installed for all users, the root key is:
+    HKEY_LOCAL_MACHINE\Software\Python\PythonCore\3.X
+
+For 32-bit interpreters installed for all users on a 64-bit operating
+system, the root key is:
+    HKEY_LOCAL_MACHINE\Software\Wow6432Node\Python\PythonCore\3.X-32
+
+For 32-bit interpreters installed for all users on a 32-bit operating
+system, the root key is:
+    HKEY_LOCAL_MACHINE\Software\Python\PythonCore\3.X-32
+
+For 64-bit interpreters installed for the current user:
+    HKEY_CURRENT_USER\Software\Python\PythonCore\3.X
+
+For 32-bit interpreters installed for the current user:
+    HKEY_CURRENT_USER\Software\Python\PythonCore\3.X-32
+
+When the core Python executables are installed, a key "InstallPath" is
+created within the root key with its default value set to the
+executable's install directory. Within this key, a key "InstallGroup" is
+created with its default value set to the product name "Python 3.X".
+
+When the Python standard library is installed, a key "PythonPath" is
+created within the root key with its default value set to the full path
+to the Lib folder followed by the path to the DLLs folder, separated by
+a semicolon.
+
+When the documentation is installed, a key "Help" is created within the
+root key, with a subkey "Main Python Documentation" with its default
+value set to the full path to the installed CHM file.
+
+
+The py.exe launcher is installed as part of a regular Python install,
+but using a separate mechanism that allows it to more easily span
+versions of Python. As a result, it has different root keys for its
+registry entries:
+
+When installed for all users on a 64-bit operating system, the
+launcher's root key is:
+    HKEY_LOCAL_MACHINE\Software\Wow6432Node\Python\Launcher
+
+When installed for all users on a 32-bit operating system, the
+launcher's root key is:
+    HKEY_LOCAL_MACHINE\Software\Python\Launcher
+
+When installed for the current user:
+    HKEY_CURRENT_USER\Software\Python\Launcher
+
+When the launcher is installed, a key "InstallPath" is created within
+its root key with its default value set to the launcher's install
+directory. File associations are also created for .py, .pyw, .pyc and
+.pyo files.
+
+Launching Python
+================
+
+When a feature offering user entry points in the Start Menu is
+installed, a folder "Python 3.X" is created. Every shortcut should be
+created within this folder, and each shortcut should include the version
+and platform to allow users to identify the shortcut in a search results
+page.
+
+The core Python executables creates a shortcut "Python 3.X (32-bit)" or
+"Python 3.X (64-bit)" depending on the interpreter.
+
+The documentation creates a shortcut "Python 3.X 32-bit Manuals" or
+"Python 3.X 64-bit Manuals". The documentation is identical for all
+platforms, but the shortcuts need to be separate to avoid uninstallation
+conflicts.
+
+Installing IDLE creates a shortcut "IDLE (Python 3.X 32-bit)" or "IDLE
+(Python 3.X 64-bit)" depending on the interpreter.
+
+
+For users who often launch Python from a Command Prompt, an option is
+provided to add the directory containing python.exe to the user or
+system PATH variable. If the option is selected, the install directory
+and the Scripts directory will be added at the start of the system PATH
+for an all users install and the user PATH for a per-user install.
+
+When the user only has one version of Python installed, this will behave
+as expected. However, because Windows searches the system PATH before
+the user PATH, users cannot override a system-wide installation of
+Python on their PATH. Further, because the installer can only prepend to
+the path, later installations of Python will take precedence over
+earlier installations, regardless of interpreter version.
+
+Because it is not possible to automatically create a sensible PATH
+configuration, users are recommended to use the py.exe launcher and
+manually modify their PATH variable to add Scripts directories in their
+preferred order. System-wide installations of Python should consider not
+modifying PATH, or using an alternative technology to modify their
+users' PATH variables.
+
+
+The py.exe launcher is recommended because it uses a consistent and
+sensible search order for Python installations. User installations are
+preferred over system-wide installs, and later versions are preferred
+regardless of installation order (with the exception that py.exe
+currently prefers 2.x versions over 3.x versions without the -3 command
+line argument).
+
+For both 32-bit and 64-bit interpreters, the 32-bit version of the
+launcher is installed. This ensures that the search order is always
+consistent (as the 64-bit launcher is subtly different from the 32-bit
+launcher) and also avoids the need to install it multiple times. Future
+versions of Python will upgrade the launcher in-place, using Windows
+Installer's upgrade functionality to avoid conflicts with earlier
+installed versions.
+
+When installed, file associations are created for .py, .pyc and .pyo
+files to launch with py.exe and .pyw files to launch with pyw.exe. This
+makes Python files respect shebang lines by default and also avoids
+conflicts between multiple Python installations.
+
+
+Repair, Modify and Uninstall
+============================
+
+After installation, Python may be modified, repaired or uninstalled by
+running the original EXE again or via the Programs and Features applet
+(formerly known as Add or Remove Programs).
+
+Modifications allow features to be added or removed. The install
+directory and kind (all users/single user) cannot be modified. Because
+Windows Installer caches installation packages, removing features will
+not require internet access unless the package cache has been corrupted
+or deleted. Adding features that were not previously installed and are
+not embedded or otherwise available will require internet access.
+
+Repairing will rerun the installation for all currently installed
+features, restoring files and registry keys that have been modified or
+removed. This operation generally will not redownload any files unless
+the cached packages have been corrupted or deleted.
+
+Removing Python will clean up all the files and registry keys that were
+created by the installer, as well as __pycache__ folders that are
+explicitly handled by the installer. Python packages installed later
+using a tool like pip will not be removed. Some components may be
+installed by other installers (such as the MSVCRT) and these will not be
+removed if another product has a dependency on them.
+
diff --git a/Tools/msi/build.bat b/Tools/msi/build.bat
new file mode 100644
--- /dev/null
+++ b/Tools/msi/build.bat
@@ -0,0 +1,46 @@
+ at echo off
+setlocal
+set D=%~dp0
+set PCBUILD=%D%..\..\PCBuild\
+
+set BUILDX86=
+set BUILDX64=
+set BUILDDOC=
+
+:CheckOpts
+if '%1'=='-x86' (set BUILDX86=1) && shift && goto CheckOpts
+if '%1'=='-x64' (set BUILDX64=1) && shift && goto CheckOpts
+if '%1'=='--doc' (set BUILDDOC=1) && shift && goto CheckOpts
+
+if not defined BUILDX86 if not defined BUILDX64 (set BUILDX86=1) && (set BUILDX64=1)
+
+call "%PCBUILD%env.bat" x86
+
+if defined BUILDX86 (
+    call "%PCBUILD%build.bat" -d
+    if errorlevel 1 goto :eof
+    call "%PCBUILD%build.bat"
+    if errorlevel 1 goto :eof
+)
+if defined BUILDX64 (
+    call "%PCBUILD%build.bat" -p x64 -d
+    if errorlevel 1 goto :eof
+    call "%PCBUILD%build.bat" -p x64
+    if errorlevel 1 goto :eof
+)
+
+if defined BUILDDOC (
+    call "%PCBUILD%..\Doc\make.bat" htmlhelp
+    if errorlevel 1 goto :eof
+)
+
+if defined BUILDX86 (
+    "%PCBUILD%win32\python.exe" "%D%get_wix.py"
+    msbuild "%D%bundle\snapshot.wixproj"
+    if errorlevel 1 goto :eof
+)
+if defined BUILDX64 (
+    "%PCBUILD%amd64\python.exe" "%D%get_wix.py"
+    msbuild "%D%bundle\snapshot.wixproj" /p:Platform=x64
+    if errorlevel 1 goto :eof
+)
diff --git a/Tools/msi/buildrelease.bat b/Tools/msi/buildrelease.bat
new file mode 100644
--- /dev/null
+++ b/Tools/msi/buildrelease.bat
@@ -0,0 +1,131 @@
+ at setlocal
+ at echo off
+
+rem This script is intended for building official releases of Python.
+rem To use it to build alternative releases, you should clone this file
+rem and modify the following three URIs.
+rem
+rem The first two will ensure that your release can be installed
+rem alongside an official Python release, while the second specifies
+rem the URL that will be used to download installation files. The
+rem files available from this URL *will* conflict with your installer.
+rem Trust me, you don't want them, even if it seems like a good idea.
+
+set RELEASE_URI_X86=http://www.python.org/win32
+set RELEASE_URI_X64=http://www.python.org/amd64
+set DOWNLOAD_URL_BASE=https://www.python.org/ftp/python
+set DOWNLOAD_URL=
+
+set D=%~dp0
+set PCBUILD=%D%..\..\PCBuild\
+
+set BUILDX86=
+set BUILDX64=
+set TARGET=Rebuild
+
+
+:CheckOpts
+if "%1" EQU "-c" (set CERTNAME=%~2) && shift && shift && goto CheckOpts
+if "%1" EQU "-o" (set OUTDIR=%~2) && shift && shift && goto CheckOpts
+if "%1" EQU "-D" (set SKIPDOC=1) && shift && goto CheckOpts
+if "%1" EQU "-B" (set SKIPBUILD=1) && shift && goto CheckOpts
+if "%1" EQU "--download" (set DOWNLOAD_URL=%~2) && shift && shift && goto CheckOpts
+if "%1" EQU "-b" (set TARGET=Build) && shift && goto CheckOpts
+if '%1' EQU '-x86' (set BUILDX86=1) && shift && goto CheckOpts
+if '%1' EQU '-x64' (set BUILDX64=1) && shift && goto CheckOpts
+
+if not defined BUILDX86 if not defined BUILDX64 (set BUILDX86=1) && (set BUILDX64=1)
+
+:builddoc
+if "%SKIPBUILD%" EQU "1" goto skipdoc
+if "%SKIPDOC%" EQU "1" goto skipdoc
+
+call "%D%..\..\doc\make.bat" htmlhelp
+if errorlevel 1 goto :eof
+:skipdoc
+
+where dlltool 2>nul >"%TEMP%\dlltool.loc"
+if errorlevel 1 dir "%D%..\..\externals\dlltool.exe" /s/b > "%TEMP%\dlltool.loc"
+if errorlevel 1 echo Cannot find binutils on PATH or in externals & exit /B 1
+set /P DLLTOOL= < "%TEMP%\dlltool.loc"
+set PATH=%PATH%;%DLLTOOL:~,-12%
+set DLLTOOL=
+del "%TEMP%\dlltool.loc"
+
+
+if defined BUILDX86 (
+    call :build x86
+    if errorlevel 1 exit /B
+)
+
+if defined BUILDX64 (
+    call :build x64
+    if errorlevel 1 exit /B
+)
+
+exit /B 0
+
+:build
+ at setlocal
+ at echo off
+
+if "%1" EQU "x86" (
+    call "%PCBUILD%env.bat" x86
+    set BUILD=%PCBUILD%win32\
+    set BUILD_PLAT=Win32
+    set OUTDIR_PLAT=win32
+    set OBJDIR_PLAT=x86
+    set RELEASE_URI=%RELEASE_URI_X86%
+) ELSE (
+    call "%PCBUILD%env.bat" x86_amd64
+    set BUILD=%PCBUILD%amd64\
+    set BUILD_PLAT=x64
+    set OUTDIR_PLAT=amd64
+    set OBJDIR_PLAT=x64
+    set RELEASE_URI=%RELEASE_URI_X64%
+)
+
+echo on
+if exist "%BUILD%en-us" (
+    echo Deleting %BUILD%en-us
+    rmdir /q/s "%BUILD%en-us"
+    if errorlevel 1 exit /B
+)
+
+echo on
+if exist "%D%obj\Release_%OBJDIR_PLAT%" (
+    echo Deleting "%D%obj\Release_%OBJDIR_PLAT%"
+    rmdir /q/s "%D%obj\Release_%OBJDIR_PLAT%"
+    if errorlevel 1 exit /B
+)
+
+if not "%CERTNAME%" EQU "" (
+    set CERTOPTS="/p:SigningCertificate=%CERTNAME%"
+) else (
+    set CERTOPTS=
+)
+
+if not "%SKIPBUILD%" EQU "1" (
+    call "%PCBUILD%build.bat" -e -p %BUILD_PLAT% -d -t %TARGET% %CERTOPTS%
+    if errorlevel 1 exit /B
+    call "%PCBUILD%build.bat" -p %BUILD_PLAT% -t %TARGET% %CERTOPTS%
+    if errorlevel 1 exit /B
+    @rem build.bat turns echo back on, so we disable it again
+    @echo off
+)
+
+"%BUILD%python.exe" "%D%get_wix.py"
+
+set BUILDOPTS=/p:Platform=%1 /p:BuildForRelease=true /p:DownloadUrl=%DOWNLOAD_URL% /p:DownloadUrlBase=%DOWNLOAD_URL_BASE% /p:ReleaseUri=%RELEASE_URI%
+msbuild "%D%bundle\releaselocal.wixproj" /t:Rebuild %BUILDOPTS% %CERTOPTS% /p:RebuildAll=true
+if errorlevel 1 exit /B
+msbuild "%D%bundle\releaseweb.wixproj" /t:Rebuild %BUILDOPTS% %CERTOPTS% /p:RebuildAll=false
+if errorlevel 1 exit /B
+
+if not "%OUTDIR%" EQU "" (
+    mkdir "%OUTDIR%\%OUTDIR_PLAT%"
+    copy /Y "%BUILD%en-us\*.cab" "%OUTDIR%\%OUTDIR_PLAT%"
+    copy /Y "%BUILD%en-us\*.exe" "%OUTDIR%\%OUTDIR_PLAT%"
+    copy /Y "%BUILD%en-us\*.msi" "%OUTDIR%\%OUTDIR_PLAT%"
+)
+
diff --git a/Tools/msi/bundle/Default.thm b/Tools/msi/bundle/Default.thm
new file mode 100644
--- /dev/null
+++ b/Tools/msi/bundle/Default.thm
@@ -0,0 +1,123 @@
+<?xml version="1.0" encoding="utf-8"?>
+<Theme xmlns="http://wixtoolset.org/schemas/thmutil/2010">
+    <Window Width="640" Height="382" HexStyle="100a0000" FontId="0">#(loc.Caption)</Window>
+    <Font Id="0" Height="-12" Weight="500" Foreground="000000" Background="ffffff">Segoe UI</Font>
+    <Font Id="1" Height="-24" Weight="500" Foreground="000000" Background="ffffff">Segoe UI</Font>
+    <Font Id="2" Height="-22" Weight="500" Foreground="666666" Background="ffffff">Segoe UI</Font>
+    <Font Id="3" Height="-12" Weight="500" Foreground="000000" Background="ffffff">Segoe UI</Font>
+    <Font Id="4" Height="-12" Weight="500" Foreground="ff0000" Background="ffffff" Underline="yes">Segoe UI</Font>
+    <Font Id="5" Height="-12" Weight="500" Foreground="666666" Background="ffffff">Segoe UI</Font>
+
+    <Page Name="Help">
+        <Text X="185" Y="11" Width="-11" Height="32" FontId="1" DisablePrefix="yes">#(loc.HelpHeader)</Text>
+        <Image X="0" Y="0" Width="162" Height="352" ImageFile="SideBar.png"/>
+
+        <Text X="185" Y="50" Width="-11" Height="-35" FontId="3" DisablePrefix="yes">#(loc.HelpText)</Text>
+        <Button Name="SuccessCancelButton" X="-11" Y="-11" Width="75" Height="23" TabStop="yes" FontId="0">#(loc.CloseButton)</Button>
+    </Page>
+    <Page Name="Install">
+        <Text X="185" Y="11" Width="-11" Height="32" FontId="1" DisablePrefix="yes">#(loc.InstallHeader)</Text>
+        <Image X="0" Y="0" Width="162" Height="352" ImageFile="SideBar.png"/>
+
+        <Text X="185" Y="50" Width="-11" Height="50" FontId="3" TabStop="yes">#(loc.InstallMessage)</Text>
+
+        <Button Name="InstallAllUsersButton" X="175" Y="101" Width="-11" Height="59" TabStop="yes" FontId="3" HexStyle="0xF">#(loc.InstallAllUsersButton)</Button>
+        <Button Name="InstallJustForMeButton" X="175" Y="171" Width="-11" Height="59" TabStop="yes" FontId="3" HexStyle="0xE">#(loc.InstallJustForMeButton)</Button>
+        <Button Name="InstallCustomButton" X="175" Y="241" Width="-11" Height="59" TabStop="yes" FontId="3" HexStyle="0xE">#(loc.InstallCustomButton)</Button>
+
+        <Checkbox Name="PrependPath" X="185" Y="-13" Width="-100" Height="20" TabStop="yes" FontId="3">#(loc.ShortPrependPathLabel)</Checkbox>
+
+        <Button Name="InstallCancelButton" X="-11" Y="-11" Width="75" Height="23" TabStop="yes" FontId="0">#(loc.CancelButton)</Button>
+    </Page>
+    <Page Name="SimpleInstall">
+        <Text X="185" Y="11" Width="-11" Height="32" FontId="1" DisablePrefix="yes">#(loc.InstallHeader)</Text>
+        <Image X="0" Y="0" Width="162" Height="352" ImageFile="SideBar.png"/>
+
+        <Button Name="InstallSimpleButton" X="175" Y="101" Width="-11" Height="59" TabStop="yes" FontId="3" HideWhenDisabled="yes" HexStyle="0xF">#(loc.InstallSimpleButton)</Button>
+
+        <Button Name="InstallCancelButton" X="-11" Y="-11" Width="75" Height="23" TabStop="yes" FontId="0">#(loc.CancelButton)</Button>
+    </Page>
+    <Page Name="Custom1">
+        <Text X="185" Y="11" Width="-11" Height="32" FontId="1" DisablePrefix="yes">#(loc.Custom1Header)</Text>
+        <Image X="0" Y="0" Width="162" Height="352" ImageFile="SideBar.png"/>
+        
+        <Checkbox Name="Include_doc" X="185" Y="51" Width="-11" Height="20" TabStop="yes" FontId="3" HideWhenDisabled="yes">#(loc.Include_docLabel)</Checkbox>
+        <Text X="205" Y="71" Width="-11" Height="20" TabStop="no" FontId="5">#(loc.Include_docHelpLabel)</Text>
+        
+        <Checkbox Name="Include_pip" X="185" Y="96" Width="-11" Height="20" TabStop="yes" FontId="3" HideWhenDisabled="yes">#(loc.Include_pipLabel)</Checkbox>
+        <Text X="205" Y="116" Width="-11" Height="20" TabStop="no" FontId="5">#(loc.Include_pipHelpLabel)</Text>
+        
+        <Checkbox Name="Include_tcltk" X="185" Y="141" Width="-11" Height="20" TabStop="yes" FontId="3" HideWhenDisabled="yes">#(loc.Include_tcltkLabel)</Checkbox>
+        <Text X="205" Y="161" Width="-11" Height="20" TabStop="no" FontId="5">#(loc.Include_tcltkHelpLabel)</Text>
+        
+        <Checkbox Name="Include_test" X="185" Y="186" Width="-11" Height="20" TabStop="yes" FontId="3" HideWhenDisabled="yes">#(loc.Include_testLabel)</Checkbox>
+        <Text X="205" Y="206" Width="-11" Height="20" TabStop="no" FontId="5">#(loc.Include_testHelpLabel)</Text>
+
+        <Checkbox Name="Include_launcher" X="185" Y="231" Width="-11" Height="20" TabStop="yes" FontId="3" HideWhenDisabled="yes">#(loc.Include_launcherLabel)</Checkbox>
+        <Text X="205" Y="251" Width="-11" Height="20" TabStop="no" FontId="5">#(loc.Include_launcherHelpLabel)</Text>
+
+        <Button Name="Custom1BackButton" X="185" Y="-11" Width="75" Height="23" TabStop="yes" FontId="0">#(loc.CustomBackButton)</Button>
+        <Button Name="CustomNextButton" X="-91" Y="-11" Width="75" Height="23" TabStop="yes" FontId="0">#(loc.CustomNextButton)</Button>
+        <Button Name="Custom1CancelButton" X="-11" Y="-11" Width="75" Height="23" TabStop="yes" FontId="0">#(loc.CancelButton)</Button>
+    </Page>
+    <Page Name="Custom2">
+        <Text X="185" Y="11" Width="-11" Height="32" FontId="1" DisablePrefix="yes">#(loc.Custom2Header)</Text>
+        <Image X="0" Y="0" Width="162" Height="352" ImageFile="SideBar.png"/>
+        
+        <Checkbox Name="AssociateFiles" X="185" Y="51" Width="-11" Height="20" TabStop="yes" FontId="3" HideWhenDisabled="no">#(loc.AssociateFilesLabel)</Checkbox>
+        <Checkbox Name="PrependPath" X="185" Y="76" Width="-11" Height="20" TabStop="yes" FontId="3" HideWhenDisabled="no">#(loc.PrependPathLabel)</Checkbox>
+        <Checkbox Name="InstallAllUsers" X="185" Y="101" Width="-11" Height="20" TabStop="yes" FontId="3" HideWhenDisabled="no">#(loc.InstallAllUsersLabel)</Checkbox>
+        <Checkbox Name="CompileAll" X="185" Y="126" Width="-11" Height="20" TabStop="yes" FontId="3" HideWhenDisabled="no">#(loc.PrecompileLabel)</Checkbox>
+        <Checkbox Name="Include_symbols" X="185" Y="151" Width="-11" Height="20" TabStop="yes" FontId="3" HideWhenDisabled="no">#(loc.Include_symbolsLabel)</Checkbox>
+        <Checkbox Name="Include_debug" X="185" Y="176" Width="-11" Height="20" TabStop="yes" FontId="3" HideWhenDisabled="no">#(loc.Include_debugLabel)</Checkbox>
+
+        <Text X="185" Y="221" Width="-11" Height="17" FontId="3">#(loc.CustomLocationLabel)</Text>
+        <Editbox Name="TargetDir" X="185" Y="237" Width="-91" Height="21" TabStop="yes" FontId="3" FileSystemAutoComplete="yes" />
+        <Button Name="CustomBrowseButton" X="-11" Y="236" Width="75" Height="23" TabStop="yes" FontId="3">#(loc.CustomBrowseButton)</Button>
+        <Text X="185" Y="261" Width="-91" Height="35" FontId="5">#(loc.CustomLocationHelpLabel)</Text>
+
+        <Button Name="Custom2BackButton" X="185" Y="-11" Width="75" Height="23" TabStop="yes" FontId="0">#(loc.CustomBackButton)</Button>
+        <Button Name="CustomInstallButton" X="-91" Y="-11" Width="95" Height="23" TabStop="yes" FontId="0">#(loc.CustomInstallButton)</Button>
+        <Button Name="Custom2CancelButton" X="-11" Y="-11" Width="75" Height="23" TabStop="yes" FontId="0">#(loc.CancelButton)</Button>
+    </Page>
+    <Page Name="Progress">
+        <Text X="185" Y="11" Width="-11" Height="32" FontId="1" DisablePrefix="yes">#(loc.ProgressHeader)</Text>
+        <Image X="0" Y="0" Width="162" Height="352" ImageFile="SideBar.png"/>
+
+        <Text X="185" Y="111" Width="70" Height="17" FontId="3" DisablePrefix="yes">#(loc.ProgressLabel)</Text>
+        <Text Name="OverallProgressPackageText" X="185" Y="141" Width="-11" Height="17" FontId="3" DisablePrefix="yes">#(loc.OverallProgressPackageText)</Text>
+        <Progressbar Name="OverallCalculatedProgressbar" X="185" Y="163" Width="-11" Height="20" />
+        <Button Name="ProgressCancelButton" X="-11" Y="-11" Width="75" Height="23" TabStop="yes" FontId="0">#(loc.CancelButton)</Button>
+    </Page>
+    <Page Name="Modify">
+        <Text X="185" Y="11" Width="-11" Height="32" FontId="1" DisablePrefix="yes">#(loc.ModifyHeader)</Text>
+        <Image X="0" Y="0" Width="162" Height="352" ImageFile="SideBar.png"/>
+
+        <Button Name="ModifyButton" X="175" Y="101" Width="-11" Height="59" TabStop="yes" FontId="3" HexStyle="0xF">#(loc.ModifyModifyButton)</Button>
+        <Button Name="RepairButton" X="175" Y="171" Width="-11" Height="59" TabStop="yes" FontId="3" HexStyle="0xE">#(loc.ModifyRepairButton)</Button>
+        <Button Name="UninstallButton" X="175" Y="241" Width="-11" Height="59" TabStop="yes" FontId="3" HexStyle="0xE">#(loc.ModifyUninstallButton)</Button>
+
+        <Button Name="ModifyCancelButton" X="-11" Y="-11" Width="75" Height="23" TabStop="yes" FontId="0">#(loc.CancelButton)</Button>
+    </Page>
+    <Page Name="Success">
+        <Text X="185" Y="11" Width="-11" Height="32" FontId="1" DisablePrefix="yes">#(loc.SuccessHeader)</Text>
+        <Image X="0" Y="0" Width="162" Height="352" ImageFile="SideBar.png"/>
+
+        <Hypertext Name="SuccessText" X="205" Y="71" Width="-31" Height="100" FontId="3" DisablePrefix="yes"></Hypertext>
+
+        <Text Name="SuccessRestartText" X="185" Y="-100" Width="-11" Height="34" FontId="3" HideWhenDisabled="yes" DisablePrefix="yes">#(loc.SuccessRestartText)</Text>
+        <Button Name="LaunchButton" X="185" Y="-50" Width="-11" Height="59" TabStop="yes" FontId="3" HexStyle="0xF" HideWhenDisabled="yes">#(loc.SuccessLaunchButton)</Button>
+        <Button Name="SuccessRestartButton" X="-91" Y="-11" Width="75" Height="23" TabStop="yes" FontId="0" HideWhenDisabled="yes">#(loc.SuccessRestartButton)</Button>
+        <Button Name="SuccessCancelButton" X="-11" Y="-11" Width="75" Height="23" TabStop="yes" FontId="0">#(loc.CloseButton)</Button>
+    </Page>
+    <Page Name="Failure">
+        <Text X="185" Y="11" Width="-11" Height="32" FontId="1" DisablePrefix="yes">#(loc.FailureHeader)</Text>
+        <Image X="0" Y="0" Width="162" Height="352" ImageFile="SideBar.png"/>
+
+        <Hypertext Name="FailureLogFileLink" X="185" Y="71" Width="-11" Height="60" FontId="3" TabStop="yes" HideWhenDisabled="yes">#(loc.FailureHyperlinkLogText)</Hypertext>
+        <Hypertext Name="FailureMessageText" X="185" Y="-100" Width="-11" Height="34" FontId="3" TabStop="yes" HideWhenDisabled="yes">Failure Message</Hypertext>
+        <Text Name="FailureRestartText" X="185" Y="-40" Width="-11" Height="34" FontId="3" HideWhenDisabled="yes" DisablePrefix="yes">#(loc.FailureRestartText)</Text>
+        <Button Name="FailureRestartButton" X="-91" Y="-11" Width="75" Height="23" TabStop="yes" FontId="0" HideWhenDisabled="yes">#(loc.FailureRestartButton)</Button>
+        <Button Name="FailureCancelButton" X="-11" Y="-11" Width="75" Height="23" TabStop="yes" FontId="0">#(loc.CloseButton)</Button>
+    </Page>
+</Theme>
\ No newline at end of file
diff --git a/Tools/msi/bundle/Default.wxl b/Tools/msi/bundle/Default.wxl
new file mode 100644
--- /dev/null
+++ b/Tools/msi/bundle/Default.wxl
@@ -0,0 +1,103 @@
+?<?xml version="1.0" encoding="utf-8"?>
+<WixLocalization Culture="en-us" Language="1033" xmlns="http://schemas.microsoft.com/wix/2006/localization">
+  <String Id="Caption">[WixBundleName] Setup</String>
+  <String Id="Title">[WixBundleName]</String>
+  <String Id="Installing">Installing</String>
+  <String Id="Installation">Setup</String>
+  <String Id="Modifying">Updating</String>
+  <String Id="Modification">Modify</String>
+  <String Id="Repairing">Repairing</String>
+  <String Id="Repair">Repair</String>
+  <String Id="Uninstalling">Removing</String>
+  <String Id="Uninstallation">Uninstall</String>
+  
+  <String Id="CancelButton">&amp;Cancel</String>
+  <String Id="CloseButton">&amp;Close</String>
+  <String Id="InstallHeader">Install [WixBundleName]</String>
+  <String Id="InstallMessage">Select to location to install Python, or choose Customize to enable or disable features.</String>
+  <String Id="InstallVersion">Version [WixBundleVersion]</String>
+  <String Id="ConfirmCancelMessage">Are you sure you want to cancel?</String>
+  <String Id="ExecuteUpgradeRelatedBundleMessage">Previous version</String>
+  <String Id="HelpHeader">Setup Help</String>
+  <String Id="HelpText">/uninstall
+    Uninstalls Python without prompting for confirmation.
+
+/layout [\[]directory[\]]
+    Downloads all components for offline installation.
+
+/passive
+    Displays progress without requiring user interaction.
+
+/quiet
+    Performs the requested action without displaying any UI.
+
+/log [\[]filename[\]]
+    Logs to a specific file. By default, log files are created in %TEMP%.</String>
+  <String Id="InstallLicenseLinkText">[WixBundleName] &lt;a href="#"&gt;license terms&lt;/a&gt;.</String>
+  <String Id="InstallAcceptCheckbox">I &amp;agree to the license terms and conditions</String>
+  <String Id="InstallAllUsersButton">Install for &amp;All Users</String>
+  <String Id="InstallAllUsersButtonNote">[DefaultAllUsersTargetDir]</String>
+  <String Id="InstallJustForMeButton">Install &amp;Just for Me</String>
+  <String Id="InstallJustForMeButtonNote">[DefaultJustForMeTargetDir]</String>
+  <String Id="InstallCustomButton">C&amp;ustomize installation</String>
+  <String Id="InstallCustomButtonNote">Choose location and features</String>
+  <String Id="InstallSimpleButton">&amp;Install</String>
+  <String Id="InstallSimpleButtonNote">Uses setting preselected by your administrator</String>
+  <String Id="Custom1Header">Optional Features</String>
+  <String Id="Custom2Header">Advanced Options</String>
+  <String Id="CustomLocationLabel">Customize install location</String>
+  <String Id="CustomLocationHelpLabel">If not installing as administrator, you will require write permissions for this location.</String>
+  <String Id="CustomInstallButton">&amp;Install</String>
+  <String Id="CustomNextButton">&amp;Next</String>
+  <String Id="CustomBackButton">&amp;Back</String>
+  <String Id="CustomBrowseButton">B&amp;rowse</String>
+  <String Id="Include_docLabel">&amp;Documentation</String>
+  <String Id="Include_docHelpLabel">Installs the Python documentation file.</String>
+  <String Id="Include_pipLabel">&amp;pip</String>
+  <String Id="Include_pipHelpLabel">Installs pip, which can download and install other Python packages.</String>
+  <String Id="Include_tcltkLabel">tcl/tk and &amp;IDLE</String>
+  <String Id="Include_tcltkHelpLabel">Installs tkinter and the IDLE development environment.</String>
+  <String Id="Include_testLabel">Python &amp;test suite</String>
+  <String Id="Include_testHelpLabel">Installs the standard library test suite.</String>
+  <String Id="Include_launcherLabel">py &amp;launcher</String>
+  <String Id="Include_launcherHelpLabel">Installs the global 'py' launcher to make it easier to start Python.</String>
+  
+  <String Id="AssociateFilesLabel">Associate &amp;files with Python (requires the py launcher)</String>
+  <String Id="PrependPathLabel">Add Python to &amp;environment variables</String>
+  <String Id="ShortPrependPathLabel">Add &amp;Python [ShortVersion] to PATH</String>
+  <String Id="InstallAllUsersLabel">Install as &amp;Administrator</String>
+  <String Id="PrecompileLabel">&amp;Precompile standard library</String>
+  <String Id="Include_symbolsLabel">Install debugging &amp;symbols</String>
+  <String Id="Include_debugLabel">Install debu&amp;g binaries</String>
+  
+  <String Id="ProgressHeader">[ActionLikeInstallation] Progress</String>
+  <String Id="ProgressLabel">[ActionLikeInstalling]:</String>
+  <String Id="OverallProgressPackageText">Initializing...</String>
+  <String Id="ModifyHeader">Modify Setup</String>
+  <String Id="ModifyModifyButton">&amp;Modify</String>
+  <String Id="ModifyButtonNote">Add or remove individual features.</String>
+  <String Id="ModifyRepairButton">&amp;Repair</String>
+  <String Id="RepairButtonNote">Ensure all current features are correctly installed.</String>
+  <String Id="ModifyUninstallButton">&amp;Uninstall</String>
+  <String Id="UninstallButtonNote">Remove the entire [WixBundleName] installation.</String>
+  <String Id="SuccessHeader">[ActionLikeInstallation] was successful</String>
+  <String Id="SuccessLaunchButton">&amp;Launch</String>
+  <String Id="SuccessRestartText">You may need to restart your computer to finish updating files.</String>
+  <String Id="SuccessRestartButton">&amp;Restart</String>
+  <String Id="SuccessInstallMessage">Special thanks to Mark Hammond, without whose years of freely shared Windows expertise, Python for Windows would still be Python for DOS.
+
+New to Python? Start with the &lt;a href="https://docs.python.org/[ShortVersion]/tutorial/index.html"&gt;online tutorial&lt;/a&gt; and &lt;a href="https://docs.python.org/[ShortVersion]/index.html"&gt;documentation&lt;/a&gt;.
+
+See &lt;a href="https://docs.python.org/[ShortVersion]/whatsnew/[ShortVersion].html"&gt;what's new&lt;/a&gt; in this release.</String>
+  <String Id="SuccessModifyMessage">Thank you for using [WixBundleName].</String>
+  <String Id="SuccessRepairMessage">Thank you for using [WixBundleName].
+
+Feel free to email &lt;a href="mailto:python-list at python.org"&gt;python-list at python.org&lt;/a&gt; if you continue to encounter issues.</String>
+  <String Id="SuccessRemoveMessage">Thank you for using [WixBundleName].
+
+Feel free to email &lt;a href="mailto:python-list at python.org"&gt;python-list at python.org&lt;/a&gt; if you encountered problems.</String>
+  <String Id="FailureHeader">Setup failed</String>
+  <String Id="FailureHyperlinkLogText">One or more issues caused the setup to fail. Please fix the issues and then retry setup. For more information see the &lt;a href="#"&gt;log file&lt;/a&gt;.</String>
+  <String Id="FailureRestartText">You must restart your computer to complete the rollback of the software.</String>
+  <String Id="FailureRestartButton">&amp;Restart</String>
+</WixLocalization>
diff --git a/Tools/msi/bundle/SideBar.png b/Tools/msi/bundle/SideBar.png
new file mode 100644
index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..9c18fff33a6ea50e3514ea938c5e5bfe4c5fe4c6
GIT binary patch
[stripped]
diff --git a/Tools/msi/bundle/bootstrap/LICENSE.txt b/Tools/msi/bundle/bootstrap/LICENSE.txt
new file mode 100644
--- /dev/null
+++ b/Tools/msi/bundle/bootstrap/LICENSE.txt
@@ -0,0 +1,25 @@
+This license applies to the bootstrapper application that is embedded within the installer. It has no impact on the licensing for the rest of the installer or Python itself, as no code covered by this license exists in any other part of the product.
+
+---
+
+Microsoft Reciprocal License (MS-RL)
+
+This license governs use of the accompanying software. If you use the software, you accept this license. If you do not accept the license, do not use the software.
+
+1. Definitions
+ The terms "reproduce," "reproduction," "derivative works," and "distribution" have the same meaning here as under U.S. copyright law.
+ A "contribution" is the original software, or any additions or changes to the software.
+ A "contributor" is any person that distributes its contribution under this license.
+ "Licensed patents" are a contributor's patent claims that read directly on its contribution.
+
+2. Grant of Rights
+ (A) Copyright Grant- Subject to the terms of this license, including the license conditions and limitations in section 3, each contributor grants you a non-exclusive, worldwide, royalty-free copyright license to reproduce its contribution, prepare derivative works of its contribution, and distribute its contribution or any derivative works that you create.
+ (B) Patent Grant- Subject to the terms of this license, including the license conditions and limitations in section 3, each contributor grants you a non-exclusive, worldwide, royalty-free license under its licensed patents to make, have made, use, sell, offer for sale, import, and/or otherwise dispose of its contribution in the software or derivative works of the contribution in the software.
+
+3. Conditions and Limitations
+ (A) Reciprocal Grants- For any file you distribute that contains code from the software (in source code or binary format), you must provide recipients the source code to that file along with a copy of this license, which license will govern that file. You may license other files that are entirely your own work and do not contain code from the software under any terms you choose.
+ (B) No Trademark License- This license does not grant you rights to use any contributors' name, logo, or trademarks.
+ (C) If you bring a patent claim against any contributor over patents that you claim are infringed by the software, your patent license from such contributor to the software ends automatically.
+ (D) If you distribute any portion of the software, you must retain all copyright, patent, trademark, and attribution notices that are present in the software.
+ (E) If you distribute any portion of the software in source code form, you may do so only under this license by including a complete copy of this license with your distribution. If you distribute any portion of the software in compiled or object code form, you may only do so under a license that complies with this license.
+ (F) The software is licensed "as-is." You bear the risk of using it. The contributors give no express warranties, guarantees or conditions. You may have additional consumer rights under your local laws which this license cannot change. To the extent permitted under your local laws, the contributors exclude the implied warranties of merchantability, fitness for a particular purpose and non-infringement.
diff --git a/Tools/msi/bundle/bootstrap/PythonBootstrapperApplication.cpp b/Tools/msi/bundle/bootstrap/PythonBootstrapperApplication.cpp
new file mode 100644
--- /dev/null
+++ b/Tools/msi/bundle/bootstrap/PythonBootstrapperApplication.cpp
@@ -0,0 +1,2645 @@
+//-------------------------------------------------------------------------------------------------
+// <copyright file="WixStandardBootstrapperApplication.cpp" company="Outercurve Foundation">
+//   Copyright (c) 2004, Outercurve Foundation.
+//   This software is released under Microsoft Reciprocal License (MS-RL).
+//   The license and further copyright text can be found in the file
+//   LICENSE.TXT at the root directory of the distribution.
+// </copyright>
+//-------------------------------------------------------------------------------------------------
+
+
+#include "pch.h"
+
+static const LPCWSTR WIXBUNDLE_VARIABLE_ELEVATED = L"WixBundleElevated";
+
+static const LPCWSTR PYBA_WINDOW_CLASS = L"PythonBA";
+static const LPCWSTR PYBA_VARIABLE_LAUNCH_TARGET_PATH = L"LaunchTarget";
+static const LPCWSTR PYBA_VARIABLE_LAUNCH_TARGET_ELEVATED_ID = L"LaunchTargetElevatedId";
+static const LPCWSTR PYBA_VARIABLE_LAUNCH_ARGUMENTS = L"LaunchArguments";
+static const LPCWSTR PYBA_VARIABLE_LAUNCH_HIDDEN = L"LaunchHidden";
+static const DWORD PYBA_ACQUIRE_PERCENTAGE = 30;
+static const LPCWSTR PYBA_VARIABLE_BUNDLE_FILE_VERSION = L"WixBundleFileVersion";
+
+enum PYBA_STATE {
+    PYBA_STATE_INITIALIZING,
+    PYBA_STATE_INITIALIZED,
+    PYBA_STATE_HELP,
+    PYBA_STATE_DETECTING,
+    PYBA_STATE_DETECTED,
+    PYBA_STATE_PLANNING,
+    PYBA_STATE_PLANNED,
+    PYBA_STATE_APPLYING,
+    PYBA_STATE_CACHING,
+    PYBA_STATE_CACHED,
+    PYBA_STATE_EXECUTING,
+    PYBA_STATE_EXECUTED,
+    PYBA_STATE_APPLIED,
+    PYBA_STATE_FAILED,
+};
+
+static const int WM_PYBA_SHOW_HELP = WM_APP + 100;
+static const int WM_PYBA_DETECT_PACKAGES = WM_APP + 101;
+static const int WM_PYBA_PLAN_PACKAGES = WM_APP + 102;
+static const int WM_PYBA_APPLY_PACKAGES = WM_APP + 103;
+static const int WM_PYBA_CHANGE_STATE = WM_APP + 104;
+static const int WM_PYBA_SHOW_FAILURE = WM_APP + 105;
+
+// This enum must be kept in the same order as the PAGE_NAMES array.
+enum PAGE {
+    PAGE_LOADING,
+    PAGE_HELP,
+    PAGE_INSTALL,
+    PAGE_SIMPLE_INSTALL,
+    PAGE_CUSTOM1,
+    PAGE_CUSTOM2,
+    PAGE_MODIFY,
+    PAGE_PROGRESS,
+    PAGE_PROGRESS_PASSIVE,
+    PAGE_SUCCESS,
+    PAGE_FAILURE,
+    COUNT_PAGE,
+};
+
+// This array must be kept in the same order as the PAGE enum.
+static LPCWSTR PAGE_NAMES[] = {
+    L"Loading",
+    L"Help",
+    L"Install",
+    L"SimpleInstall",
+    L"Custom1",
+    L"Custom2",
+    L"Modify",
+    L"Progress",
+    L"ProgressPassive",
+    L"Success",
+    L"Failure",
+};
+
+enum CONTROL_ID {
+    // Non-paged controls
+    ID_CLOSE_BUTTON = THEME_FIRST_ASSIGN_CONTROL_ID,
+    ID_MINIMIZE_BUTTON,
+
+    // Welcome page
+    ID_INSTALL_ALL_USERS_BUTTON,
+    ID_INSTALL_JUST_FOR_ME_BUTTON,
+    ID_INSTALL_CUSTOM_BUTTON,
+    ID_INSTALL_SIMPLE_BUTTON,
+    ID_INSTALL_CANCEL_BUTTON,
+    
+    // Customize Page
+    ID_TARGETDIR_EDITBOX,
+    ID_CUSTOM_ASSOCIATE_FILES_CHECKBOX,
+    ID_CUSTOM_INSTALL_ALL_USERS_CHECKBOX,
+    ID_CUSTOM_BROWSE_BUTTON,
+    ID_CUSTOM_INSTALL_BUTTON,
+    ID_CUSTOM_NEXT_BUTTON,
+    ID_CUSTOM1_BACK_BUTTON,
+    ID_CUSTOM2_BACK_BUTTON,
+    ID_CUSTOM1_CANCEL_BUTTON,
+    ID_CUSTOM2_CANCEL_BUTTON,
+
+    // Modify page
+    ID_MODIFY_BUTTON,
+    ID_REPAIR_BUTTON,
+    ID_UNINSTALL_BUTTON,
+    ID_MODIFY_CANCEL_BUTTON,
+
+    // Progress page
+    ID_CACHE_PROGRESS_PACKAGE_TEXT,
+    ID_CACHE_PROGRESS_BAR,
+    ID_CACHE_PROGRESS_TEXT,
+
+    ID_EXECUTE_PROGRESS_PACKAGE_TEXT,
+    ID_EXECUTE_PROGRESS_BAR,
+    ID_EXECUTE_PROGRESS_TEXT,
+    ID_EXECUTE_PROGRESS_ACTIONDATA_TEXT,
+
+    ID_OVERALL_PROGRESS_PACKAGE_TEXT,
+    ID_OVERALL_PROGRESS_BAR,
+    ID_OVERALL_CALCULATED_PROGRESS_BAR,
+    ID_OVERALL_PROGRESS_TEXT,
+
+    ID_PROGRESS_CANCEL_BUTTON,
+
+    // Success page
+    ID_LAUNCH_BUTTON,
+    ID_SUCCESS_TEXT,
+    ID_SUCCESS_RESTART_TEXT,
+    ID_SUCCESS_RESTART_BUTTON,
+    ID_SUCCESS_CANCEL_BUTTON,
+
+    // Failure page
+    ID_FAILURE_LOGFILE_LINK,
+    ID_FAILURE_MESSAGE_TEXT,
+    ID_FAILURE_RESTART_TEXT,
+    ID_FAILURE_RESTART_BUTTON,
+    ID_FAILURE_CANCEL_BUTTON
+};
+
+static THEME_ASSIGN_CONTROL_ID CONTROL_ID_NAMES[] = {
+    { ID_CLOSE_BUTTON, L"CloseButton" },
+    { ID_MINIMIZE_BUTTON, L"MinimizeButton" },
+
+    { ID_INSTALL_ALL_USERS_BUTTON, L"InstallAllUsersButton" },
+    { ID_INSTALL_JUST_FOR_ME_BUTTON, L"InstallJustForMeButton" },
+    { ID_INSTALL_CUSTOM_BUTTON, L"InstallCustomButton" },
+    { ID_INSTALL_SIMPLE_BUTTON, L"InstallSimpleButton" },
+    { ID_INSTALL_CANCEL_BUTTON, L"InstallCancelButton" },
+
+    { ID_TARGETDIR_EDITBOX, L"TargetDir" },
+    { ID_CUSTOM_ASSOCIATE_FILES_CHECKBOX, L"AssociateFiles" },
+    { ID_CUSTOM_INSTALL_ALL_USERS_CHECKBOX, L"InstallAllUsers" },
+    { ID_CUSTOM_BROWSE_BUTTON, L"CustomBrowseButton" },
+    { ID_CUSTOM_INSTALL_BUTTON, L"CustomInstallButton" },
+    { ID_CUSTOM_NEXT_BUTTON, L"CustomNextButton" },
+    { ID_CUSTOM1_BACK_BUTTON, L"Custom1BackButton" },
+    { ID_CUSTOM2_BACK_BUTTON, L"Custom2BackButton" },
+    { ID_CUSTOM1_CANCEL_BUTTON, L"Custom1CancelButton" },
+    { ID_CUSTOM2_CANCEL_BUTTON, L"Custom2CancelButton" },
+
+    { ID_MODIFY_BUTTON, L"ModifyButton" },
+    { ID_REPAIR_BUTTON, L"RepairButton" },
+    { ID_UNINSTALL_BUTTON, L"UninstallButton" },
+    { ID_MODIFY_CANCEL_BUTTON, L"ModifyCancelButton" },
+
+    { ID_CACHE_PROGRESS_PACKAGE_TEXT, L"CacheProgressPackageText" },
+    { ID_CACHE_PROGRESS_BAR, L"CacheProgressbar" },
+    { ID_CACHE_PROGRESS_TEXT, L"CacheProgressText" },
+    { ID_EXECUTE_PROGRESS_PACKAGE_TEXT, L"ExecuteProgressPackageText" },
+    { ID_EXECUTE_PROGRESS_BAR, L"ExecuteProgressbar" },
+    { ID_EXECUTE_PROGRESS_TEXT, L"ExecuteProgressText" },
+    { ID_EXECUTE_PROGRESS_ACTIONDATA_TEXT, L"ExecuteProgressActionDataText" },
+    { ID_OVERALL_PROGRESS_PACKAGE_TEXT, L"OverallProgressPackageText" },
+    { ID_OVERALL_PROGRESS_BAR, L"OverallProgressbar" },
+    { ID_OVERALL_CALCULATED_PROGRESS_BAR, L"OverallCalculatedProgressbar" },
+    { ID_OVERALL_PROGRESS_TEXT, L"OverallProgressText" },
+    { ID_PROGRESS_CANCEL_BUTTON, L"ProgressCancelButton" },
+
+    { ID_LAUNCH_BUTTON, L"LaunchButton" },
+    { ID_SUCCESS_TEXT, L"SuccessText" },
+    { ID_SUCCESS_RESTART_TEXT, L"SuccessRestartText" },
+    { ID_SUCCESS_RESTART_BUTTON, L"SuccessRestartButton" },
+    { ID_SUCCESS_CANCEL_BUTTON, L"SuccessCancelButton" },
+
+    { ID_FAILURE_LOGFILE_LINK, L"FailureLogFileLink" },
+    { ID_FAILURE_MESSAGE_TEXT, L"FailureMessageText" },
+    { ID_FAILURE_RESTART_TEXT, L"FailureRestartText" },
+    { ID_FAILURE_RESTART_BUTTON, L"FailureRestartButton" },
+    { ID_FAILURE_CANCEL_BUTTON, L"FailureCancelButton" },
+};
+
+class PythonBootstrapperApplication : public CBalBaseBootstrapperApplication {
+    void ShowPage(DWORD newPageId) {
+        // Process each control for special handling in the new page.
+        ProcessPageControls(ThemeGetPage(_theme, newPageId));
+
+        // Enable disable controls per-page.
+        if (_pageIds[PAGE_INSTALL] == newPageId || _pageIds[PAGE_SIMPLE_INSTALL] == newPageId) {
+            InstallPage_Show();
+        } else if (_pageIds[PAGE_CUSTOM1] == newPageId) {
+            Custom1Page_Show();
+        } else if (_pageIds[PAGE_CUSTOM2] == newPageId) {
+            Custom2Page_Show();
+        } else if (_pageIds[PAGE_MODIFY] == newPageId) {
+            ModifyPage_Show();
+        } else if (_pageIds[PAGE_SUCCESS] == newPageId) {
+            SuccessPage_Show();
+        } else if (_pageIds[PAGE_FAILURE] == newPageId) {
+            FailurePage_Show();
+        }
+
+        // Prevent repainting while switching page to avoid ugly flickering
+        _suppressPaint = TRUE;
+        ThemeShowPage(_theme, newPageId, SW_SHOW);
+        ThemeShowPage(_theme, _visiblePageId, SW_HIDE);
+        _suppressPaint = FALSE;
+        InvalidateRect(_theme->hwndParent, nullptr, TRUE);
+        _visiblePageId = newPageId;
+
+        // On the install page set the focus to the install button or
+        // the next enabled control if install is disabled
+        if (_pageIds[PAGE_INSTALL] == newPageId) {
+            ThemeSetFocus(_theme, ID_INSTALL_ALL_USERS_BUTTON);
+        } else if (_pageIds[PAGE_SIMPLE_INSTALL] == newPageId) {
+            ThemeSetFocus(_theme, ID_INSTALL_SIMPLE_BUTTON);
+        }
+    }
+
+    //
+    // Handles control clicks
+    //
+    void OnCommand(CONTROL_ID id) {
+        LPWSTR defaultDir = nullptr;
+        LPWSTR targetDir = nullptr;
+        LONGLONG elevated;
+        BOOL checked;
+        WCHAR wzPath[MAX_PATH] = { };
+        BROWSEINFOW browseInfo = { };
+        PIDLIST_ABSOLUTE pidl = nullptr;
+        HRESULT hr = S_OK;
+
+        switch(id) {
+        case ID_CLOSE_BUTTON:
+            OnClickCloseButton();
+            break;
+
+        // Install commands
+        case ID_INSTALL_SIMPLE_BUTTON:
+            hr = BalGetStringVariable(L"TargetDir", &targetDir);
+            if (FAILED(hr) || !targetDir || !*targetDir) {
+                LONGLONG installAll;
+                if (SUCCEEDED(BalGetNumericVariable(L"InstallAllUsers", &installAll)) && installAll) {
+                    hr = BalGetStringVariable(L"DefaultAllUsersTargetDir", &defaultDir);
+                    BalExitOnFailure(hr, "Failed to get the default all users install directory");
+                } else {
+                    hr = BalGetStringVariable(L"DefaultJustForMeTargetDir", &defaultDir);
+                    BalExitOnFailure(hr, "Failed to get the default per-user install directory");
+                }
+
+                if (!defaultDir || !*defaultDir) {
+                    BalLogError(E_INVALIDARG, "Default install directory is blank");
+                }
+
+                hr = BalFormatString(defaultDir, &targetDir);
+                BalExitOnFailure1(hr, "Failed to format '%ls'", defaultDir);
+
+                hr = _engine->SetVariableString(L"TargetDir", targetDir);
+                ReleaseStr(targetDir);
+                BalExitOnFailure(hr, "Failed to set install target directory");
+            } else {
+                ReleaseStr(targetDir);
+            }
+
+            OnPlan(BOOTSTRAPPER_ACTION_INSTALL);
+            break;
+
+        case ID_INSTALL_ALL_USERS_BUTTON:
+            SavePageSettings();
+            
+            hr = _engine->SetVariableNumeric(L"InstallAllUsers", 1);
+            ExitOnFailure(hr, L"Failed to set install scope");
+
+            hr = _engine->SetVariableNumeric(L"CompileAll", 1);
+            ExitOnFailure(hr, L"Failed to set compile all setting");
+
+            hr = BalGetStringVariable(L"DefaultAllUsersTargetDir", &defaultDir);
+            BalExitOnFailure(hr, "Failed to get the default all users install directory");
+
+            if (!defaultDir || !*defaultDir) {
+                BalLogError(E_INVALIDARG, "Default install directory is blank");
+            }
+
+            hr = BalFormatString(defaultDir, &targetDir);
+            BalExitOnFailure1(hr, "Failed to format '%ls'", defaultDir);
+
+            hr = _engine->SetVariableString(L"TargetDir", targetDir);
+            ReleaseStr(targetDir);
+            BalExitOnFailure(hr, "Failed to set install target directory");
+
+            OnPlan(BOOTSTRAPPER_ACTION_INSTALL);
+            break;
+
+        case ID_INSTALL_JUST_FOR_ME_BUTTON:
+            SavePageSettings();
+            
+            hr = _engine->SetVariableNumeric(L"InstallAllUsers", 0);
+            ExitOnFailure(hr, L"Failed to set install scope");
+
+            hr = BalGetStringVariable(L"DefaultJustForMeTargetDir", &defaultDir);
+            BalExitOnFailure(hr, "Failed to get the default per-user install directory");
+
+            if (!defaultDir || !*defaultDir) {
+                BalLogError(E_INVALIDARG, "Default install directory is blank");
+            }
+
+            hr = BalFormatString(defaultDir, &targetDir);
+            BalExitOnFailure1(hr, "Failed to format '%ls'", defaultDir);
+
+            hr = _engine->SetVariableString(L"TargetDir", targetDir);
+            ReleaseStr(targetDir);
+            BalExitOnFailure(hr, "Failed to set install target directory");
+
+            OnPlan(BOOTSTRAPPER_ACTION_INSTALL);
+            break;
+
+        case ID_CUSTOM1_BACK_BUTTON:
+            SavePageSettings();
+            GoToPage(PAGE_INSTALL);
+            break;
+
+        case ID_INSTALL_CUSTOM_BUTTON: __fallthrough;
+        case ID_CUSTOM2_BACK_BUTTON:
+            SavePageSettings();
+            GoToPage(PAGE_CUSTOM1);
+            break;
+
+        case ID_CUSTOM_NEXT_BUTTON:
+            SavePageSettings();
+            GoToPage(PAGE_CUSTOM2);
+            break;
+
+        case ID_CUSTOM_INSTALL_BUTTON:
+            SavePageSettings();
+
+            hr = BalGetStringVariable(L"TargetDir", &targetDir);
+            if (SUCCEEDED(hr)) {
+                // TODO: Check whether directory exists and contains another installation
+                ReleaseStr(targetDir);
+            }
+
+            OnPlan(_command.action);
+            break;
+
+        case ID_CUSTOM_INSTALL_ALL_USERS_CHECKBOX:
+            hr = BalGetNumericVariable(L"WixBundleElevated", &elevated);
+            checked = ThemeIsControlChecked(_theme, ID_CUSTOM_INSTALL_ALL_USERS_CHECKBOX);
+            ThemeControlElevates(_theme, ID_CUSTOM_INSTALL_BUTTON, checked && (FAILED(hr) || !elevated));
+            ThemeGetTextControl(_theme, ID_TARGETDIR_EDITBOX, &targetDir);
+            if (targetDir) {
+                // Check the current value against the default to see
+                // if we should switch it automatically.
+                hr = BalGetStringVariable(
+                    checked ? L"DefaultJustForMeTargetDir" : L"DefaultAllUsersTargetDir",
+                    &defaultDir
+                );
+                
+                if (SUCCEEDED(hr) && defaultDir) {
+                    LPWSTR formatted = nullptr;
+                    if (defaultDir[0] && SUCCEEDED(BalFormatString(defaultDir, &formatted))) {
+                        if (wcscmp(formatted, targetDir) == 0) {
+                            ReleaseStr(defaultDir);
+                            defaultDir = nullptr;
+                            ReleaseStr(formatted);
+                            formatted = nullptr;
+                            
+                            hr = BalGetStringVariable(
+                                checked ? L"DefaultAllUsersTargetDir" : L"DefaultJustForMeTargetDir",
+                                &defaultDir
+                            );
+                            if (SUCCEEDED(hr) && defaultDir && defaultDir[0] && SUCCEEDED(BalFormatString(defaultDir, &formatted))) {
+                                ThemeSetTextControl(_theme, ID_TARGETDIR_EDITBOX, formatted);
+                                ReleaseStr(formatted);
+                            }
+                        } else {
+                            ReleaseStr(formatted);
+                        }
+                    }
+                    
+                    ReleaseStr(defaultDir);
+                }
+            }
+            break;
+
+        case ID_CUSTOM_BROWSE_BUTTON:
+            browseInfo.hwndOwner = _hWnd;
+            browseInfo.pszDisplayName = wzPath;
+            browseInfo.lpszTitle = _theme->sczCaption;
+            browseInfo.ulFlags = BIF_RETURNONLYFSDIRS | BIF_USENEWUI;
+            pidl = ::SHBrowseForFolderW(&browseInfo);
+            if (pidl && ::SHGetPathFromIDListW(pidl, wzPath)) {
+                ThemeSetTextControl(_theme, ID_TARGETDIR_EDITBOX, wzPath);
+            }
+
+            if (pidl) {
+                ::CoTaskMemFree(pidl);
+            }
+            break;
+
+        // Modify commands
+        case ID_MODIFY_BUTTON:
+            // Some variables cannot be modified
+            _engine->SetVariableString(L"InstallAllUsersState", L"disable");
+            _engine->SetVariableString(L"TargetDirState", L"disable");
+            _engine->SetVariableString(L"CustomBrowseButtonState", L"disable");
+            GoToPage(PAGE_CUSTOM1);
+            break;
+
+        case ID_REPAIR_BUTTON:
+            OnPlan(BOOTSTRAPPER_ACTION_REPAIR);
+            break;
+
+        case ID_UNINSTALL_BUTTON:
+            OnPlan(BOOTSTRAPPER_ACTION_UNINSTALL);
+            break;
+        }
+
+    LExit:
+        return;
+    }
+
+    void InstallPage_Show() {
+        // Ensure the All Users install button has a UAC shield
+        LONGLONG elevated, installAll;
+
+        if (FAILED(BalGetNumericVariable(L"WixBundleElevated", &elevated))) {
+            elevated = 0;
+        }
+
+        ThemeControlElevates(_theme, ID_INSTALL_ALL_USERS_BUTTON, !elevated);
+
+        if (SUCCEEDED(BalGetNumericVariable(L"InstallAllUsers", &installAll)) && installAll) {
+            ThemeControlElevates(_theme, ID_INSTALL_SIMPLE_BUTTON, !elevated);
+        }
+    }
+
+    void Custom1Page_Show() {
+    }
+
+    void Custom2Page_Show() {
+        HRESULT hr;
+        LONGLONG installAll, elevated, includeLauncher;
+        
+        if (FAILED(BalGetNumericVariable(L"WixBundleElevated", &elevated))) {
+            elevated = 0;
+        }
+        if (SUCCEEDED(BalGetNumericVariable(L"InstallAllUsers", &installAll))) {
+            ThemeControlElevates(_theme, ID_CUSTOM_INSTALL_BUTTON, installAll && !elevated);
+        } else {
+            installAll = 0;
+        }
+
+        if (SUCCEEDED(BalGetNumericVariable(L"Include_launcher", &includeLauncher)) && includeLauncher) {
+            ThemeControlEnable(_theme, ID_CUSTOM_ASSOCIATE_FILES_CHECKBOX, TRUE);
+        } else {
+            ThemeSendControlMessage(_theme, ID_CUSTOM_ASSOCIATE_FILES_CHECKBOX, BM_SETCHECK, BST_UNCHECKED, 0);
+            ThemeControlEnable(_theme, ID_CUSTOM_ASSOCIATE_FILES_CHECKBOX, FALSE);
+        }
+
+        LPWSTR targetDir = nullptr;
+        hr = BalGetStringVariable(L"TargetDir", &targetDir);
+        if (SUCCEEDED(hr) && targetDir && targetDir[0]) {
+            ThemeSetTextControl(_theme, ID_TARGETDIR_EDITBOX, targetDir);
+            StrFree(targetDir);
+        } else if (SUCCEEDED(hr)) {
+            StrFree(targetDir);
+            targetDir = nullptr;
+
+            LPWSTR defaultTargetDir = nullptr;
+            hr = BalGetStringVariable(L"DefaultCustomTargetDir", &defaultTargetDir);
+            if (SUCCEEDED(hr) && defaultTargetDir && !defaultTargetDir[0]) {
+                StrFree(defaultTargetDir);
+                defaultTargetDir = nullptr;
+                
+                hr = BalGetStringVariable(
+                    installAll ? L"DefaultAllUsersTargetDir" : L"DefaultJustForMeTargetDir",
+                    &defaultTargetDir
+                );
+            }
+            if (SUCCEEDED(hr) && defaultTargetDir) {
+                if (defaultTargetDir[0] && SUCCEEDED(BalFormatString(defaultTargetDir, &targetDir))) {
+                    ThemeSetTextControl(_theme, ID_TARGETDIR_EDITBOX, targetDir);
+                    StrFree(targetDir);
+                }
+                StrFree(defaultTargetDir);
+            }
+        }
+    }
+
+    void ModifyPage_Show() {
+        ThemeControlEnable(_theme, ID_REPAIR_BUTTON, !_suppressRepair);
+    }
+
+    void SuccessPage_Show() {
+        // on the "Success" page, check if the restart or launch button should be enabled.
+        BOOL showRestartButton = FALSE;
+        BOOL launchTargetExists = FALSE;
+        LOC_STRING *successText = nullptr;
+        HRESULT hr = S_OK;
+        
+        if (_restartRequired) {
+            if (BOOTSTRAPPER_RESTART_PROMPT == _command.restart) {
+                showRestartButton = TRUE;
+            }
+        } else if (ThemeControlExists(_theme, ID_LAUNCH_BUTTON)) {
+            launchTargetExists = BalStringVariableExists(PYBA_VARIABLE_LAUNCH_TARGET_PATH);
+        }
+
+        switch (_plannedAction) {
+        case BOOTSTRAPPER_ACTION_INSTALL:
+            hr = LocGetString(_wixLoc, L"#(loc.SuccessInstallMessage)", &successText);
+            break;
+        case BOOTSTRAPPER_ACTION_MODIFY:
+            hr = LocGetString(_wixLoc, L"#(loc.SuccessModifyMessage)", &successText);
+            break;
+        case BOOTSTRAPPER_ACTION_REPAIR:
+            hr = LocGetString(_wixLoc, L"#(loc.SuccessRepairMessage)", &successText);
+            break;
+        case BOOTSTRAPPER_ACTION_UNINSTALL:
+            hr = LocGetString(_wixLoc, L"#(loc.SuccessRemoveMessage)", &successText);
+            break;
+        }
+
+        if (successText) {
+            LPWSTR formattedString = nullptr;
+            BalFormatString(successText->wzText, &formattedString);
+            if (formattedString) {
+                ThemeSetTextControl(_theme, ID_SUCCESS_TEXT, formattedString);
+                StrFree(formattedString);
+            }
+        }
+
+        ThemeControlEnable(_theme, ID_LAUNCH_BUTTON, launchTargetExists && BOOTSTRAPPER_ACTION_UNINSTALL < _plannedAction);
+        ThemeControlEnable(_theme, ID_SUCCESS_RESTART_TEXT, showRestartButton);
+        ThemeControlEnable(_theme, ID_SUCCESS_RESTART_BUTTON, showRestartButton);
+    }
+
+    void FailurePage_Show() {
+        // on the "Failure" page, show error message and check if the restart button should be enabled.
+
+        // if there is a log file variable then we'll assume the log file exists.
+        BOOL showLogLink = (_bundle.sczLogVariable && *_bundle.sczLogVariable);
+        BOOL showErrorMessage = FALSE;
+        BOOL showRestartButton = FALSE;
+
+        if (FAILED(_hrFinal)) {
+            LPWSTR unformattedText = nullptr;
+            LPWSTR text = nullptr;
+
+            // If we know the failure message, use that.
+            if (_failedMessage && *_failedMessage) {
+                StrAllocString(&unformattedText, _failedMessage, 0);
+            } else {
+                // try to get the error message from the error code.
+                StrAllocFromError(&unformattedText, _hrFinal, nullptr);
+                if (!unformattedText || !*unformattedText) {
+                    StrAllocFromError(&unformattedText, E_FAIL, nullptr);
+                }
+            }
+
+            if (E_WIXSTDBA_CONDITION_FAILED == _hrFinal) {
+                if (unformattedText) {
+                    StrAllocString(&text, unformattedText, 0);
+                }
+            } else {
+                StrAllocFormatted(&text, L"0x%08x - %ls", _hrFinal, unformattedText);
+            }
+
+            if (text) {
+                ThemeSetTextControl(_theme, ID_FAILURE_MESSAGE_TEXT, text);
+                showErrorMessage = TRUE;
+            }
+
+            ReleaseStr(text);
+            ReleaseStr(unformattedText);
+        }
+
+        if (_restartRequired && BOOTSTRAPPER_RESTART_PROMPT == _command.restart) {
+            showRestartButton = TRUE;
+        }
+
+        ThemeControlEnable(_theme, ID_FAILURE_LOGFILE_LINK, showLogLink);
+        ThemeControlEnable(_theme, ID_FAILURE_MESSAGE_TEXT, showErrorMessage);
+        ThemeControlEnable(_theme, ID_FAILURE_RESTART_TEXT, showRestartButton);
+        ThemeControlEnable(_theme, ID_FAILURE_RESTART_BUTTON, showRestartButton);
+    }
+
+
+public: // IBootstrapperApplication
+    virtual STDMETHODIMP OnStartup() {
+        HRESULT hr = S_OK;
+        DWORD dwUIThreadId = 0;
+
+        // create UI thread
+        _hUiThread = ::CreateThread(nullptr, 0, UiThreadProc, this, 0, &dwUIThreadId);
+        if (!_hUiThread) {
+            ExitWithLastError(hr, "Failed to create UI thread.");
+        }
+
+    LExit:
+        return hr;
+    }
+
+
+    virtual STDMETHODIMP_(int) OnShutdown() {
+        int nResult = IDNOACTION;
+
+        // wait for UI thread to terminate
+        if (_hUiThread) {
+            ::WaitForSingleObject(_hUiThread, INFINITE);
+            ReleaseHandle(_hUiThread);
+        }
+
+        // If a restart was required.
+        if (_restartRequired && _allowRestart) {
+            nResult = IDRESTART;
+        }
+
+        return nResult;
+    }
+
+
+    virtual STDMETHODIMP_(int) OnDetectRelatedBundle(
+        __in LPCWSTR wzBundleId,
+        __in BOOTSTRAPPER_RELATION_TYPE relationType,
+        __in LPCWSTR /*wzBundleTag*/,
+        __in BOOL fPerMachine,
+        __in DWORD64 /*dw64Version*/,
+        __in BOOTSTRAPPER_RELATED_OPERATION operation
+    ) {
+        BalInfoAddRelatedBundleAsPackage(&_bundle.packages, wzBundleId, relationType, fPerMachine);
+
+        // Remember when our bundle would cause a downgrade.
+        if (BOOTSTRAPPER_RELATED_OPERATION_DOWNGRADE == operation) {
+            _downgrading = TRUE;
+        }
+
+        return CheckCanceled() ? IDCANCEL : IDOK;
+    }
+
+
+    virtual STDMETHODIMP_(void) OnDetectPackageComplete(
+        __in LPCWSTR wzPackageId,
+        __in HRESULT /*hrStatus*/,
+        __in BOOTSTRAPPER_PACKAGE_STATE state
+    ) { }
+
+
+    virtual STDMETHODIMP_(void) OnDetectComplete(__in HRESULT hrStatus) {
+        if (SUCCEEDED(hrStatus) && _baFunction) {
+            BalLog(BOOTSTRAPPER_LOG_LEVEL_STANDARD, "Running detect complete BA function");
+            _baFunction->OnDetectComplete();
+        }
+
+        if (SUCCEEDED(hrStatus)) {
+            hrStatus = EvaluateConditions();
+        }
+
+        SetState(PYBA_STATE_DETECTED, hrStatus);
+
+        // If we're not interacting with the user or we're doing a layout or we're just after a force restart
+        // then automatically start planning.
+        if (BOOTSTRAPPER_DISPLAY_FULL > _command.display ||
+            BOOTSTRAPPER_ACTION_LAYOUT == _command.action ||
+            BOOTSTRAPPER_ACTION_UNINSTALL == _command.action ||
+            BOOTSTRAPPER_RESUME_TYPE_REBOOT == _command.resumeType) {
+            if (SUCCEEDED(hrStatus)) {
+                ::PostMessageW(_hWnd, WM_PYBA_PLAN_PACKAGES, 0, _command.action);
+            }
+        }
+    }
+
+
+    virtual STDMETHODIMP_(int) OnPlanRelatedBundle(
+        __in_z LPCWSTR /*wzBundleId*/,
+        __inout_z BOOTSTRAPPER_REQUEST_STATE* pRequestedState
+    ) {
+        return CheckCanceled() ? IDCANCEL : IDOK;
+    }
+
+
+    virtual STDMETHODIMP_(int) OnPlanPackageBegin(
+        __in_z LPCWSTR wzPackageId,
+        __inout BOOTSTRAPPER_REQUEST_STATE *pRequestState
+    ) {
+        HRESULT hr = S_OK;
+        BAL_INFO_PACKAGE* pPackage = nullptr;
+
+        if (_nextPackageAfterRestart) // after force restart, skip packages until after the package that caused the restart.
+        {
+            // After restart we need to finish the dependency registration for our package so allow the package
+            // to go present.
+            if (CSTR_EQUAL == ::CompareStringW(LOCALE_NEUTRAL, 0, wzPackageId, -1, _nextPackageAfterRestart, -1)) {
+                // Do not allow a repair because that could put us in a perpetual restart loop.
+                if (BOOTSTRAPPER_REQUEST_STATE_REPAIR == *pRequestState) {
+                    *pRequestState = BOOTSTRAPPER_REQUEST_STATE_PRESENT;
+                }
+
+                ReleaseNullStr(_nextPackageAfterRestart); // no more skipping now.
+            } else {
+                // not the matching package, so skip it.
+                BalLog(BOOTSTRAPPER_LOG_LEVEL_STANDARD, "Skipping package: %ls, after restart because it was applied before the restart.", wzPackageId);
+
+                *pRequestState = BOOTSTRAPPER_REQUEST_STATE_NONE;
+            }
+        }
+
+        return CheckCanceled() ? IDCANCEL : IDOK;
+    }
+
+    virtual STDMETHODIMP_(int) OnPlanMsiFeature(
+        __in_z LPCWSTR wzPackageId,
+        __in_z LPCWSTR wzFeatureId,
+        __inout BOOTSTRAPPER_FEATURE_STATE* pRequestedState
+    ) {
+        LONGLONG install;
+        
+        if (wcscmp(wzFeatureId, L"AssociateFiles") == 0) {
+            if (SUCCEEDED(_engine->GetVariableNumeric(L"AssociateFiles", &install)) && install) {
+                *pRequestedState = BOOTSTRAPPER_FEATURE_STATE_LOCAL;
+            } else {
+                *pRequestedState = BOOTSTRAPPER_FEATURE_STATE_ABSENT;
+            }
+        }
+        return CheckCanceled() ? IDCANCEL : IDNOACTION;
+    }
+
+    virtual STDMETHODIMP_(void) OnPlanComplete(__in HRESULT hrStatus) {
+        if (SUCCEEDED(hrStatus) && _baFunction) {
+            BalLog(BOOTSTRAPPER_LOG_LEVEL_STANDARD, "Running plan complete BA function");
+            _baFunction->OnPlanComplete();
+        }
+
+        SetState(PYBA_STATE_PLANNED, hrStatus);
+
+        if (SUCCEEDED(hrStatus)) {
+            ::PostMessageW(_hWnd, WM_PYBA_APPLY_PACKAGES, 0, 0);
+        }
+
+        _startedExecution = FALSE;
+        _calculatedCacheProgress = 0;
+        _calculatedExecuteProgress = 0;
+    }
+
+
+    virtual STDMETHODIMP_(int) OnCachePackageBegin(
+        __in_z LPCWSTR wzPackageId,
+        __in DWORD cCachePayloads,
+        __in DWORD64 dw64PackageCacheSize
+    ) {
+        if (wzPackageId && *wzPackageId) {
+            BAL_INFO_PACKAGE* pPackage = nullptr;
+            HRESULT hr = BalInfoFindPackageById(&_bundle.packages, wzPackageId, &pPackage);
+            LPCWSTR wz = (SUCCEEDED(hr) && pPackage->sczDisplayName) ? pPackage->sczDisplayName : wzPackageId;
+
+            ThemeSetTextControl(_theme, ID_CACHE_PROGRESS_PACKAGE_TEXT, wz);
+
+            // If something started executing, leave it in the overall progress text.
+            if (!_startedExecution) {
+                ThemeSetTextControl(_theme, ID_OVERALL_PROGRESS_PACKAGE_TEXT, wz);
+            }
+        }
+
+        return __super::OnCachePackageBegin(wzPackageId, cCachePayloads, dw64PackageCacheSize);
+    }
+
+
+    virtual STDMETHODIMP_(int) OnCacheAcquireProgress(
+        __in_z LPCWSTR wzPackageOrContainerId,
+        __in_z_opt LPCWSTR wzPayloadId,
+        __in DWORD64 dw64Progress,
+        __in DWORD64 dw64Total,
+        __in DWORD dwOverallPercentage
+    ) {
+        WCHAR wzProgress[5] = { };
+
+#ifdef DEBUG
+        BalLog(BOOTSTRAPPER_LOG_LEVEL_STANDARD, "PYBA: OnCacheAcquireProgress() - container/package: %ls, payload: %ls, progress: %I64u, total: %I64u, overall progress: %u%%", wzPackageOrContainerId, wzPayloadId, dw64Progress, dw64Total, dwOverallPercentage);
+#endif
+
+        ::StringCchPrintfW(wzProgress, countof(wzProgress), L"%u%%", dwOverallPercentage);
+        ThemeSetTextControl(_theme, ID_CACHE_PROGRESS_TEXT, wzProgress);
+
+        ThemeSetProgressControl(_theme, ID_CACHE_PROGRESS_BAR, dwOverallPercentage);
+
+        _calculatedCacheProgress = dwOverallPercentage * PYBA_ACQUIRE_PERCENTAGE / 100;
+        ThemeSetProgressControl(_theme, ID_OVERALL_CALCULATED_PROGRESS_BAR, _calculatedCacheProgress + _calculatedExecuteProgress);
+
+        SetTaskbarButtonProgress(_calculatedCacheProgress + _calculatedExecuteProgress);
+
+        return __super::OnCacheAcquireProgress(wzPackageOrContainerId, wzPayloadId, dw64Progress, dw64Total, dwOverallPercentage);
+    }
+
+
+    virtual STDMETHODIMP_(int) OnCacheAcquireComplete(
+        __in_z LPCWSTR wzPackageOrContainerId,
+        __in_z_opt LPCWSTR wzPayloadId,
+        __in HRESULT hrStatus,
+        __in int nRecommendation
+    ) {
+        SetProgressState(hrStatus);
+        return __super::OnCacheAcquireComplete(wzPackageOrContainerId, wzPayloadId, hrStatus, nRecommendation);
+    }
+
+
+    virtual STDMETHODIMP_(int) OnCacheVerifyComplete(
+        __in_z LPCWSTR wzPackageId,
+        __in_z LPCWSTR wzPayloadId,
+        __in HRESULT hrStatus,
+        __in int nRecommendation
+    ) {
+        SetProgressState(hrStatus);
+        return __super::OnCacheVerifyComplete(wzPackageId, wzPayloadId, hrStatus, nRecommendation);
+    }
+
+
+    virtual STDMETHODIMP_(void) OnCacheComplete(__in HRESULT /*hrStatus*/) {
+        ThemeSetTextControl(_theme, ID_CACHE_PROGRESS_PACKAGE_TEXT, L"");
+        SetState(PYBA_STATE_CACHED, S_OK); // we always return success here and let OnApplyComplete() deal with the error.
+    }
+
+
+    virtual STDMETHODIMP_(int) OnError(
+        __in BOOTSTRAPPER_ERROR_TYPE errorType,
+        __in LPCWSTR wzPackageId,
+        __in DWORD dwCode,
+        __in_z LPCWSTR wzError,
+        __in DWORD dwUIHint,
+        __in DWORD /*cData*/,
+        __in_ecount_z_opt(cData) LPCWSTR* /*rgwzData*/,
+        __in int nRecommendation
+    ) {
+        int nResult = nRecommendation;
+        LPWSTR sczError = nullptr;
+
+        if (BOOTSTRAPPER_DISPLAY_EMBEDDED == _command.display) {
+            HRESULT hr = _engine->SendEmbeddedError(dwCode, wzError, dwUIHint, &nResult);
+            if (FAILED(hr)) {
+                nResult = IDERROR;
+            }
+        } else if (BOOTSTRAPPER_DISPLAY_FULL == _command.display) {
+            // If this is an authentication failure, let the engine try to handle it for us.
+            if (BOOTSTRAPPER_ERROR_TYPE_HTTP_AUTH_SERVER == errorType || BOOTSTRAPPER_ERROR_TYPE_HTTP_AUTH_PROXY == errorType) {
+                nResult = IDTRYAGAIN;
+            } else // show a generic error message box.
+            {
+                BalRetryErrorOccurred(wzPackageId, dwCode);
+
+                if (!_showingInternalUIThisPackage) {
+                    // If no error message was provided, use the error code to try and get an error message.
+                    if (!wzError || !*wzError || BOOTSTRAPPER_ERROR_TYPE_WINDOWS_INSTALLER != errorType) {
+                        HRESULT hr = StrAllocFromError(&sczError, dwCode, nullptr);
+                        if (FAILED(hr) || !sczError || !*sczError) {
+                            StrAllocFormatted(&sczError, L"0x%x", dwCode);
+                        }
+                    }
+
+                    nResult = ::MessageBoxW(_hWnd, sczError ? sczError : wzError, _theme->sczCaption, dwUIHint);
+                }
+            }
+
+            SetProgressState(HRESULT_FROM_WIN32(dwCode));
+        } else {
+            // just take note of the error code and let things continue.
+            BalRetryErrorOccurred(wzPackageId, dwCode);
+        }
+
+        ReleaseStr(sczError);
+        return nResult;
+    }
+
+
+    virtual STDMETHODIMP_(int) OnExecuteMsiMessage(
+        __in_z LPCWSTR wzPackageId,
+        __in INSTALLMESSAGE mt,
+        __in UINT uiFlags,
+        __in_z LPCWSTR wzMessage,
+        __in DWORD cData,
+        __in_ecount_z_opt(cData) LPCWSTR* rgwzData,
+        __in int nRecommendation
+    ) {
+#ifdef DEBUG
+        BalLog(BOOTSTRAPPER_LOG_LEVEL_STANDARD, "PYBA: OnExecuteMsiMessage() - package: %ls, message: %ls", wzPackageId, wzMessage);
+#endif
+        if (BOOTSTRAPPER_DISPLAY_FULL == _command.display && (INSTALLMESSAGE_WARNING == mt || INSTALLMESSAGE_USER == mt)) {
+            int nResult = ::MessageBoxW(_hWnd, wzMessage, _theme->sczCaption, uiFlags);
+            return nResult;
+        }
+
+        if (INSTALLMESSAGE_ACTIONSTART == mt) {
+            ThemeSetTextControl(_theme, ID_EXECUTE_PROGRESS_ACTIONDATA_TEXT, wzMessage);
+        }
+
+        return __super::OnExecuteMsiMessage(wzPackageId, mt, uiFlags, wzMessage, cData, rgwzData, nRecommendation);
+    }
+
+
+    virtual STDMETHODIMP_(int) OnProgress(__in DWORD dwProgressPercentage, __in DWORD dwOverallProgressPercentage) {
+        WCHAR wzProgress[5] = { };
+
+#ifdef DEBUG
+        BalLog(BOOTSTRAPPER_LOG_LEVEL_STANDARD, "PYBA: OnProgress() - progress: %u%%, overall progress: %u%%", dwProgressPercentage, dwOverallProgressPercentage);
+#endif
+
+        ::StringCchPrintfW(wzProgress, countof(wzProgress), L"%u%%", dwOverallProgressPercentage);
+        ThemeSetTextControl(_theme, ID_OVERALL_PROGRESS_TEXT, wzProgress);
+
+        ThemeSetProgressControl(_theme, ID_OVERALL_PROGRESS_BAR, dwOverallProgressPercentage);
+        SetTaskbarButtonProgress(dwOverallProgressPercentage);
+
+        return __super::OnProgress(dwProgressPercentage, dwOverallProgressPercentage);
+    }
+
+
+    virtual STDMETHODIMP_(int) OnExecutePackageBegin(__in_z LPCWSTR wzPackageId, __in BOOL fExecute) {
+        LPWSTR sczFormattedString = nullptr;
+
+        _startedExecution = TRUE;
+
+        if (wzPackageId && *wzPackageId) {
+            BAL_INFO_PACKAGE* pPackage = nullptr;
+            BalInfoFindPackageById(&_bundle.packages, wzPackageId, &pPackage);
+
+            LPCWSTR wz = wzPackageId;
+            if (pPackage) {
+                LOC_STRING* pLocString = nullptr;
+
+                switch (pPackage->type) {
+                case BAL_INFO_PACKAGE_TYPE_BUNDLE_ADDON:
+                    LocGetString(_wixLoc, L"#(loc.ExecuteAddonRelatedBundleMessage)", &pLocString);
+                    break;
+
+                case BAL_INFO_PACKAGE_TYPE_BUNDLE_PATCH:
+                    LocGetString(_wixLoc, L"#(loc.ExecutePatchRelatedBundleMessage)", &pLocString);
+                    break;
+
+                case BAL_INFO_PACKAGE_TYPE_BUNDLE_UPGRADE:
+                    LocGetString(_wixLoc, L"#(loc.ExecuteUpgradeRelatedBundleMessage)", &pLocString);
+                    break;
+                }
+
+                if (pLocString) {
+                    // If the wix developer is showing a hidden variable in the UI, then obviously they don't care about keeping it safe
+                    // so don't go down the rabbit hole of making sure that this is securely freed.
+                    BalFormatString(pLocString->wzText, &sczFormattedString);
+                }
+
+                wz = sczFormattedString ? sczFormattedString : pPackage->sczDisplayName ? pPackage->sczDisplayName : wzPackageId;
+            }
+
+            _showingInternalUIThisPackage = pPackage && pPackage->fDisplayInternalUI;
+
+            ThemeSetTextControl(_theme, ID_EXECUTE_PROGRESS_PACKAGE_TEXT, wz);
+            ThemeSetTextControl(_theme, ID_OVERALL_PROGRESS_PACKAGE_TEXT, wz);
+        } else {
+            _showingInternalUIThisPackage = FALSE;
+        }
+
+        ReleaseStr(sczFormattedString);
+        return __super::OnExecutePackageBegin(wzPackageId, fExecute);
+    }
+
+
+    virtual int __stdcall OnExecuteProgress(
+        __in_z LPCWSTR wzPackageId,
+        __in DWORD dwProgressPercentage,
+        __in DWORD dwOverallProgressPercentage
+    ) {
+        WCHAR wzProgress[8] = { };
+
+#ifdef DEBUG
+        BalLog(BOOTSTRAPPER_LOG_LEVEL_STANDARD, "PYBA: OnExecuteProgress() - package: %ls, progress: %u%%, overall progress: %u%%", wzPackageId, dwProgressPercentage, dwOverallProgressPercentage);
+#endif
+
+        ::StringCchPrintfW(wzProgress, countof(wzProgress), L"%u%%", dwOverallProgressPercentage);
+        ThemeSetTextControl(_theme, ID_EXECUTE_PROGRESS_TEXT, wzProgress);
+
+        ThemeSetProgressControl(_theme, ID_EXECUTE_PROGRESS_BAR, dwOverallProgressPercentage);
+
+        _calculatedExecuteProgress = dwOverallProgressPercentage * (100 - PYBA_ACQUIRE_PERCENTAGE) / 100;
+        ThemeSetProgressControl(_theme, ID_OVERALL_CALCULATED_PROGRESS_BAR, _calculatedCacheProgress + _calculatedExecuteProgress);
+
+        SetTaskbarButtonProgress(_calculatedCacheProgress + _calculatedExecuteProgress);
+
+        return __super::OnExecuteProgress(wzPackageId, dwProgressPercentage, dwOverallProgressPercentage);
+    }
+
+
+    virtual STDMETHODIMP_(int) OnExecutePackageComplete(
+        __in_z LPCWSTR wzPackageId,
+        __in HRESULT hrExitCode,
+        __in BOOTSTRAPPER_APPLY_RESTART restart,
+        __in int nRecommendation
+    ) {
+        SetProgressState(hrExitCode);
+
+        if (_wcsnicmp(wzPackageId, L"path_", 5) == 0 && SUCCEEDED(hrExitCode)) {
+            SendMessageTimeoutW(
+                HWND_BROADCAST,
+                WM_SETTINGCHANGE,
+                0,
+                reinterpret_cast<LPARAM>(L"Environment"),
+                SMTO_ABORTIFHUNG,
+                1000,
+                nullptr
+            );
+        }
+
+        int nResult = __super::OnExecutePackageComplete(wzPackageId, hrExitCode, restart, nRecommendation);
+
+        return nResult;
+    }
+
+
+    virtual STDMETHODIMP_(void) OnExecuteComplete(__in HRESULT hrStatus) {
+        ThemeSetTextControl(_theme, ID_EXECUTE_PROGRESS_PACKAGE_TEXT, L"");
+        ThemeSetTextControl(_theme, ID_EXECUTE_PROGRESS_ACTIONDATA_TEXT, L"");
+        ThemeSetTextControl(_theme, ID_OVERALL_PROGRESS_PACKAGE_TEXT, L"");
+        ThemeControlEnable(_theme, ID_PROGRESS_CANCEL_BUTTON, FALSE); // no more cancel.
+
+        SetState(PYBA_STATE_EXECUTED, S_OK); // we always return success here and let OnApplyComplete() deal with the error.
+        SetProgressState(hrStatus);
+    }
+
+
+    virtual STDMETHODIMP_(int) OnResolveSource(
+        __in_z LPCWSTR wzPackageOrContainerId,
+        __in_z_opt LPCWSTR wzPayloadId,
+        __in_z LPCWSTR wzLocalSource,
+        __in_z_opt LPCWSTR wzDownloadSource
+    ) {
+        int nResult = IDERROR; // assume we won't resolve source and that is unexpected.
+
+        if (BOOTSTRAPPER_DISPLAY_FULL == _command.display) {
+            if (wzDownloadSource) {
+                nResult = IDDOWNLOAD;
+            } else {
+                // prompt to change the source location.
+                OPENFILENAMEW ofn = { };
+                WCHAR wzFile[MAX_PATH] = { };
+
+                ::StringCchCopyW(wzFile, countof(wzFile), wzLocalSource);
+
+                ofn.lStructSize = sizeof(ofn);
+                ofn.hwndOwner = _hWnd;
+                ofn.lpstrFile = wzFile;
+                ofn.nMaxFile = countof(wzFile);
+                ofn.lpstrFilter = L"All Files\0*.*\0";
+                ofn.nFilterIndex = 1;
+                ofn.Flags = OFN_PATHMUSTEXIST | OFN_FILEMUSTEXIST;
+                ofn.lpstrTitle = _theme->sczCaption;
+
+                if (::GetOpenFileNameW(&ofn)) {
+                    HRESULT hr = _engine->SetLocalSource(wzPackageOrContainerId, wzPayloadId, ofn.lpstrFile);
+                    nResult = SUCCEEDED(hr) ? IDRETRY : IDERROR;
+                } else {
+                    nResult = IDCANCEL;
+                }
+            }
+        } else if (wzDownloadSource) {
+            // If doing a non-interactive install and download source is available, let's try downloading the package silently
+            nResult = IDDOWNLOAD;
+        }
+        // else there's nothing more we can do in non-interactive mode
+
+        return CheckCanceled() ? IDCANCEL : nResult;
+    }
+
+
+    virtual STDMETHODIMP_(int) OnApplyComplete(__in HRESULT hrStatus, __in BOOTSTRAPPER_APPLY_RESTART restart) {
+        _restartResult = restart; // remember the restart result so we return the correct error code no matter what the user chooses to do in the UI.
+
+        // If a restart was encountered and we are not suppressing restarts, then restart is required.
+        _restartRequired = (BOOTSTRAPPER_APPLY_RESTART_NONE != restart && BOOTSTRAPPER_RESTART_NEVER < _command.restart);
+        // If a restart is required and we're not displaying a UI or we are not supposed to prompt for restart then allow the restart.
+        _allowRestart = _restartRequired && (BOOTSTRAPPER_DISPLAY_FULL > _command.display || BOOTSTRAPPER_RESTART_PROMPT < _command.restart);
+
+        // If we are showing UI, wait a beat before moving to the final screen.
+        if (BOOTSTRAPPER_DISPLAY_NONE < _command.display) {
+            ::Sleep(250);
+        }
+
+        SetState(PYBA_STATE_APPLIED, hrStatus);
+        SetTaskbarButtonProgress(100); // show full progress bar, green, yellow, or red
+
+        return IDNOACTION;
+    }
+
+    virtual STDMETHODIMP_(void) OnLaunchApprovedExeComplete(__in HRESULT hrStatus, __in DWORD /*processId*/) {
+        if (HRESULT_FROM_WIN32(ERROR_ACCESS_DENIED) == hrStatus) {
+            //try with ShelExec next time
+            OnClickLaunchButton();
+        } else {
+            ::PostMessageW(_hWnd, WM_CLOSE, 0, 0);
+        }
+    }
+
+
+private:
+    //
+    // UiThreadProc - entrypoint for UI thread.
+    //
+    static DWORD WINAPI UiThreadProc(__in LPVOID pvContext) {
+        HRESULT hr = S_OK;
+        PythonBootstrapperApplication* pThis = (PythonBootstrapperApplication*)pvContext;
+        BOOL comInitialized = FALSE;
+        BOOL ret = FALSE;
+        MSG msg = { };
+
+        // Initialize COM and theme.
+        hr = ::CoInitialize(nullptr);
+        BalExitOnFailure(hr, "Failed to initialize COM.");
+        comInitialized = TRUE;
+
+        hr = ThemeInitialize(pThis->_hModule);
+        BalExitOnFailure(hr, "Failed to initialize theme manager.");
+
+        hr = pThis->InitializeData();
+        BalExitOnFailure(hr, "Failed to initialize data in bootstrapper application.");
+
+        // Create main window.
+        pThis->InitializeTaskbarButton();
+        hr = pThis->CreateMainWindow();
+        BalExitOnFailure(hr, "Failed to create main window.");
+
+        if (FAILED(pThis->_hrFinal)) {
+            pThis->SetState(PYBA_STATE_FAILED, hr);
+            ::PostMessageW(pThis->_hWnd, WM_PYBA_SHOW_FAILURE, 0, 0);
+        } else {
+            // Okay, we're ready for packages now.
+            pThis->SetState(PYBA_STATE_INITIALIZED, hr);
+            ::PostMessageW(pThis->_hWnd, BOOTSTRAPPER_ACTION_HELP == pThis->_command.action ? WM_PYBA_SHOW_HELP : WM_PYBA_DETECT_PACKAGES, 0, 0);
+        }
+
+        // message pump
+        while (0 != (ret = ::GetMessageW(&msg, nullptr, 0, 0))) {
+            if (-1 == ret) {
+                hr = E_UNEXPECTED;
+                BalExitOnFailure(hr, "Unexpected return value from message pump.");
+            } else if (!ThemeHandleKeyboardMessage(pThis->_theme, msg.hwnd, &msg)) {
+                ::TranslateMessage(&msg);
+                ::DispatchMessageW(&msg);
+            }
+        }
+
+        // Succeeded thus far, check to see if anything went wrong while actually
+        // executing changes.
+        if (FAILED(pThis->_hrFinal)) {
+            hr = pThis->_hrFinal;
+        } else if (pThis->CheckCanceled()) {
+            hr = HRESULT_FROM_WIN32(ERROR_INSTALL_USEREXIT);
+        }
+
+    LExit:
+        // destroy main window
+        pThis->DestroyMainWindow();
+
+        // initiate engine shutdown
+        DWORD dwQuit = HRESULT_CODE(hr);
+        if (BOOTSTRAPPER_APPLY_RESTART_INITIATED == pThis->_restartResult) {
+            dwQuit = ERROR_SUCCESS_REBOOT_INITIATED;
+        } else if (BOOTSTRAPPER_APPLY_RESTART_REQUIRED == pThis->_restartResult) {
+            dwQuit = ERROR_SUCCESS_REBOOT_REQUIRED;
+        }
+        pThis->_engine->Quit(dwQuit);
+
+        ReleaseTheme(pThis->_theme);
+        ThemeUninitialize();
+
+        // uninitialize COM
+        if (comInitialized) {
+            ::CoUninitialize();
+        }
+
+        return hr;
+    }
+
+
+    //
+    // InitializeData - initializes all the package information.
+    //
+    HRESULT InitializeData() {
+        HRESULT hr = S_OK;
+        LPWSTR sczModulePath = nullptr;
+        IXMLDOMDocument *pixdManifest = nullptr;
+
+        hr = BalManifestLoad(_hModule, &pixdManifest);
+        BalExitOnFailure(hr, "Failed to load bootstrapper application manifest.");
+
+        hr = ParseOverridableVariablesFromXml(pixdManifest);
+        BalExitOnFailure(hr, "Failed to read overridable variables.");
+
+        hr = ProcessCommandLine(&_language);
+        ExitOnFailure(hr, "Unknown commandline parameters.");
+
+        hr = PathRelativeToModule(&sczModulePath, nullptr, _hModule);
+        BalExitOnFailure(hr, "Failed to get module path.");
+
+        hr = LoadLocalization(sczModulePath, _language);
+        ExitOnFailure(hr, "Failed to load localization.");
+
+        hr = LoadTheme(sczModulePath, _language);
+        ExitOnFailure(hr, "Failed to load theme.");
+
+        hr = BalInfoParseFromXml(&_bundle, pixdManifest);
+        BalExitOnFailure(hr, "Failed to load bundle information.");
+
+        hr = BalConditionsParseFromXml(&_conditions, pixdManifest, _wixLoc);
+        BalExitOnFailure(hr, "Failed to load conditions from XML.");
+
+        hr = LoadBootstrapperBAFunctions();
+        BalExitOnFailure(hr, "Failed to load bootstrapper functions.");
+
+        GetBundleFileVersion();
+        // don't fail if we couldn't get the version info; best-effort only
+
+    LExit:
+        ReleaseObject(pixdManifest);
+        ReleaseStr(sczModulePath);
+
+        return hr;
+    }
+
+
+    //
+    // ProcessCommandLine - process the provided command line arguments.
+    //
+    HRESULT ProcessCommandLine(__inout LPWSTR* psczLanguage) {
+        HRESULT hr = S_OK;
+        int argc = 0;
+        LPWSTR* argv = nullptr;
+        LPWSTR sczVariableName = nullptr;
+        LPWSTR sczVariableValue = nullptr;
+
+        if (_command.wzCommandLine && *_command.wzCommandLine) {
+            argv = ::CommandLineToArgvW(_command.wzCommandLine, &argc);
+            ExitOnNullWithLastError(argv, hr, "Failed to get command line.");
+
+            for (int i = 0; i < argc; ++i) {
+                if (argv[i][0] == L'-' || argv[i][0] == L'/') {
+                    if (CSTR_EQUAL == ::CompareStringW(LOCALE_INVARIANT, NORM_IGNORECASE, &argv[i][1], -1, L"lang", -1)) {
+                        if (i + 1 >= argc) {
+                            hr = E_INVALIDARG;
+                            BalExitOnFailure(hr, "Must specify a language.");
+                        }
+
+                        ++i;
+
+                        hr = StrAllocString(psczLanguage, &argv[i][0], 0);
+                        BalExitOnFailure(hr, "Failed to copy language.");
+                    } 
+                } else if (_overridableVariables) {
+                    int value;
+                    const wchar_t* pwc = wcschr(argv[i], L'=');
+                    if (pwc) {
+                        hr = StrAllocString(&sczVariableName, argv[i], pwc - argv[i]);
+                        BalExitOnFailure(hr, "Failed to copy variable name.");
+
+                        hr = DictKeyExists(_overridableVariables, sczVariableName);
+                        if (E_NOTFOUND == hr) {
+                            BalLog(BOOTSTRAPPER_LOG_LEVEL_ERROR, "Ignoring attempt to set non-overridable variable: '%ls'.", sczVariableName);
+                            hr = S_OK;
+                            continue;
+                        }
+                        ExitOnFailure(hr, "Failed to check the dictionary of overridable variables.");
+
+                        hr = StrAllocString(&sczVariableValue, ++pwc, 0);
+                        BalExitOnFailure(hr, "Failed to copy variable value.");
+
+                        if (::StrToIntEx(sczVariableValue, STIF_DEFAULT, &value)) {
+                            hr = _engine->SetVariableNumeric(sczVariableName, value);
+                        } else {
+                            hr = _engine->SetVariableString(sczVariableName, sczVariableValue);
+                        }
+                        BalExitOnFailure(hr, "Failed to set variable.");
+                    } else {
+                        BalLog(BOOTSTRAPPER_LOG_LEVEL_STANDARD, "Ignoring unknown argument: %ls", argv[i]);
+                    }
+                }
+            }
+        }
+
+    LExit:
+        if (argv) {
+            ::LocalFree(argv);
+        }
+
+        ReleaseStr(sczVariableName);
+        ReleaseStr(sczVariableValue);
+
+        return hr;
+    }
+
+    HRESULT LoadLocalization(__in_z LPCWSTR wzModulePath, __in_z_opt LPCWSTR wzLanguage) {
+        HRESULT hr = S_OK;
+        LPWSTR sczLocPath = nullptr;
+        LPCWSTR wzLocFileName = L"Default.wxl";
+
+        hr = LocProbeForFile(wzModulePath, wzLocFileName, wzLanguage, &sczLocPath);
+        BalExitOnFailure2(hr, "Failed to probe for loc file: %ls in path: %ls", wzLocFileName, wzModulePath);
+
+        hr = LocLoadFromFile(sczLocPath, &_wixLoc);
+        BalExitOnFailure1(hr, "Failed to load loc file from path: %ls", sczLocPath);
+
+        if (WIX_LOCALIZATION_LANGUAGE_NOT_SET != _wixLoc->dwLangId) {
+            ::SetThreadLocale(_wixLoc->dwLangId);
+        }
+
+        hr = StrAllocString(&_confirmCloseMessage, L"#(loc.ConfirmCancelMessage)", 0);
+        ExitOnFailure(hr, "Failed to initialize confirm message loc identifier.");
+
+        hr = LocLocalizeString(_wixLoc, &_confirmCloseMessage);
+        BalExitOnFailure1(hr, "Failed to localize confirm close message: %ls", _confirmCloseMessage);
+
+    LExit:
+        ReleaseStr(sczLocPath);
+
+        return hr;
+    }
+
+
+    HRESULT LoadTheme(__in_z LPCWSTR wzModulePath, __in_z_opt LPCWSTR wzLanguage) {
+        HRESULT hr = S_OK;
+        LPWSTR sczThemePath = nullptr;
+        LPCWSTR wzThemeFileName = L"Default.thm";
+        LPWSTR sczCaption = nullptr;
+
+        hr = LocProbeForFile(wzModulePath, wzThemeFileName, wzLanguage, &sczThemePath);
+        BalExitOnFailure2(hr, "Failed to probe for theme file: %ls in path: %ls", wzThemeFileName, wzModulePath);
+
+        hr = ThemeLoadFromFile(sczThemePath, &_theme);
+        BalExitOnFailure1(hr, "Failed to load theme from path: %ls", sczThemePath);
+
+        hr = ThemeLocalize(_theme, _wixLoc);
+        BalExitOnFailure1(hr, "Failed to localize theme: %ls", sczThemePath);
+
+        // Update the caption if there are any formatted strings in it.
+        // If the wix developer is showing a hidden variable in the UI, then
+        // obviously they don't care about keeping it safe so don't go down the
+        // rabbit hole of making sure that this is securely freed.
+        hr = BalFormatString(_theme->sczCaption, &sczCaption);
+        if (SUCCEEDED(hr)) {
+            ThemeUpdateCaption(_theme, sczCaption);
+        }
+
+    LExit:
+        ReleaseStr(sczCaption);
+        ReleaseStr(sczThemePath);
+
+        return hr;
+    }
+
+
+    HRESULT ParseOverridableVariablesFromXml(__in IXMLDOMDocument* pixdManifest) {
+        HRESULT hr = S_OK;
+        IXMLDOMNode* pNode = nullptr;
+        IXMLDOMNodeList* pNodes = nullptr;
+        DWORD cNodes = 0;
+        LPWSTR scz = nullptr;
+        BOOL hidden = FALSE;
+
+        // get the list of variables users can override on the command line
+        hr = XmlSelectNodes(pixdManifest, L"/BootstrapperApplicationData/WixStdbaOverridableVariable", &pNodes);
+        if (S_FALSE == hr) {
+            ExitFunction1(hr = S_OK);
+        }
+        ExitOnFailure(hr, "Failed to select overridable variable nodes.");
+
+        hr = pNodes->get_length((long*)&cNodes);
+        ExitOnFailure(hr, "Failed to get overridable variable node count.");
+
+        if (cNodes) {
+            hr = DictCreateStringList(&_overridableVariables, 32, DICT_FLAG_NONE);
+            ExitOnFailure(hr, "Failed to create the string dictionary.");
+
+            for (DWORD i = 0; i < cNodes; ++i) {
+                hr = XmlNextElement(pNodes, &pNode, nullptr);
+                ExitOnFailure(hr, "Failed to get next node.");
+
+                // @Name
+                hr = XmlGetAttributeEx(pNode, L"Name", &scz);
+                ExitOnFailure(hr, "Failed to get @Name.");
+
+                hr = XmlGetYesNoAttribute(pNode, L"Hidden", &hidden);
+
+                if (!hidden) {
+                    hr = DictAddKey(_overridableVariables, scz);
+                    ExitOnFailure1(hr, "Failed to add \"%ls\" to the string dictionary.", scz);
+                }
+
+                // prepare next iteration
+                ReleaseNullObject(pNode);
+            }
+        }
+
+    LExit:
+        ReleaseObject(pNode);
+        ReleaseObject(pNodes);
+        ReleaseStr(scz);
+        return hr;
+    }
+
+
+    //
+    // Get the file version of the bootstrapper and record in bootstrapper log file
+    //
+    HRESULT GetBundleFileVersion() {
+        HRESULT hr = S_OK;
+        ULARGE_INTEGER uliVersion = { };
+        LPWSTR sczCurrentPath = nullptr;
+
+        hr = PathForCurrentProcess(&sczCurrentPath, nullptr);
+        BalExitOnFailure(hr, "Failed to get bundle path.");
+
+        hr = FileVersion(sczCurrentPath, &uliVersion.HighPart, &uliVersion.LowPart);
+        BalExitOnFailure(hr, "Failed to get bundle file version.");
+
+        hr = _engine->SetVariableVersion(PYBA_VARIABLE_BUNDLE_FILE_VERSION, uliVersion.QuadPart);
+        BalExitOnFailure(hr, "Failed to set WixBundleFileVersion variable.");
+
+    LExit:
+        ReleaseStr(sczCurrentPath);
+
+        return hr;
+    }
+
+
+    //
+    // CreateMainWindow - creates the main install window.
+    //
+    HRESULT CreateMainWindow() {
+        HRESULT hr = S_OK;
+        HICON hIcon = reinterpret_cast<HICON>(_theme->hIcon);
+        WNDCLASSW wc = { };
+        DWORD dwWindowStyle = 0;
+        int x = CW_USEDEFAULT;
+        int y = CW_USEDEFAULT;
+        POINT ptCursor = { };
+        HMONITOR hMonitor = nullptr;
+        MONITORINFO mi = { };
+
+        // If the theme did not provide an icon, try using the icon from the bundle engine.
+        if (!hIcon) {
+            HMODULE hBootstrapperEngine = ::GetModuleHandleW(nullptr);
+            if (hBootstrapperEngine) {
+                hIcon = ::LoadIconW(hBootstrapperEngine, MAKEINTRESOURCEW(1));
+            }
+        }
+
+        // Register the window class and create the window.
+        wc.lpfnWndProc = PythonBootstrapperApplication::WndProc;
+        wc.hInstance = _hModule;
+        wc.hIcon = hIcon;
+        wc.hCursor = ::LoadCursorW(nullptr, (LPCWSTR)IDC_ARROW);
+        wc.hbrBackground = _theme->rgFonts[_theme->dwFontId].hBackground;
+        wc.lpszMenuName = nullptr;
+        wc.lpszClassName = PYBA_WINDOW_CLASS;
+        if (!::RegisterClassW(&wc)) {
+            ExitWithLastError(hr, "Failed to register window.");
+        }
+
+        _registered = TRUE;
+
+        // Calculate the window style based on the theme style and command display value.
+        dwWindowStyle = _theme->dwStyle;
+        if (BOOTSTRAPPER_DISPLAY_NONE >= _command.display) {
+            dwWindowStyle &= ~WS_VISIBLE;
+        }
+
+        // Don't show the window if there is a splash screen (it will be made visible when the splash screen is hidden)
+        if (::IsWindow(_command.hwndSplashScreen)) {
+            dwWindowStyle &= ~WS_VISIBLE;
+        }
+
+        // Center the window on the monitor with the mouse.
+        if (::GetCursorPos(&ptCursor)) {
+            hMonitor = ::MonitorFromPoint(ptCursor, MONITOR_DEFAULTTONEAREST);
+            if (hMonitor) {
+                mi.cbSize = sizeof(mi);
+                if (::GetMonitorInfoW(hMonitor, &mi)) {
+                    x = mi.rcWork.left + (mi.rcWork.right - mi.rcWork.left - _theme->nWidth) / 2;
+                    y = mi.rcWork.top + (mi.rcWork.bottom - mi.rcWork.top - _theme->nHeight) / 2;
+                }
+            }
+        }
+
+        _hWnd = ::CreateWindowExW(
+            0,
+            wc.lpszClassName,
+            _theme->sczCaption,
+            dwWindowStyle,
+            x,
+            y,
+            _theme->nWidth,
+            _theme->nHeight,
+            HWND_DESKTOP,
+            nullptr,
+            _hModule,
+            this
+        );
+        ExitOnNullWithLastError(_hWnd, hr, "Failed to create window.");
+
+        hr = S_OK;
+
+    LExit:
+        return hr;
+    }
+
+
+    //
+    // InitializeTaskbarButton - initializes taskbar button for progress.
+    //
+    void InitializeTaskbarButton() {
+        HRESULT hr = S_OK;
+
+        hr = ::CoCreateInstance(CLSID_TaskbarList, nullptr, CLSCTX_ALL, __uuidof(ITaskbarList3), reinterpret_cast<LPVOID*>(&_taskbarList));
+        if (REGDB_E_CLASSNOTREG == hr) {
+            // not supported before Windows 7
+            ExitFunction1(hr = S_OK);
+        }
+        BalExitOnFailure(hr, "Failed to create ITaskbarList3. Continuing.");
+
+        _taskbarButtonCreatedMessage = ::RegisterWindowMessageW(L"TaskbarButtonCreated");
+        BalExitOnNullWithLastError(_taskbarButtonCreatedMessage, hr, "Failed to get TaskbarButtonCreated message. Continuing.");
+
+    LExit:
+        return;
+    }
+
+    //
+    // DestroyMainWindow - clean up all the window registration.
+    //
+    void DestroyMainWindow() {
+        if (::IsWindow(_hWnd)) {
+            ::DestroyWindow(_hWnd);
+            _hWnd = nullptr;
+            _taskbarButtonOK = FALSE;
+        }
+
+        if (_registered) {
+            ::UnregisterClassW(PYBA_WINDOW_CLASS, _hModule);
+            _registered = FALSE;
+        }
+    }
+
+
+    //
+    // WndProc - standard windows message handler.
+    //
+    static LRESULT CALLBACK WndProc(
+        __in HWND hWnd,
+        __in UINT uMsg,
+        __in WPARAM wParam,
+        __in LPARAM lParam
+    ) {
+#pragma warning(suppress:4312)
+        auto pBA = reinterpret_cast<PythonBootstrapperApplication*>(::GetWindowLongPtrW(hWnd, GWLP_USERDATA));
+
+        switch (uMsg) {
+        case WM_NCCREATE: {
+            LPCREATESTRUCT lpcs = reinterpret_cast<LPCREATESTRUCT>(lParam);
+            pBA = reinterpret_cast<PythonBootstrapperApplication*>(lpcs->lpCreateParams);
+#pragma warning(suppress:4244)
+            ::SetWindowLongPtrW(hWnd, GWLP_USERDATA, reinterpret_cast<LONG_PTR>(pBA));
+            break;
+        }
+
+        case WM_NCDESTROY: {
+            LRESULT lres = ThemeDefWindowProc(pBA ? pBA->_theme : nullptr, hWnd, uMsg, wParam, lParam);
+            ::SetWindowLongPtrW(hWnd, GWLP_USERDATA, 0);
+            return lres;
+        }
+
+        case WM_CREATE: 
+            if (!pBA->OnCreate(hWnd)) {
+                return -1;
+            }
+            break;
+
+        case WM_QUERYENDSESSION:
+            return IDCANCEL != pBA->OnSystemShutdown(static_cast<DWORD>(lParam), IDCANCEL);
+
+        case WM_CLOSE:
+            // If the user chose not to close, do *not* let the default window proc handle the message.
+            if (!pBA->OnClose()) {
+                return 0;
+            }
+            break;
+
+        case WM_DESTROY:
+            ::PostQuitMessage(0);
+            break;
+
+        case WM_PAINT: __fallthrough;
+        case WM_ERASEBKGND:
+            if (pBA && pBA->_suppressPaint) {
+                return TRUE;
+            }
+            break;
+
+        case WM_PYBA_SHOW_HELP:
+            pBA->OnShowHelp();
+            return 0;
+
+        case WM_PYBA_DETECT_PACKAGES:
+            pBA->OnDetect();
+            return 0;
+
+        case WM_PYBA_PLAN_PACKAGES:
+            pBA->OnPlan(static_cast<BOOTSTRAPPER_ACTION>(lParam));
+            return 0;
+
+        case WM_PYBA_APPLY_PACKAGES:
+            pBA->OnApply();
+            return 0;
+
+        case WM_PYBA_CHANGE_STATE:
+            pBA->OnChangeState(static_cast<PYBA_STATE>(lParam));
+            return 0;
+
+        case WM_PYBA_SHOW_FAILURE:
+            pBA->OnShowFailure();
+            return 0;
+
+        case WM_COMMAND:
+            switch (LOWORD(wParam)) {
+            // Customize commands
+            // Success/failure commands
+            case ID_LAUNCH_BUTTON:
+                pBA->OnClickLaunchButton();
+                return 0;
+
+            case ID_SUCCESS_RESTART_BUTTON: __fallthrough;
+            case ID_FAILURE_RESTART_BUTTON:
+                pBA->OnClickRestartButton();
+                return 0;
+
+            case IDCANCEL: __fallthrough;
+            case ID_INSTALL_CANCEL_BUTTON: __fallthrough;
+            case ID_CUSTOM1_CANCEL_BUTTON: __fallthrough;
+            case ID_CUSTOM2_CANCEL_BUTTON: __fallthrough;
+            case ID_MODIFY_CANCEL_BUTTON: __fallthrough;
+            case ID_PROGRESS_CANCEL_BUTTON: __fallthrough;
+            case ID_SUCCESS_CANCEL_BUTTON: __fallthrough;
+            case ID_FAILURE_CANCEL_BUTTON: __fallthrough;
+            case ID_CLOSE_BUTTON:
+                pBA->OnCommand(ID_CLOSE_BUTTON);
+                return 0;
+
+            default:
+                pBA->OnCommand((CONTROL_ID)LOWORD(wParam));
+            }
+            break;
+
+        case WM_NOTIFY:
+            if (lParam) {
+                LPNMHDR pnmhdr = reinterpret_cast<LPNMHDR>(lParam);
+                switch (pnmhdr->code) {
+                case NM_CLICK: __fallthrough;
+                case NM_RETURN:
+                    switch (static_cast<DWORD>(pnmhdr->idFrom)) {
+                    case ID_FAILURE_LOGFILE_LINK:
+                        pBA->OnClickLogFileLink();
+                        return 1;
+                    }
+                }
+            }
+            break;
+
+        case WM_CTLCOLORBTN:
+            if (pBA) {
+                HWND button = (HWND)lParam;
+                DWORD style = GetWindowLong(button, GWL_STYLE) & BS_TYPEMASK;
+                if (style == BS_COMMANDLINK || style == BS_DEFCOMMANDLINK) {
+                    return (LRESULT)pBA->_theme->rgFonts[pBA->_theme->dwFontId].hBackground;
+                }
+            }
+            break;
+        }
+
+        if (pBA && pBA->_taskbarList && uMsg == pBA->_taskbarButtonCreatedMessage) {
+            pBA->_taskbarButtonOK = TRUE;
+            return 0;
+        }
+
+        return ThemeDefWindowProc(pBA ? pBA->_theme : nullptr, hWnd, uMsg, wParam, lParam);
+    }
+
+    //
+    // OnCreate - finishes loading the theme.
+    //
+    BOOL OnCreate(__in HWND hWnd) {
+        HRESULT hr = S_OK;
+
+        hr = ThemeLoadControls(_theme, hWnd, CONTROL_ID_NAMES, countof(CONTROL_ID_NAMES));
+        BalExitOnFailure(hr, "Failed to load theme controls.");
+
+        C_ASSERT(COUNT_PAGE == countof(PAGE_NAMES));
+        C_ASSERT(countof(_pageIds) == countof(PAGE_NAMES));
+
+        ThemeGetPageIds(_theme, PAGE_NAMES, _pageIds, countof(_pageIds));
+
+        // Initialize the text on all "application" (non-page) controls.
+        for (DWORD i = 0; i < _theme->cControls; ++i) {
+            THEME_CONTROL* pControl = _theme->rgControls + i;
+            LPWSTR text = nullptr;
+            LPWSTR name = nullptr;
+            LOC_STRING *locText = nullptr;
+
+            // If a command link has a note, then add it.
+            if ((pControl->dwStyle & BS_TYPEMASK) == BS_COMMANDLINK ||
+                (pControl->dwStyle & BS_TYPEMASK) == BS_DEFCOMMANDLINK) {
+                hr = StrAllocFormatted(&name, L"#(loc.%lsNote)", pControl->sczName);
+                if (SUCCEEDED(hr)) {
+                    hr = LocGetString(_wixLoc, name, &locText);
+                    ReleaseStr(name);
+                    if (SUCCEEDED(hr) && locText && locText->wzText && locText->wzText[0]) {
+                        hr = BalFormatString(locText->wzText, &text);
+                        if (SUCCEEDED(hr) && text && text[0]) {
+                            ThemeSendControlMessage(_theme, pControl->wId, BCM_SETNOTE, 0, (LPARAM)text);
+                            ReleaseStr(text);
+                            text = nullptr;
+                        }
+                    }
+                }
+                hr = S_OK;
+            }
+
+            if (!pControl->wPageId && pControl->sczText && *pControl->sczText) {
+                HRESULT hrFormat;
+                
+                // If the wix developer is showing a hidden variable in the UI,
+                // then obviously they don't care about keeping it safe so don't
+                // go down the rabbit hole of making sure that this is securely
+                // freed.
+                hrFormat = BalFormatString(pControl->sczText, &text);
+                if (SUCCEEDED(hrFormat)) {
+                    ThemeSetTextControl(_theme, pControl->wId, text);
+                    ReleaseStr(text);
+                }
+            }
+        }
+
+    LExit:
+        return SUCCEEDED(hr);
+    }
+
+
+    //
+    // OnShowFailure - display the failure page.
+    //
+    void OnShowFailure() {
+        SetState(PYBA_STATE_FAILED, S_OK);
+
+        // If the UI should be visible, display it now and hide the splash screen
+        if (BOOTSTRAPPER_DISPLAY_NONE < _command.display) {
+            ::ShowWindow(_theme->hwndParent, SW_SHOW);
+        }
+
+        _engine->CloseSplashScreen();
+
+        return;
+    }
+
+
+    //
+    // OnShowHelp - display the help page.
+    //
+    void OnShowHelp() {
+        SetState(PYBA_STATE_HELP, S_OK);
+
+        // If the UI should be visible, display it now and hide the splash screen
+        if (BOOTSTRAPPER_DISPLAY_NONE < _command.display) {
+            ::ShowWindow(_theme->hwndParent, SW_SHOW);
+        }
+
+        _engine->CloseSplashScreen();
+
+        return;
+    }
+
+
+    //
+    // OnDetect - start the processing of packages.
+    //
+    void OnDetect() {
+        HRESULT hr = S_OK;
+
+        if (_baFunction) {
+            BalLog(BOOTSTRAPPER_LOG_LEVEL_STANDARD, "Running detect BA function");
+            hr = _baFunction->OnDetect();
+            BalExitOnFailure(hr, "Failed calling detect BA function.");
+        }
+
+        SetState(PYBA_STATE_DETECTING, hr);
+
+        // If the UI should be visible, display it now and hide the splash screen
+        if (BOOTSTRAPPER_DISPLAY_NONE < _command.display) {
+            ::ShowWindow(_theme->hwndParent, SW_SHOW);
+        }
+
+        _engine->CloseSplashScreen();
+
+        // Tell the core we're ready for the packages to be processed now.
+        hr = _engine->Detect();
+        BalExitOnFailure(hr, "Failed to start detecting chain.");
+
+    LExit:
+        if (FAILED(hr)) {
+            SetState(PYBA_STATE_DETECTING, hr);
+        }
+
+        return;
+    }
+
+
+    //
+    // OnPlan - plan the detected changes.
+    //
+    void OnPlan(__in BOOTSTRAPPER_ACTION action) {
+        HRESULT hr = S_OK;
+        LPCWSTR likeInstalling = nullptr;
+        LPCWSTR likeInstallation = nullptr;
+
+        _plannedAction = action;
+
+        switch (action) {
+        case BOOTSTRAPPER_ACTION_INSTALL:
+            likeInstalling = L"Installing";
+            likeInstallation = L"Installation";
+            break;
+        case BOOTSTRAPPER_ACTION_MODIFY:
+            // For modify, we actually want to pass INSTALL
+            action = BOOTSTRAPPER_ACTION_INSTALL;
+            likeInstalling = L"Modifying";
+            likeInstallation = L"Modification";
+            break;
+        case BOOTSTRAPPER_ACTION_REPAIR:
+            likeInstalling = L"Repairing";
+            likeInstallation = L"Repair";
+            break;
+        case BOOTSTRAPPER_ACTION_UNINSTALL:
+            likeInstalling = L"Uninstalling";
+            likeInstallation = L"Uninstallation";
+            break;
+        }
+
+        if (likeInstalling) {
+            LPWSTR locName = nullptr;
+            LOC_STRING *locText = nullptr;
+            hr = StrAllocFormatted(&locName, L"#(loc.%ls)", likeInstalling);
+            if (SUCCEEDED(hr)) {
+                hr = LocGetString(_wixLoc, locName, &locText);
+                ReleaseStr(locName);
+            }
+            _engine->SetVariableString(
+                L"ActionLikeInstalling",
+                SUCCEEDED(hr) && locText ? locText->wzText : likeInstalling
+            );
+        }
+
+        if (likeInstallation) {
+            LPWSTR locName = nullptr;
+            LOC_STRING *locText = nullptr;
+            hr = StrAllocFormatted(&locName, L"#(loc.%ls)", likeInstallation);
+            if (SUCCEEDED(hr)) {
+                hr = LocGetString(_wixLoc, locName, &locText);
+                ReleaseStr(locName);
+            }
+            _engine->SetVariableString(
+                L"ActionLikeInstallation",
+                SUCCEEDED(hr) && locText ? locText->wzText : likeInstallation
+            );
+        }
+
+        // If we are going to apply a downgrade, bail.
+        if (_downgrading && BOOTSTRAPPER_ACTION_UNINSTALL < action) {
+            if (_suppressDowngradeFailure) {
+                BalLog(BOOTSTRAPPER_LOG_LEVEL_STANDARD, "A newer version of this product is installed but downgrade failure has been suppressed; continuing...");
+            } else {
+                hr = HRESULT_FROM_WIN32(ERROR_PRODUCT_VERSION);
+                BalExitOnFailure(hr, "Cannot install a product when a newer version is installed.");
+            }
+        }
+
+        SetState(PYBA_STATE_PLANNING, hr);
+
+        if (_baFunction) {
+            BalLog(BOOTSTRAPPER_LOG_LEVEL_STANDARD, "Running plan BA function");
+            _baFunction->OnPlan();
+        }
+
+        hr = _engine->Plan(action);
+        BalExitOnFailure(hr, "Failed to start planning packages.");
+
+    LExit:
+        if (FAILED(hr)) {
+            SetState(PYBA_STATE_PLANNING, hr);
+        }
+
+        return;
+    }
+
+
+    //
+    // OnApply - apply the packages.
+    //
+    void OnApply() {
+        HRESULT hr = S_OK;
+
+        SetState(PYBA_STATE_APPLYING, hr);
+        SetProgressState(hr);
+        SetTaskbarButtonProgress(0);
+
+        hr = _engine->Apply(_hWnd);
+        BalExitOnFailure(hr, "Failed to start applying packages.");
+
+        ThemeControlEnable(_theme, ID_PROGRESS_CANCEL_BUTTON, TRUE); // ensure the cancel button is enabled before starting.
+
+    LExit:
+        if (FAILED(hr)) {
+            SetState(PYBA_STATE_APPLYING, hr);
+        }
+
+        return;
+    }
+
+
+    //
+    // OnChangeState - change state.
+    //
+    void OnChangeState(__in PYBA_STATE state) {
+        LPWSTR unformattedText = nullptr;
+
+        _state = state;
+
+        // If our install is at the end (success or failure) and we're not showing full UI
+        // then exit (prompt for restart if required).
+        if ((PYBA_STATE_APPLIED <= _state && BOOTSTRAPPER_DISPLAY_FULL > _command.display)) {
+            // If a restart was required but we were not automatically allowed to
+            // accept the reboot then do the prompt.
+            if (_restartRequired && !_allowRestart) {
+                StrAllocFromError(&unformattedText, HRESULT_FROM_WIN32(ERROR_SUCCESS_REBOOT_REQUIRED), nullptr);
+
+                _allowRestart = IDOK == ::MessageBoxW(
+                    _hWnd,
+                    unformattedText ? unformattedText : L"The requested operation is successful. Changes will not be effective until the system is rebooted.",
+                    _theme->sczCaption,
+                    MB_ICONEXCLAMATION | MB_OKCANCEL
+                );
+            }
+
+            // Quietly exit.
+            ::PostMessageW(_hWnd, WM_CLOSE, 0, 0);
+        } else { // try to change the pages.
+            DWORD newPageId = 0;
+            DeterminePageId(_state, &newPageId);
+
+            if (_visiblePageId != newPageId) {
+                ShowPage(newPageId);
+            }
+        }
+
+        ReleaseStr(unformattedText);
+    }
+
+    //
+    // Called before showing a page to handle all controls.
+    //
+    void ProcessPageControls(THEME_PAGE *pPage) {
+        if (!pPage) {
+            return;
+        }
+
+        for (DWORD i = 0; i < pPage->cControlIndices; ++i) {
+            THEME_CONTROL* pControl = _theme->rgControls + pPage->rgdwControlIndices[i];
+
+            // If this is a named control, try to set its default state.
+            if (pControl->sczName && *pControl->sczName) {
+                // If this is a checkable control, try to set its default state
+                // to the state of a matching named Burn variable.
+                if (IsCheckable(pControl)) {
+                    LONGLONG llValue = 0;
+                    HRESULT hr = BalGetNumericVariable(pControl->sczName, &llValue);
+
+                    // If the control value isn't set then disable it.
+                    if (!SUCCEEDED(hr)) {
+                        ThemeControlEnable(_theme, pControl->wId, FALSE);
+                    } else {
+                        ThemeSendControlMessage(
+                            _theme,
+                            pControl->wId,
+                            BM_SETCHECK,
+                            SUCCEEDED(hr) && llValue ? BST_CHECKED : BST_UNCHECKED,
+                            0
+                        );
+                    }
+                }
+
+                // Hide or disable controls based on the control name with 'State' appended
+                LPWSTR controlName = nullptr;
+                HRESULT hr = StrAllocFormatted(&controlName, L"%lsState", pControl->sczName);
+                if (SUCCEEDED(hr)) {
+                    LPWSTR controlState = nullptr;
+                    hr = BalGetStringVariable(controlName, &controlState);
+                    if (SUCCEEDED(hr) && controlState && *controlState) {
+                        if (CSTR_EQUAL == ::CompareStringW(LOCALE_NEUTRAL, 0, controlState, -1, L"disable", -1)) {
+                            BalLog(BOOTSTRAPPER_LOG_LEVEL_STANDARD, "Disable control %ls", pControl->sczName);
+                            ThemeControlEnable(_theme, pControl->wId, FALSE);
+                        } else if (CSTR_EQUAL == ::CompareStringW(LOCALE_NEUTRAL, 0, controlState, -1, L"hide", -1)) {
+                            BalLog(BOOTSTRAPPER_LOG_LEVEL_STANDARD, "Hide control %ls", pControl->sczName);
+                            // TODO: This doesn't work
+                            ThemeShowControl(_theme, pControl->wId, SW_HIDE);
+                        }
+                    }
+                    StrFree(controlState);
+                }
+                StrFree(controlName);
+            }
+
+            // Format the text in each of the new page's controls
+            if (pControl->sczText && *pControl->sczText) {
+                // If the wix developer is showing a hidden variable
+                // in the UI, then obviously they don't care about
+                // keeping it safe so don't go down the rabbit hole
+                // of making sure that this is securely freed.
+                LPWSTR text = nullptr;
+                HRESULT hr = BalFormatString(pControl->sczText, &text);
+                if (SUCCEEDED(hr)) {
+                    ThemeSetTextControl(_theme, pControl->wId, text);
+                }
+            }
+        }
+    }
+
+    //
+    // OnClose - called when the window is trying to be closed.
+    //
+    BOOL OnClose() {
+        BOOL close = FALSE;
+
+        // If we've already succeeded or failed or showing the help page, just close (prompts are annoying if the bootstrapper is done).
+        if (PYBA_STATE_APPLIED <= _state || PYBA_STATE_HELP == _state) {
+            close = TRUE;
+        } else {
+            // prompt the user or force the cancel if there is no UI.
+            close = PromptCancel(
+                _hWnd,
+                BOOTSTRAPPER_DISPLAY_FULL != _command.display,
+                _confirmCloseMessage ? _confirmCloseMessage : L"Are you sure you want to cancel?",
+                _theme->sczCaption
+            );
+        }
+
+        // If we're doing progress then we never close, we just cancel to let rollback occur.
+        if (PYBA_STATE_APPLYING <= _state && PYBA_STATE_APPLIED > _state) {
+            // If we canceled disable cancel button since clicking it again is silly.
+            if (close) {
+                ThemeControlEnable(_theme, ID_PROGRESS_CANCEL_BUTTON, FALSE);
+            }
+
+            close = FALSE;
+        }
+
+        return close;
+    }
+
+    //
+    // OnClickCloseButton - close the application.
+    //
+    void OnClickCloseButton() {
+        ::SendMessageW(_hWnd, WM_CLOSE, 0, 0);
+    }
+
+
+    //
+    // OnClickLaunchButton - launch the app from the success page.
+    //
+    void OnClickLaunchButton() {
+        HRESULT hr = S_OK;
+        LPWSTR sczUnformattedLaunchTarget = nullptr;
+        LPWSTR sczLaunchTarget = nullptr;
+        LPWSTR sczLaunchTargetElevatedId = nullptr;
+        LPWSTR sczUnformattedArguments = nullptr;
+        LPWSTR sczArguments = nullptr;
+        int nCmdShow = SW_SHOWNORMAL;
+
+        hr = BalGetStringVariable(PYBA_VARIABLE_LAUNCH_TARGET_PATH, &sczUnformattedLaunchTarget);
+        BalExitOnFailure1(hr, "Failed to get launch target variable '%ls'.", PYBA_VARIABLE_LAUNCH_TARGET_PATH);
+
+        hr = BalFormatString(sczUnformattedLaunchTarget, &sczLaunchTarget);
+        BalExitOnFailure1(hr, "Failed to format launch target variable: %ls", sczUnformattedLaunchTarget);
+
+        if (BalStringVariableExists(PYBA_VARIABLE_LAUNCH_TARGET_ELEVATED_ID)) {
+            hr = BalGetStringVariable(PYBA_VARIABLE_LAUNCH_TARGET_ELEVATED_ID, &sczLaunchTargetElevatedId);
+            BalExitOnFailure1(hr, "Failed to get launch target elevated id '%ls'.", PYBA_VARIABLE_LAUNCH_TARGET_ELEVATED_ID);
+        }
+
+        if (BalStringVariableExists(PYBA_VARIABLE_LAUNCH_ARGUMENTS)) {
+            hr = BalGetStringVariable(PYBA_VARIABLE_LAUNCH_ARGUMENTS, &sczUnformattedArguments);
+            BalExitOnFailure1(hr, "Failed to get launch arguments '%ls'.", PYBA_VARIABLE_LAUNCH_ARGUMENTS);
+        }
+
+        if (BalStringVariableExists(PYBA_VARIABLE_LAUNCH_HIDDEN)) {
+            nCmdShow = SW_HIDE;
+        }
+
+        if (sczLaunchTargetElevatedId && !_triedToLaunchElevated) {
+            _triedToLaunchElevated = TRUE;
+            hr = _engine->LaunchApprovedExe(_hWnd, sczLaunchTargetElevatedId, sczUnformattedArguments, 0);
+            if (FAILED(hr)) {
+                BalLogError(hr, "Failed to launch elevated target: %ls", sczLaunchTargetElevatedId);
+
+                //try with ShelExec next time
+                OnClickLaunchButton();
+            }
+        } else {
+            if (sczUnformattedArguments) {
+                hr = BalFormatString(sczUnformattedArguments, &sczArguments);
+                BalExitOnFailure1(hr, "Failed to format launch arguments variable: %ls", sczUnformattedArguments);
+            }
+
+            hr = ShelExec(sczLaunchTarget, sczArguments, L"open", nullptr, nCmdShow, _hWnd, nullptr);
+            BalExitOnFailure1(hr, "Failed to launch target: %ls", sczLaunchTarget);
+
+            ::PostMessageW(_hWnd, WM_CLOSE, 0, 0);
+        }
+
+    LExit:
+        StrSecureZeroFreeString(sczArguments);
+        ReleaseStr(sczUnformattedArguments);
+        ReleaseStr(sczLaunchTargetElevatedId);
+        StrSecureZeroFreeString(sczLaunchTarget);
+        ReleaseStr(sczUnformattedLaunchTarget);
+
+        return;
+    }
+
+
+    //
+    // OnClickRestartButton - allows the restart and closes the app.
+    //
+    void OnClickRestartButton() {
+        AssertSz(_restartRequired, "Restart must be requested to be able to click on the restart button.");
+
+        _allowRestart = TRUE;
+        ::SendMessageW(_hWnd, WM_CLOSE, 0, 0);
+
+        return;
+    }
+
+
+    //
+    // OnClickLogFileLink - show the log file.
+    //
+    void OnClickLogFileLink() {
+        HRESULT hr = S_OK;
+        LPWSTR sczLogFile = nullptr;
+
+        hr = BalGetStringVariable(_bundle.sczLogVariable, &sczLogFile);
+        BalExitOnFailure1(hr, "Failed to get log file variable '%ls'.", _bundle.sczLogVariable);
+
+        hr = ShelExec(L"notepad.exe", sczLogFile, L"open", nullptr, SW_SHOWDEFAULT, _hWnd, nullptr);
+        BalExitOnFailure1(hr, "Failed to open log file target: %ls", sczLogFile);
+
+    LExit:
+        ReleaseStr(sczLogFile);
+
+        return;
+    }
+
+
+    //
+    // SetState
+    //
+    void SetState(__in PYBA_STATE state, __in HRESULT hrStatus) {
+        if (FAILED(hrStatus)) {
+            _hrFinal = hrStatus;
+        }
+
+        if (FAILED(_hrFinal)) {
+            state = PYBA_STATE_FAILED;
+        }
+
+        if (_state != state) {
+            ::PostMessageW(_hWnd, WM_PYBA_CHANGE_STATE, 0, state);
+        }
+    }
+
+    //
+    // GoToPage
+    //
+    void GoToPage(__in PAGE page) {
+        _installPage = page;
+        ::PostMessageW(_hWnd, WM_PYBA_CHANGE_STATE, 0, _state);
+    }
+
+    void DeterminePageId(__in PYBA_STATE state, __out DWORD* pdwPageId) {
+        LONGLONG simple;
+
+        if (BOOTSTRAPPER_DISPLAY_PASSIVE == _command.display) {
+            switch (state) {
+            case PYBA_STATE_INITIALIZED:
+                *pdwPageId = BOOTSTRAPPER_ACTION_HELP == _command.action
+                    ? _pageIds[PAGE_HELP]
+                    : _pageIds[PAGE_LOADING];
+                break;
+
+            case PYBA_STATE_HELP:
+                *pdwPageId = _pageIds[PAGE_HELP];
+                break;
+
+            case PYBA_STATE_DETECTING:
+                *pdwPageId = _pageIds[PAGE_LOADING]
+                    ? _pageIds[PAGE_LOADING]
+                    : _pageIds[PAGE_PROGRESS_PASSIVE]
+                        ? _pageIds[PAGE_PROGRESS_PASSIVE]
+                        : _pageIds[PAGE_PROGRESS];
+                break;
+
+            case PYBA_STATE_DETECTED: __fallthrough;
+            case PYBA_STATE_PLANNING: __fallthrough;
+            case PYBA_STATE_PLANNED: __fallthrough;
+            case PYBA_STATE_APPLYING: __fallthrough;
+            case PYBA_STATE_CACHING: __fallthrough;
+            case PYBA_STATE_CACHED: __fallthrough;
+            case PYBA_STATE_EXECUTING: __fallthrough;
+            case PYBA_STATE_EXECUTED:
+                *pdwPageId = _pageIds[PAGE_PROGRESS_PASSIVE]
+                    ? _pageIds[PAGE_PROGRESS_PASSIVE]
+                    : _pageIds[PAGE_PROGRESS];
+                break;
+
+            default:
+                *pdwPageId = 0;
+                break;
+            }
+        } else if (BOOTSTRAPPER_DISPLAY_FULL == _command.display) {
+            switch (state) {
+            case PYBA_STATE_INITIALIZING:
+                *pdwPageId = 0;
+                break;
+
+            case PYBA_STATE_INITIALIZED:
+                *pdwPageId = BOOTSTRAPPER_ACTION_HELP == _command.action
+                    ? _pageIds[PAGE_HELP]
+                    : _pageIds[PAGE_LOADING];
+                break;
+
+            case PYBA_STATE_HELP:
+                *pdwPageId = _pageIds[PAGE_HELP];
+                break;
+
+            case PYBA_STATE_DETECTING:
+                *pdwPageId = _pageIds[PAGE_LOADING];
+                break;
+
+            case PYBA_STATE_DETECTED:
+                if (_installPage == PAGE_LOADING) {
+                    switch (_command.action) {
+                    case BOOTSTRAPPER_ACTION_INSTALL:
+                        if (SUCCEEDED(BalGetNumericVariable(L"SimpleInstall", &simple)) && simple) {
+                            _installPage = PAGE_SIMPLE_INSTALL;
+                        } else {
+                            _installPage = PAGE_INSTALL;
+                        }
+                        break;
+
+                    case BOOTSTRAPPER_ACTION_MODIFY: __fallthrough;
+                    case BOOTSTRAPPER_ACTION_REPAIR: __fallthrough;
+                    case BOOTSTRAPPER_ACTION_UNINSTALL:
+                        _installPage = PAGE_MODIFY;
+                        break;
+                    }
+                }
+                *pdwPageId = _pageIds[_installPage];
+                break;
+
+            case PYBA_STATE_PLANNING: __fallthrough;
+            case PYBA_STATE_PLANNED: __fallthrough;
+            case PYBA_STATE_APPLYING: __fallthrough;
+            case PYBA_STATE_CACHING: __fallthrough;
+            case PYBA_STATE_CACHED: __fallthrough;
+            case PYBA_STATE_EXECUTING: __fallthrough;
+            case PYBA_STATE_EXECUTED:
+                *pdwPageId = _pageIds[PAGE_PROGRESS];
+                break;
+
+            case PYBA_STATE_APPLIED:
+                *pdwPageId = _pageIds[PAGE_SUCCESS];
+                break;
+
+            case PYBA_STATE_FAILED:
+                *pdwPageId = _pageIds[PAGE_FAILURE];
+                break;
+            }
+        }
+    }
+
+
+    HRESULT EvaluateConditions() {
+        HRESULT hr = S_OK;
+        BOOL result = FALSE;
+
+        for (DWORD i = 0; i < _conditions.cConditions; ++i) {
+            BAL_CONDITION* pCondition = _conditions.rgConditions + i;
+
+            hr = BalConditionEvaluate(pCondition, _engine, &result, &_failedMessage);
+            BalExitOnFailure(hr, "Failed to evaluate condition.");
+
+            if (!result) {
+                // Hope they didn't have hidden variables in their message, because it's going in the log in plaintext.
+                BalLog(BOOTSTRAPPER_LOG_LEVEL_ERROR, "%ls", _failedMessage);
+
+                hr = E_WIXSTDBA_CONDITION_FAILED;
+                // todo: remove in WiX v4, in case people are relying on v3.x logging behavior
+                BalExitOnFailure1(hr, "Bundle condition evaluated to false: %ls", pCondition->sczCondition);
+            }
+        }
+
+        ReleaseNullStrSecure(_failedMessage);
+
+    LExit:
+        return hr;
+    }
+
+
+    void SetTaskbarButtonProgress(__in DWORD dwOverallPercentage) {
+        HRESULT hr = S_OK;
+
+        if (_taskbarButtonOK) {
+            hr = _taskbarList->SetProgressValue(_hWnd, dwOverallPercentage, 100UL);
+            BalExitOnFailure1(hr, "Failed to set taskbar button progress to: %d%%.", dwOverallPercentage);
+        }
+
+    LExit:
+        return;
+    }
+
+
+    void SetTaskbarButtonState(__in TBPFLAG tbpFlags) {
+        HRESULT hr = S_OK;
+
+        if (_taskbarButtonOK) {
+            hr = _taskbarList->SetProgressState(_hWnd, tbpFlags);
+            BalExitOnFailure1(hr, "Failed to set taskbar button state.", tbpFlags);
+        }
+
+    LExit:
+        return;
+    }
+
+
+    void SetProgressState(__in HRESULT hrStatus) {
+        TBPFLAG flag = TBPF_NORMAL;
+
+        if (IsCanceled() || HRESULT_FROM_WIN32(ERROR_INSTALL_USEREXIT) == hrStatus) {
+            flag = TBPF_PAUSED;
+        } else if (IsRollingBack() || FAILED(hrStatus)) {
+            flag = TBPF_ERROR;
+        }
+
+        SetTaskbarButtonState(flag);
+    }
+
+
+    HRESULT LoadBootstrapperBAFunctions() {
+        HRESULT hr = S_OK;
+        LPWSTR sczBafPath = nullptr;
+
+        hr = PathRelativeToModule(&sczBafPath, L"bafunctions.dll", _hModule);
+        BalExitOnFailure(hr, "Failed to get path to BA function DLL.");
+
+#ifdef DEBUG
+        BalLog(BOOTSTRAPPER_LOG_LEVEL_STANDARD, "PYBA: LoadBootstrapperBAFunctions() - BA function DLL %ls", sczBafPath);
+#endif
+
+        _hBAFModule = ::LoadLibraryW(sczBafPath);
+        if (_hBAFModule) {
+            auto pfnBAFunctionCreate = reinterpret_cast<PFN_BOOTSTRAPPER_BA_FUNCTION_CREATE>(::GetProcAddress(_hBAFModule, "CreateBootstrapperBAFunction"));
+            BalExitOnNullWithLastError1(pfnBAFunctionCreate, hr, "Failed to get CreateBootstrapperBAFunction entry-point from: %ls", sczBafPath);
+
+            hr = pfnBAFunctionCreate(_engine, _hBAFModule, &_baFunction);
+            BalExitOnFailure(hr, "Failed to create BA function.");
+        }
+#ifdef DEBUG
+        else {
+            BalLogError(HRESULT_FROM_WIN32(::GetLastError()), "PYBA: LoadBootstrapperBAFunctions() - Failed to load DLL %ls", sczBafPath);
+        }
+#endif
+
+    LExit:
+        if (_hBAFModule && !_baFunction) {
+            ::FreeLibrary(_hBAFModule);
+            _hBAFModule = nullptr;
+        }
+        ReleaseStr(sczBafPath);
+
+        return hr;
+    }
+
+    BOOL IsCheckable(THEME_CONTROL* pControl) {
+        if (!pControl->sczName || !pControl->sczName[0]) {
+            return FALSE;
+        }
+
+        if (pControl->type == THEME_CONTROL_TYPE_CHECKBOX) {
+            return TRUE;
+        }
+
+        if (pControl->type == THEME_CONTROL_TYPE_BUTTON) {
+            if ((pControl->dwStyle & BS_TYPEMASK) == BS_AUTORADIOBUTTON) {
+                return TRUE;
+            }
+        }
+
+        return FALSE;
+    }
+
+    void SavePageSettings() {
+        DWORD pageId = 0;
+        THEME_PAGE* pPage = nullptr;
+
+        DeterminePageId(_state, &pageId);
+        pPage = ThemeGetPage(_theme, pageId);
+        if (!pPage) {
+            return;
+        }
+
+        for (DWORD i = 0; i < pPage->cControlIndices; ++i) {
+            // Loop through all the checkable controls and set a Burn variable
+            // with that name to true or false.
+            THEME_CONTROL* pControl = _theme->rgControls + pPage->rgdwControlIndices[i];
+            if (IsCheckable(pControl) && ThemeControlEnabled(_theme, pControl->wId)) {
+                BOOL checked = ThemeIsControlChecked(_theme, pControl->wId);
+                _engine->SetVariableNumeric(pControl->sczName, checked ? 1 : 0);
+            }
+
+            // Loop through all the editbox controls with names and set a
+            // Burn variable with that name to the contents.
+            if (THEME_CONTROL_TYPE_EDITBOX == pControl->type && pControl->sczName && *pControl->sczName) {
+                LPWSTR sczValue = nullptr;
+                ThemeGetTextControl(_theme, pControl->wId, &sczValue);
+                _engine->SetVariableString(pControl->sczName, sczValue);
+            }
+        }
+    }
+
+
+public:
+    //
+    // Constructor - initialize member variables.
+    //
+    PythonBootstrapperApplication(
+        __in HMODULE hModule,
+        __in BOOL fPrereq,
+        __in HRESULT hrHostInitialization,
+        __in IBootstrapperEngine* pEngine,
+        __in const BOOTSTRAPPER_COMMAND* pCommand
+    ) : CBalBaseBootstrapperApplication(pEngine, pCommand, 3, 3000) {
+        _hModule = hModule;
+        memcpy_s(&_command, sizeof(_command), pCommand, sizeof(BOOTSTRAPPER_COMMAND));
+
+        LONGLONG llInstalled = 0;
+        HRESULT hr = BalGetNumericVariable(L"WixBundleInstalled", &llInstalled);
+        if (SUCCEEDED(hr) && BOOTSTRAPPER_RESUME_TYPE_REBOOT != _command.resumeType && 0 < llInstalled && BOOTSTRAPPER_ACTION_INSTALL == _command.action) {
+            _command.action = BOOTSTRAPPER_ACTION_MODIFY;
+        } else if (0 == llInstalled && (BOOTSTRAPPER_ACTION_MODIFY == _command.action || BOOTSTRAPPER_ACTION_REPAIR == _command.action)) {
+            _command.action = BOOTSTRAPPER_ACTION_INSTALL;
+        }
+
+        _plannedAction = BOOTSTRAPPER_ACTION_UNKNOWN;
+
+
+        // When resuming from restart doing some install-like operation, try to find the package that forced the
+        // restart. We'll use this information during planning.
+        _nextPackageAfterRestart = nullptr;
+
+        if (BOOTSTRAPPER_RESUME_TYPE_REBOOT == _command.resumeType && BOOTSTRAPPER_ACTION_UNINSTALL < _command.action) {
+            // Ensure the forced restart package variable is null when it is an empty string.
+            HRESULT hr = BalGetStringVariable(L"WixBundleForcedRestartPackage", &_nextPackageAfterRestart);
+            if (FAILED(hr) || !_nextPackageAfterRestart || !*_nextPackageAfterRestart) {
+                ReleaseNullStr(_nextPackageAfterRestart);
+            }
+        }
+
+        _wixLoc = nullptr;
+        memset(&_bundle, 0, sizeof(_bundle));
+        memset(&_conditions, 0, sizeof(_conditions));
+        _confirmCloseMessage = nullptr;
+        _failedMessage = nullptr;
+
+        _language = nullptr;
+        _theme = nullptr;
+        memset(_pageIds, 0, sizeof(_pageIds));
+        _hUiThread = nullptr;
+        _registered = FALSE;
+        _hWnd = nullptr;
+
+        _state = PYBA_STATE_INITIALIZING;
+        _visiblePageId = 0;
+        _installPage = PAGE_LOADING;
+        _hrFinal = hrHostInitialization;
+
+        _downgrading = FALSE;
+        _restartResult = BOOTSTRAPPER_APPLY_RESTART_NONE;
+        _restartRequired = FALSE;
+        _allowRestart = FALSE;
+
+        _suppressDowngradeFailure = FALSE;
+        _suppressRepair = FALSE;
+
+        _overridableVariables = nullptr;
+        _taskbarList = nullptr;
+        _taskbarButtonCreatedMessage = UINT_MAX;
+        _taskbarButtonOK = FALSE;
+        _showingInternalUIThisPackage = FALSE;
+        _triedToLaunchElevated = FALSE;
+
+        _suppressPaint = FALSE;
+
+        pEngine->AddRef();
+        _engine = pEngine;
+
+        _hBAFModule = nullptr;
+        _baFunction = nullptr;
+    }
+
+
+    //
+    // Destructor - release member variables.
+    //
+    ~PythonBootstrapperApplication() {
+        AssertSz(!::IsWindow(_hWnd), "Window should have been destroyed before destructor.");
+        AssertSz(!_theme, "Theme should have been released before destructor.");
+
+        ReleaseObject(_taskbarList);
+        ReleaseDict(_overridableVariables);
+        ReleaseStr(_failedMessage);
+        ReleaseStr(_confirmCloseMessage);
+        BalConditionsUninitialize(&_conditions);
+        BalInfoUninitialize(&_bundle);
+        LocFree(_wixLoc);
+
+        ReleaseStr(_language);
+        ReleaseStr(_nextPackageAfterRestart);
+        ReleaseNullObject(_engine);
+
+        if (_hBAFModule) {
+            ::FreeLibrary(_hBAFModule);
+            _hBAFModule = nullptr;
+        }
+    }
+
+private:
+    HMODULE _hModule;
+    BOOTSTRAPPER_COMMAND _command;
+    IBootstrapperEngine* _engine;
+    BOOTSTRAPPER_ACTION _plannedAction;
+
+    LPWSTR _nextPackageAfterRestart;
+
+    WIX_LOCALIZATION* _wixLoc;
+    BAL_INFO_BUNDLE _bundle;
+    BAL_CONDITIONS _conditions;
+    LPWSTR _failedMessage;
+    LPWSTR _confirmCloseMessage;
+
+    LPWSTR _language;
+    THEME* _theme;
+    DWORD _pageIds[countof(PAGE_NAMES)];
+    HANDLE _hUiThread;
+    BOOL _registered;
+    HWND _hWnd;
+
+    PYBA_STATE _state;
+    HRESULT _hrFinal;
+    DWORD _visiblePageId;
+    PAGE _installPage;
+
+    BOOL _startedExecution;
+    DWORD _calculatedCacheProgress;
+    DWORD _calculatedExecuteProgress;
+
+    BOOL _downgrading;
+    BOOTSTRAPPER_APPLY_RESTART _restartResult;
+    BOOL _restartRequired;
+    BOOL _allowRestart;
+
+    BOOL _suppressDowngradeFailure;
+    BOOL _suppressRepair;
+
+    STRINGDICT_HANDLE _overridableVariables;
+
+    ITaskbarList3* _taskbarList;
+    UINT _taskbarButtonCreatedMessage;
+    BOOL _taskbarButtonOK;
+    BOOL _showingInternalUIThisPackage;
+    BOOL _triedToLaunchElevated;
+
+    BOOL _suppressPaint;
+
+    HMODULE _hBAFModule;
+    IBootstrapperBAFunction* _baFunction;
+};
+
+//
+// CreateBootstrapperApplication - creates a new IBootstrapperApplication object.
+//
+HRESULT CreateBootstrapperApplication(
+    __in HMODULE hModule,
+    __in BOOL fPrereq,
+    __in HRESULT hrHostInitialization,
+    __in IBootstrapperEngine* pEngine,
+    __in const BOOTSTRAPPER_COMMAND* pCommand,
+    __out IBootstrapperApplication** ppApplication
+    ) {
+    HRESULT hr = S_OK;
+
+    if (fPrereq) {
+        hr = E_INVALIDARG;
+        ExitWithLastError(hr, "Failed to create UI thread.");
+    }
+
+    PythonBootstrapperApplication* pApplication = nullptr;
+
+    pApplication = new PythonBootstrapperApplication(hModule, fPrereq, hrHostInitialization, pEngine, pCommand);
+    ExitOnNull(pApplication, hr, E_OUTOFMEMORY, "Failed to create new standard bootstrapper application object.");
+
+    *ppApplication = pApplication;
+    pApplication = nullptr;
+
+LExit:
+    ReleaseObject(pApplication);
+    return hr;
+}
diff --git a/Tools/msi/bundle/bootstrap/pch.cpp b/Tools/msi/bundle/bootstrap/pch.cpp
new file mode 100644
--- /dev/null
+++ b/Tools/msi/bundle/bootstrap/pch.cpp
@@ -0,0 +1,1 @@
+#include "pch.h"
diff --git a/Tools/msi/bundle/bootstrap/pch.h b/Tools/msi/bundle/bootstrap/pch.h
new file mode 100644
--- /dev/null
+++ b/Tools/msi/bundle/bootstrap/pch.h
@@ -0,0 +1,59 @@
+//-------------------------------------------------------------------------------------------------
+// <copyright file="precomp.h" company="Outercurve Foundation">
+//   Copyright (c) 2004, Outercurve Foundation.
+//   This software is released under Microsoft Reciprocal License (MS-RL).
+//   The license and further copyright text can be found in the file
+//   LICENSE.TXT at the root directory of the distribution.
+// </copyright>
+// 
+// <summary>
+//    Precompiled header for standard bootstrapper application.
+// </summary>
+//-------------------------------------------------------------------------------------------------
+
+#pragma once
+
+#include <windows.h>
+#include <gdiplus.h>
+#include <Uxtheme.h>
+#include <msiquery.h>
+#include <objbase.h>
+#include <shlobj.h>
+#include <shlwapi.h>
+#include <stdlib.h>
+#include <strsafe.h>
+#include <stddef.h>
+
+#include "dutil.h"
+#include "memutil.h"
+#include "dictutil.h"
+#include "dirutil.h"
+#include "fileutil.h"
+#include "locutil.h"
+#include "logutil.h"
+#include "pathutil.h"
+#include "resrutil.h"
+#include "shelutil.h"
+#include "strutil.h"
+#include "thmutil.h"
+#include "uriutil.h"
+#include "xmlutil.h"
+
+#include "IBootstrapperEngine.h"
+#include "IBootstrapperApplication.h"
+
+#include "BalBaseBootstrapperApplication.h"
+#include "balinfo.h"
+#include "balcondition.h"
+
+HRESULT CreateBootstrapperApplication(
+    __in HMODULE hModule,
+    __in BOOL fPrereq,
+    __in HRESULT hrHostInitialization,
+    __in IBootstrapperEngine* pEngine,
+    __in const BOOTSTRAPPER_COMMAND* pCommand,
+    __out IBootstrapperApplication** ppApplication
+);
+
+#include "IBootstrapperBAFunction.h"
+
diff --git a/Tools/msi/bundle/bootstrap/pythonba.cpp b/Tools/msi/bundle/bootstrap/pythonba.cpp
new file mode 100644
--- /dev/null
+++ b/Tools/msi/bundle/bootstrap/pythonba.cpp
@@ -0,0 +1,76 @@
+//-------------------------------------------------------------------------------------------------
+// <copyright file="wixstdba.cpp" company="Outercurve Foundation">
+//   Copyright (c) 2004, Outercurve Foundation.
+//   This software is released under Microsoft Reciprocal License (MS-RL).
+//   The license and further copyright text can be found in the file
+//   LICENSE.TXT at the root directory of the distribution.
+// </copyright>
+//
+// <summary>
+// Setup chainer/bootstrapper standard UI for WiX toolset.
+// </summary>
+//-------------------------------------------------------------------------------------------------
+
+#include "pch.h"
+
+static HINSTANCE vhInstance = NULL;
+
+extern "C" BOOL WINAPI DllMain(
+    IN HINSTANCE hInstance,
+    IN DWORD dwReason,
+    IN LPVOID /* pvReserved */
+    )
+{
+    switch(dwReason)
+    {
+    case DLL_PROCESS_ATTACH:
+        ::DisableThreadLibraryCalls(hInstance);
+        vhInstance = hInstance;
+        break;
+
+    case DLL_PROCESS_DETACH:
+        vhInstance = NULL;
+        break;
+    }
+
+    return TRUE;
+}
+
+
+extern "C" HRESULT WINAPI BootstrapperApplicationCreate(
+    __in IBootstrapperEngine* pEngine,
+    __in const BOOTSTRAPPER_COMMAND* pCommand,
+    __out IBootstrapperApplication** ppApplication
+    )
+{
+    HRESULT hr = S_OK;
+
+    BalInitialize(pEngine);
+
+    hr = CreateBootstrapperApplication(vhInstance, FALSE, S_OK, pEngine, pCommand, ppApplication);
+    BalExitOnFailure(hr, "Failed to create bootstrapper application interface.");
+
+LExit:
+    return hr;
+}
+
+
+extern "C" void WINAPI BootstrapperApplicationDestroy()
+{
+    BalUninitialize();
+}
+
+
+extern "C" HRESULT WINAPI MbaPrereqBootstrapperApplicationCreate(
+    __in HRESULT hrHostInitialization,
+    __in IBootstrapperEngine* pEngine,
+    __in const BOOTSTRAPPER_COMMAND* pCommand,
+    __out IBootstrapperApplication** ppApplication
+    )
+{
+    return E_NOTIMPL;
+}
+
+
+extern "C" void WINAPI MbaPrereqBootstrapperApplicationDestroy()
+{ }
diff --git a/Tools/msi/bundle/bootstrap/pythonba.def b/Tools/msi/bundle/bootstrap/pythonba.def
new file mode 100644
--- /dev/null
+++ b/Tools/msi/bundle/bootstrap/pythonba.def
@@ -0,0 +1,18 @@
+;-------------------------------------------------------------------------------------------------
+; <copyright file="wixstdba.def" company="Outercurve Foundation">
+;   Copyright (c) 2004, Outercurve Foundation.
+;   This software is released under Microsoft Reciprocal License (MS-RL).
+;   The license and further copyright text can be found in the file
+;   LICENSE.TXT at the root directory of the distribution.
+; </copyright>
+; 
+; <summary>
+;    WiX Standard Bootstrapper Application DLL entry points.
+; </summary>
+;-------------------------------------------------------------------------------------------------
+
+EXPORTS
+    BootstrapperApplicationCreate
+    BootstrapperApplicationDestroy
+    MbaPrereqBootstrapperApplicationCreate
+    MbaPrereqBootstrapperApplicationDestroy
diff --git a/Tools/msi/bundle/bootstrap/pythonba.sln b/Tools/msi/bundle/bootstrap/pythonba.sln
new file mode 100644
--- /dev/null
+++ b/Tools/msi/bundle/bootstrap/pythonba.sln
@@ -0,0 +1,22 @@
+?
+Microsoft Visual Studio Solution File, Format Version 12.00
+# Visual Studio 2013
+VisualStudioVersion = 12.0.30501.0
+MinimumVisualStudioVersion = 10.0.40219.1
+Project("{8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942}") = "pythonba", "pythonba.vcxproj", "{7A09B132-B3EE-499B-A700-A4B2157FEA3D}"
+EndProject
+Global
+	GlobalSection(SolutionConfigurationPlatforms) = preSolution
+		Debug|Win32 = Debug|Win32
+		Release|Win32 = Release|Win32
+	EndGlobalSection
+	GlobalSection(ProjectConfigurationPlatforms) = postSolution
+		{7A09B132-B3EE-499B-A700-A4B2157FEA3D}.Debug|Win32.ActiveCfg = Debug|Win32
+		{7A09B132-B3EE-499B-A700-A4B2157FEA3D}.Debug|Win32.Build.0 = Debug|Win32
+		{7A09B132-B3EE-499B-A700-A4B2157FEA3D}.Release|Win32.ActiveCfg = Release|Win32
+		{7A09B132-B3EE-499B-A700-A4B2157FEA3D}.Release|Win32.Build.0 = Release|Win32
+	EndGlobalSection
+	GlobalSection(SolutionProperties) = preSolution
+		HideSolutionNode = FALSE
+	EndGlobalSection
+EndGlobal
diff --git a/Tools/msi/bundle/bootstrap/pythonba.vcxproj b/Tools/msi/bundle/bootstrap/pythonba.vcxproj
new file mode 100644
--- /dev/null
+++ b/Tools/msi/bundle/bootstrap/pythonba.vcxproj
@@ -0,0 +1,69 @@
+?<?xml version="1.0" encoding="utf-8"?>
+<!--
+  <copyright file="wixstdba.vcxproj" company="Outercurve Foundation">
+    Copyright (c) 2004, Outercurve Foundation.
+    This software is released under Microsoft Reciprocal License (MS-RL).
+    The license and further copyright text can be found in the file
+    LICENSE.TXT at the root directory of the distribution.
+  </copyright>
+-->
+<Project DefaultTargets="Build" ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+  <ItemGroup Label="ProjectConfigurations">
+    <ProjectConfiguration Include="Debug|Win32">
+      <Configuration>Debug</Configuration>
+      <Platform>Win32</Platform>
+    </ProjectConfiguration>
+    <ProjectConfiguration Include="Release|Win32">
+      <Configuration>Release</Configuration>
+      <Platform>Win32</Platform>
+    </ProjectConfiguration>
+  </ItemGroup>
+  <PropertyGroup Label="Globals">
+    <Configuration Condition="'$(Configuration)' == ''">Release</Configuration>
+    <Platform Condition="'$(Platform)' == ''">Win32</Platform>
+    <PlatformToolset Condition="'$(PlatformToolset)' == '' and '$(VCTargetsPath14)' != ''">v140</PlatformToolset>
+    <PlatformToolset Condition="'$(PlatformToolset)' == '' and '$(VCTargetsPath12)' != ''">v120</PlatformToolset>
+    <ProjectGuid>{7A09B132-B3EE-499B-A700-A4B2157FEA3D}</ProjectGuid>
+    <TargetName>PythonBA</TargetName>
+  </PropertyGroup>
+  <Import Project="$(VCTargetsPath)\Microsoft.Cpp.Default.props" />
+  <Import Project="..\..\wix.props" />
+  <PropertyGroup Label="Configuration">
+    <ConfigurationType>DynamicLibrary</ConfigurationType>
+    <CharacterSet>Unicode</CharacterSet>
+    <IntDir>$(ProjectDir)..\..\obj\$(Configuration)_Bootstrap\</IntDir>
+    <OutDir>$(IntDir)</OutDir>
+  </PropertyGroup>
+  <Import Project="$(VCTargetsPath)\Microsoft.Cpp.props" />
+  <ItemDefinitionGroup>
+    <ClCompile>
+      <PreprocessorDefinitions>_CRT_STDIO_LEGACY_WIDE_SPECIFIERS=1;%(PreprocessorDefinitions)</PreprocessorDefinitions>
+      <AdditionalIncludeDirectories>$(WixInstallPath)sdk\inc</AdditionalIncludeDirectories>
+      <PrecompiledHeader>Use</PrecompiledHeader>
+      <PrecompiledHeaderFile>pch.h</PrecompiledHeaderFile>
+      <RuntimeLibrary>MultiThreaded</RuntimeLibrary>
+    </ClCompile>
+    <Link>
+      <AdditionalDependencies>comctl32.lib;gdiplus.lib;msimg32.lib;shlwapi.lib;wininet.lib;dutil.lib;balutil.lib;version.lib;uxtheme.lib;%(AdditionalDependencies)</AdditionalDependencies>
+      <AdditionalLibraryDirectories Condition="'$(PlatformToolset)' == 'v140'">$(WixInstallPath)sdk\vs2015\lib\x86</AdditionalLibraryDirectories>
+      <AdditionalLibraryDirectories Condition="'$(PlatformToolset)' == 'v120'">$(WixInstallPath)sdk\vs2013\lib\x86</AdditionalLibraryDirectories>
+      <ModuleDefinitionFile>pythonba.def</ModuleDefinitionFile>
+      <GenerateDebugInformation Condition="'$(Configuration)'=='Debug'">true</GenerateDebugInformation>
+    </Link>
+  </ItemDefinitionGroup>
+  <ItemGroup>
+    <ClCompile Include="PythonBootstrapperApplication.cpp" />
+    <ClCompile Include="pythonba.cpp" />
+    <ClCompile Include="pch.cpp">
+      <PrecompiledHeader>Create</PrecompiledHeader>
+    </ClCompile>
+  </ItemGroup>
+  <ItemGroup>
+    <ClInclude Include="pch.h" />
+    <ClInclude Include="resource.h" />
+  </ItemGroup>
+  <ItemGroup>
+    <None Include="pythonba.def" />
+  </ItemGroup>
+  <Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
+</Project>
\ No newline at end of file
diff --git a/Tools/msi/bundle/bootstrap/resource.h b/Tools/msi/bundle/bootstrap/resource.h
new file mode 100644
--- /dev/null
+++ b/Tools/msi/bundle/bootstrap/resource.h
@@ -0,0 +1,25 @@
+//-------------------------------------------------------------------------------------------------
+// <copyright file="resource.h" company="Outercurve Foundation">
+//   Copyright (c) 2004, Outercurve Foundation.
+//   This software is released under Microsoft Reciprocal License (MS-RL).
+//   The license and further copyright text can be found in the file
+//   LICENSE.TXT at the root directory of the distribution.
+// </copyright>
+//-------------------------------------------------------------------------------------------------
+
+//{{NO_DEPENDENCIES}}
+// Microsoft Visual C++ generated include file.
+//
+#define IDC_STATIC                      -1
+
+
+// Next default values for new objects
+// 
+#ifdef APSTUDIO_INVOKED
+#ifndef APSTUDIO_READONLY_SYMBOLS
+#define _APS_NEXT_RESOURCE_VALUE        102
+#define _APS_NEXT_COMMAND_VALUE         40001
+#define _APS_NEXT_CONTROL_VALUE         1003
+#define _APS_NEXT_SYMED_VALUE           101
+#endif
+#endif
diff --git a/Tools/msi/bundle/bundle.ico b/Tools/msi/bundle/bundle.ico
new file mode 100644
index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..1ab629eff26946e1b8c0cd3223e80257b74deb88
GIT binary patch
[stripped]
diff --git a/Tools/msi/bundle/bundle.targets b/Tools/msi/bundle/bundle.targets
new file mode 100644
--- /dev/null
+++ b/Tools/msi/bundle/bundle.targets
@@ -0,0 +1,99 @@
+<?xml version="1.0" encoding="utf-8"?>
+<Project ToolsVersion="4.0" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+    <PropertyGroup>
+        <SchemaVersion>2.0</SchemaVersion>
+        <OutputType>Bundle</OutputType>
+        
+        <BootstrapConfiguration Condition="'$(BootstrapConfiguration)' == ''">Release</BootstrapConfiguration>
+        <LinkerSuppressSpecificWarnings>1132;1135;1140</LinkerSuppressSpecificWarnings>
+        <OutputName Condition="$(BuildForRelease)">$(OutputName)-$(PythonVersion)</OutputName>
+        <OutputName Condition="!$(BuildForRelease)">$(OutputName)-$(MajorVersionNumber).$(MinorVersionNumber).$(MicroVersionNumber).$(RevisionNumber)</OutputName>
+        <OutputName Condition="$(Platform) == 'x64'">$(OutputName)-amd64</OutputName>
+        <OutputName Condition="'$(OutputSuffix)' != ''">$(OutputName)-$(OutputSuffix)</OutputName>
+
+        <OutputPath>$(OutputPath)en-us\</OutputPath>
+        <OutDir>$(OutputPath)</OutDir>
+        
+        <DownloadUrl Condition="'$(DownloadUrl)' == '' and '$(DownloadUrlBase)' != ''">$(DownloadUrlBase.TrimEnd(`/`))/$(PythonVersion)/$(ArchName)/{2}</DownloadUrl>
+        <DefineConstants Condition="'$(DownloadUrl)' != ''">$(DefineConstants);DownloadUrl=$(DownloadUrl)</DefineConstants>
+        <DefineConstants Condition="'$(DownloadUrl)' == ''">$(DefineConstants);DownloadUrl={2}</DefineConstants>
+    </PropertyGroup>
+    
+    <ItemGroup>
+        <WixExtension Include="WixUtilExtension">
+            <HintPath>WixUtilExtension</HintPath>
+            <Name>WixUtilExtension</Name>
+        </WixExtension>
+        <WixExtension Include="WixDependencyExtension">
+            <HintPath>WixDependencyExtension</HintPath>
+            <Name>WixDependencyExtension</Name>
+        </WixExtension>
+        <WixExtension Include="WixBalExtension">
+            <HintPath>WixBalExtension</HintPath>
+            <Name>WixBalExtension</Name>
+        </WixExtension>
+    </ItemGroup>
+    <ItemGroup>
+        <Compile Include="bundle.wxs" />
+        <Compile Include="packagegroups\*.wxs" />
+    </ItemGroup>
+    <ItemGroup>
+        <Content Include="Default.thm" />
+        <Content Include="Default.wxl" />
+        <Content Include="SideBar.png" />
+    </ItemGroup>
+    <ItemGroup>
+        <WxlTemplate Include="*_en-US.wxl_template" />
+    </ItemGroup>
+    <ItemGroup>
+        <LinkerBindInputPaths Include="$(OutputPath)">
+            <BindName></BindName>
+        </LinkerBindInputPaths>
+    </ItemGroup>
+    
+    <ItemDefinitionGroup>
+        <Package>
+            <Properties>BuildForRelease=$(BuildForRelease)</Properties>
+        </Package>
+    </ItemDefinitionGroup>
+    <ItemGroup>
+        <Package Include="..\core\core*.wixproj" />
+        <Package Include="..\crt\crt*.wixproj" />
+        <Package Include="..\dev\dev*.wixproj" />
+        <Package Include="..\doc\doc*.wixproj" />
+        <Package Include="..\exe\exe*.wixproj" />
+        <Package Include="..\lib\lib*.wixproj" />
+        <Package Include="..\path\path*.wixproj" />
+        <Package Include="..\pip\pip*.wixproj" />
+        <Package Include="..\tcltk\tcltk*.wixproj" />
+        <Package Include="..\test\test*.wixproj" />
+        <Package Include="..\tools\tools*.wixproj" />
+    </ItemGroup>
+    
+    <Target Name="BuildPackages" BeforeTargets="BeforeBuild" Condition="'$(RebuildAll)' != 'false'">
+        <MSBuild Projects="@(Package)" BuildInParallel="true" />
+    </Target>
+    
+    <Target Name="BuildLauncher" BeforeTargets="BeforeBuild" Condition="'$(RebuildAll)' != 'false'">
+        <!-- Build the launcher MSI using Exec rather than MSBuild -->
+        <Exec Command='msbuild ..\launcher\launcher.wixproj /p:Platform=x86 /p:OutputPath="$(BuildPath.TrimEnd(`\`))" /p:OutputSuffix=$(Platform) /p:BuildForRelease=$(BuildForRelease)'
+              ContinueOnError="false" />
+    </Target>
+    
+    <Target Name="BuildBootstrapApplication" BeforeTargets="BeforeBuild">
+        <Message Text="Building bootstrap app" Importance="high" />
+        
+        <MSBuild Projects="bootstrap\pythonba.vcxproj"
+                 Targets="Build;GetNativeTargetPath"
+                 UseResultsCache="true"
+                 Properties="Configuration=$(BootstrapConfiguration);Platform=Win32">
+            <Output TaskParameter="TargetOutputs" PropertyName="BootstrapAppPath" />
+        </MSBuild>
+        
+        <PropertyGroup>
+            <DefineConstants>$(DefineConstants);BootstrapApp=$(BootstrapAppPath)</DefineConstants>
+        </PropertyGroup>
+    </Target>
+    
+    <Import Project="..\msi.targets" />
+</Project>
\ No newline at end of file
diff --git a/Tools/msi/bundle/bundle.wxs b/Tools/msi/bundle/bundle.wxs
new file mode 100644
--- /dev/null
+++ b/Tools/msi/bundle/bundle.wxs
@@ -0,0 +1,80 @@
+?<?xml version="1.0"?>
+<Wix xmlns="http://schemas.microsoft.com/wix/2006/wi"
+     xmlns:bal="http://schemas.microsoft.com/wix/BalExtension">
+  <Bundle Name="!(loc.FullProductName)"
+          UpgradeCode="$(var.UpgradeCode)"
+          Version="$(var.Version)"
+          IconSourceFile="bundle.ico"
+          Manufacturer="!(loc.Manufacturer)"
+          UpdateUrl="http://www.python.org/"
+          DisableModify="button"
+          Compressed="no">
+    <BootstrapperApplication Id="PythonBA" SourceFile="$(var.BootstrapApp)">
+        <Payload Compressed='yes' SourceFile='Default.thm' />
+        <Payload Compressed='yes' SourceFile='Default.wxl' />
+        <Payload Compressed='yes' SourceFile='SideBar.png' />
+    </BootstrapperApplication>
+
+    <!-- May be set to "Removing" or "Repairing" -->
+    <Variable Name="ActionLikeInstalling" Value="Installing" />
+    <!-- May be set to "Uninstallation" or "Repair" -->
+    <Variable Name="ActionLikeInstallation" Value="Setup" />
+
+    <Variable Name="ShortVersion" Value="$(var.MajorVersionNumber).$(var.MinorVersionNumber)" />
+    <Variable Name="ShortVersionNoDot" Value="$(var.MajorVersionNumber)$(var.MinorVersionNumber)" />
+
+    <Variable Name="InstallAllUsers" Value="1" Persisted="yes" bal:Overridable="yes" />
+    <Variable Name="TargetDir" Value="" Persisted="yes" bal:Overridable="yes" />
+    <?if $(var.Platform)~="x64" ?>
+    <Variable Name="DefaultAllUsersTargetDir" Value="[ProgramFiles64Folder]Python [ShortVersion]" bal:Overridable="yes" />
+    <?else ?>
+    <Variable Name="DefaultAllUsersTargetDir" Value="[ProgramFilesFolder]Python [ShortVersion]" bal:Overridable="yes" />
+    <?endif ?>
+    <Variable Name="DefaultJustForMeTargetDir" Value="[LocalAppDataFolder]Programs\Python\Python[ShortVersionNoDot]$(var.Suffix32)" bal:Overridable="yes" />
+    
+    <!--
+    An empty string will use the other defaults based on InstallAllUsers
+    (and switch dynamically in the UI). To get the old default, pass
+    this property on the command line:
+        DefaultCustomTargetDir=[WindowsVolume]Python[ShortVersionNoDot]
+    -->
+    <Variable Name="DefaultCustomTargetDir" Value="" bal:Overridable="yes" />
+
+    <Variable Name="InstallAllUsersState" Value="enabled" />
+    <Variable Name="TargetDirState" Value="enabled" />
+    <Variable Name="CustomBrowseButtonState" Value="enabled" />
+
+    <Variable Name="Include_core" Value="1" Persisted="yes" />
+    <Variable Name="Include_exe" Value="1" Persisted="yes" bal:Overridable="yes" />
+    <Variable Name="Include_dev" Value="1" Persisted="yes" bal:Overridable="yes" />
+    <Variable Name="Include_lib" Value="1" Persisted="yes" bal:Overridable="yes" />
+    <Variable Name="Include_test" Value="1" Persisted="yes" bal:Overridable="yes" />
+    <Variable Name="Include_doc" Value="1" Persisted="yes" bal:Overridable="yes" />
+    <Variable Name="Include_tools" Value="1" Persisted="yes" bal:Overridable="yes" />
+    <Variable Name="Include_tcltk" Value="1" Persisted="yes" bal:Overridable="yes" />
+    <Variable Name="Include_pip" Value="1" Persisted="yes" bal:Overridable="yes" />
+    <Variable Name="Include_launcher" Value="1" Persisted="yes" bal:Overridable="yes" />
+    <Variable Name="Include_symbols" Value="0" Persisted="yes" bal:Overridable="yes" />
+    <Variable Name="Include_debug" Value="0" Persisted="yes" bal:Overridable="yes" />
+    
+    <Variable Name="AssociateFiles" Value="1" Persisted="yes" bal:Overridable="yes" />
+    <Variable Name="PrependPath" Value="0" Persisted="yes" bal:Overridable="yes" />
+    <Variable Name="CompileAll" Value="0" Persisted="yes" bal:Overridable="yes" />
+    
+    <Variable Name="SimpleInstall" Value="0" bal:Overridable="yes" />
+    
+    <Chain ParallelCache="yes">
+      <PackageGroupRef Id="crt" />
+      <PackageGroupRef Id="core" />
+      <PackageGroupRef Id="dev" />
+      <PackageGroupRef Id="exe" />
+      <PackageGroupRef Id="lib" />
+      <PackageGroupRef Id="test" />
+      <PackageGroupRef Id="doc" />
+      <PackageGroupRef Id="tools" />
+      <PackageGroupRef Id="tcltk" />
+      <PackageGroupRef Id="launcher" />
+      <PackageGroupRef Id="postinstall" />
+    </Chain>
+  </Bundle>
+</Wix>
diff --git a/Tools/msi/bundle/full.wixproj b/Tools/msi/bundle/full.wixproj
new file mode 100644
--- /dev/null
+++ b/Tools/msi/bundle/full.wixproj
@@ -0,0 +1,21 @@
+<?xml version="1.0" encoding="utf-8"?>
+<Project ToolsVersion="4.0" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+    <PropertyGroup>
+        <ProjectGuid>{3E204ADD-238D-4D10-852C-4F859325C839}</ProjectGuid>
+        <OutputName>python</OutputName>
+        <OutputSuffix>full</OutputSuffix>
+    </PropertyGroup>
+
+    <Import Project="..\msi.props" />
+
+    <PropertyGroup>
+        <DefineConstants>
+            $(DefineConstants);
+            CompressMSI=yes;
+            CompressPDB=yes;
+            CompressMSI_D=yes;
+        </DefineConstants>
+    </PropertyGroup>
+    
+    <Import Project="bundle.targets" />
+</Project>
\ No newline at end of file
diff --git a/Tools/msi/bundle/packagegroups/core.wxs b/Tools/msi/bundle/packagegroups/core.wxs
new file mode 100644
--- /dev/null
+++ b/Tools/msi/bundle/packagegroups/core.wxs
@@ -0,0 +1,56 @@
+<?xml version="1.0"?>
+<Wix xmlns="http://schemas.microsoft.com/wix/2006/wi">
+    <Fragment>
+        <PackageGroup Id="core">
+            <MsiPackage Id="core_AllUsers"
+                        SourceFile="core.msi"
+                        Compressed="$(var.CompressMSI)"
+                        DownloadUrl="$(var.DownloadUrl)"
+                        ForcePerMachine="yes"
+                        InstallCondition="InstallAllUsers and (Include_core or Include_exe or Include_launcher or Include_pip)">
+                <MsiProperty Name="TARGETDIR" Value="[TargetDir]" />
+            </MsiPackage>
+            <MsiPackage Id="core_AllUsers_pdb"
+                        SourceFile="core_pdb.msi"
+                        Compressed="$(var.CompressPDB)"
+                        DownloadUrl="$(var.DownloadUrl)"
+                        ForcePerMachine="yes"
+                        InstallCondition="InstallAllUsers and (Include_core or Include_exe or Include_launcher or Include_pip) and Include_symbols">
+                <MsiProperty Name="TARGETDIR" Value="[TargetDir]" />
+            </MsiPackage>
+            <MsiPackage Id="core_AllUsers_d"
+                        SourceFile="core_d.msi"
+                        Compressed="$(var.CompressMSI_D)"
+                        DownloadUrl="$(var.DownloadUrl)"
+                        ForcePerMachine="yes"
+                        InstallCondition="InstallAllUsers and (Include_core or Include_exe or Include_launcher or Include_pip) and Include_debug">
+                <MsiProperty Name="TARGETDIR" Value="[TargetDir]" />
+            </MsiPackage>
+
+            <MsiPackage Id="core_JustForMe"
+                        SourceFile="core.msi"
+                        Compressed="$(var.CompressMSI)"
+                        DownloadUrl="$(var.DownloadUrl)"
+                        ForcePerMachine="no"
+                        InstallCondition="not InstallAllUsers and (Include_core or Include_exe or Include_launcher or Include_pip)">
+                <MsiProperty Name="TARGETDIR" Value="[TargetDir]" />
+            </MsiPackage>
+            <MsiPackage Id="core_JustForMe_pdb"
+                        SourceFile="core_pdb.msi"
+                        Compressed="$(var.CompressPDB)"
+                        DownloadUrl="$(var.DownloadUrl)"
+                        ForcePerMachine="no"
+                        InstallCondition="not InstallAllUsers and (Include_core or Include_exe or Include_launcher or Include_pip) and Include_symbols">
+                <MsiProperty Name="TARGETDIR" Value="[TargetDir]" />
+            </MsiPackage>
+            <MsiPackage Id="core_JustForMe_d"
+                        SourceFile="core_d.msi"
+                        Compressed="$(var.CompressMSI_D)"
+                        DownloadUrl="$(var.DownloadUrl)"
+                        ForcePerMachine="no"
+                        InstallCondition="not InstallAllUsers and (Include_core or Include_exe or Include_launcher or Include_pip) and Include_debug">
+                <MsiProperty Name="TARGETDIR" Value="[TargetDir]" />
+            </MsiPackage>
+        </PackageGroup>
+    </Fragment>
+</Wix>
\ No newline at end of file
diff --git a/Tools/msi/bundle/packagegroups/crt.wxs b/Tools/msi/bundle/packagegroups/crt.wxs
new file mode 100644
--- /dev/null
+++ b/Tools/msi/bundle/packagegroups/crt.wxs
@@ -0,0 +1,25 @@
+<?xml version="1.0"?>
+<Wix xmlns="http://schemas.microsoft.com/wix/2006/wi">
+    <Fragment>
+        <PackageGroup Id="crt">
+            <MsiPackage Id="crt_AllUsers"
+                        SourceFile="crt.msi"
+                        Compressed="$(var.CompressMSI)"
+                        DownloadUrl="$(var.DownloadUrl)"
+                        ForcePerMachine="yes"
+                        InstallCondition="InstallAllUsers and (Include_core or Include_exe or Include_launcher or Include_pip)">
+                <!-- TARGETDIR will be ignored, but must still be provided -->
+                <MsiProperty Name="TARGETDIR" Value="[TargetDir]" />
+            </MsiPackage>
+            
+            <MsiPackage Id="crt_JustForMe"
+                        SourceFile="crt.msi"
+                        Compressed="$(var.CompressMSI)"
+                        DownloadUrl="$(var.DownloadUrl)"
+                        ForcePerMachine="no"
+                        InstallCondition="not InstallAllUsers and (Include_core or Include_exe or Include_launcher or Include_pip)">
+                <MsiProperty Name="TARGETDIR" Value="[TargetDir]" />
+            </MsiPackage>
+        </PackageGroup>
+    </Fragment>
+</Wix>
\ No newline at end of file
diff --git a/Tools/msi/bundle/packagegroups/dev.wxs b/Tools/msi/bundle/packagegroups/dev.wxs
new file mode 100644
--- /dev/null
+++ b/Tools/msi/bundle/packagegroups/dev.wxs
@@ -0,0 +1,40 @@
+<?xml version="1.0"?>
+<Wix xmlns="http://schemas.microsoft.com/wix/2006/wi">
+    <Fragment>
+        <PackageGroup Id="dev">
+            <MsiPackage Id="dev_AllUsers"
+                        SourceFile="dev.msi"
+                        Compressed="$(var.CompressMSI)"
+                        DownloadUrl="$(var.DownloadUrl)"
+                        ForcePerMachine="yes"
+                        InstallCondition="InstallAllUsers and Include_dev">
+                <MsiProperty Name="TARGETDIR" Value="[TargetDir]" />
+            </MsiPackage>
+            <MsiPackage Id="dev_AllUsers_d"
+                        SourceFile="dev_d.msi"
+                        Compressed="$(var.CompressMSI_D)"
+                        DownloadUrl="$(var.DownloadUrl)"
+                        ForcePerMachine="yes"
+                        InstallCondition="InstallAllUsers and Include_dev and Include_debug">
+                <MsiProperty Name="TARGETDIR" Value="[TargetDir]" />
+            </MsiPackage>
+
+            <MsiPackage Id="dev_JustForMe"
+                        SourceFile="dev.msi"
+                        Compressed="$(var.CompressMSI)"
+                        DownloadUrl="$(var.DownloadUrl)"
+                        ForcePerMachine="no"
+                        InstallCondition="not InstallAllUsers and Include_dev">
+                <MsiProperty Name="TARGETDIR" Value="[TargetDir]" />
+            </MsiPackage>
+            <MsiPackage Id="dev_JustForMe_d"
+                        SourceFile="dev_d.msi"
+                        Compressed="$(var.CompressMSI_D)"
+                        DownloadUrl="$(var.DownloadUrl)"
+                        ForcePerMachine="no"
+                        InstallCondition="not InstallAllUsers and Include_dev and Include_debug">
+                <MsiProperty Name="TARGETDIR" Value="[TargetDir]" />
+            </MsiPackage>
+        </PackageGroup>
+    </Fragment>
+</Wix>
\ No newline at end of file
diff --git a/Tools/msi/bundle/packagegroups/doc.wxs b/Tools/msi/bundle/packagegroups/doc.wxs
new file mode 100644
--- /dev/null
+++ b/Tools/msi/bundle/packagegroups/doc.wxs
@@ -0,0 +1,24 @@
+<?xml version="1.0"?>
+<Wix xmlns="http://schemas.microsoft.com/wix/2006/wi">
+    <Fragment>
+        <PackageGroup Id="doc">
+            <MsiPackage Id="doc_AllUsers"
+                        SourceFile="doc.msi"
+                        Compressed="$(var.CompressMSI)"
+                        DownloadUrl="$(var.DownloadUrl)"
+                        ForcePerMachine="yes"
+                        InstallCondition="InstallAllUsers and Include_doc">
+                <MsiProperty Name="TARGETDIR" Value="[TargetDir]" />
+            </MsiPackage>
+            
+            <MsiPackage Id="doc_JustForMe"
+                        SourceFile="doc.msi"
+                        Compressed="$(var.CompressMSI)"
+                        DownloadUrl="$(var.DownloadUrl)"
+                        ForcePerMachine="no"
+                        InstallCondition="not InstallAllUsers and Include_doc">
+                <MsiProperty Name="TARGETDIR" Value="[TargetDir]" />
+            </MsiPackage>
+        </PackageGroup>
+    </Fragment>
+</Wix>
\ No newline at end of file
diff --git a/Tools/msi/bundle/packagegroups/exe.wxs b/Tools/msi/bundle/packagegroups/exe.wxs
new file mode 100644
--- /dev/null
+++ b/Tools/msi/bundle/packagegroups/exe.wxs
@@ -0,0 +1,56 @@
+<?xml version="1.0"?>
+<Wix xmlns="http://schemas.microsoft.com/wix/2006/wi">
+    <Fragment>
+        <PackageGroup Id="exe">
+            <MsiPackage Id="exe_AllUsers"
+                        SourceFile="exe.msi"
+                        ForcePerMachine="yes"
+                        Compressed="$(var.CompressMSI)"
+                        DownloadUrl="$(var.DownloadUrl)"
+                        InstallCondition="InstallAllUsers and (Include_exe or Include_launcher or Include_pip)">
+                <MsiProperty Name="TARGETDIR" Value="[TargetDir]" />
+            </MsiPackage>
+            <MsiPackage Id="exe_AllUsers_pdb"
+                        SourceFile="exe_pdb.msi"
+                        ForcePerMachine="yes"
+                        Compressed="$(var.CompressPDB)"
+                        DownloadUrl="$(var.DownloadUrl)"
+                        InstallCondition="InstallAllUsers and (Include_exe or Include_launcher or Include_pip) and Include_symbols">
+                <MsiProperty Name="TARGETDIR" Value="[TargetDir]" />
+            </MsiPackage>
+            <MsiPackage Id="exe_AllUsers_d"
+                        SourceFile="exe_d.msi"
+                        ForcePerMachine="yes"
+                        Compressed="$(var.CompressMSI_D)"
+                        DownloadUrl="$(var.DownloadUrl)"
+                        InstallCondition="InstallAllUsers and (Include_exe or Include_launcher or Include_pip) and Include_debug">
+                <MsiProperty Name="TARGETDIR" Value="[TargetDir]" />
+            </MsiPackage>
+            
+            <MsiPackage Id="exe_JustForMe"
+                        SourceFile="exe.msi"
+                        ForcePerMachine="no"
+                        Compressed="$(var.CompressMSI)"
+                        DownloadUrl="$(var.DownloadUrl)"
+                        InstallCondition="not InstallAllUsers and (Include_exe or Include_launcher or Include_pip)">
+                <MsiProperty Name="TARGETDIR" Value="[TargetDir]" />
+            </MsiPackage>
+            <MsiPackage Id="exe_JustForMe_pdb"
+                        SourceFile="exe_pdb.msi"
+                        ForcePerMachine="no"
+                        Compressed="$(var.CompressPDB)"
+                        DownloadUrl="$(var.DownloadUrl)"
+                        InstallCondition="not InstallAllUsers and (Include_exe or Include_launcher or Include_pip) and Include_symbols">
+                <MsiProperty Name="TARGETDIR" Value="[TargetDir]" />
+            </MsiPackage>
+            <MsiPackage Id="exe_JustForMe_d"
+                        SourceFile="exe_d.msi"
+                        ForcePerMachine="no"
+                        Compressed="$(var.CompressMSI_D)"
+                        DownloadUrl="$(var.DownloadUrl)"
+                        InstallCondition="not InstallAllUsers and (Include_exe or Include_launcher or Include_pip) and Include_debug">
+                <MsiProperty Name="TARGETDIR" Value="[TargetDir]" />
+            </MsiPackage>
+        </PackageGroup>
+    </Fragment>
+</Wix>
\ No newline at end of file
diff --git a/Tools/msi/bundle/packagegroups/launcher.wxs b/Tools/msi/bundle/packagegroups/launcher.wxs
new file mode 100644
--- /dev/null
+++ b/Tools/msi/bundle/packagegroups/launcher.wxs
@@ -0,0 +1,23 @@
+<?xml version="1.0"?>
+<Wix xmlns="http://schemas.microsoft.com/wix/2006/wi">
+    <Fragment>
+        <PackageGroup Id="launcher">
+            <!-- The All Users launcher is always the 32-bit version -->
+            <MsiPackage Id="launcher_AllUsers"
+                        SourceFile="launcher.msi"
+                        Compressed="$(var.CompressMSI)"
+                        DownloadUrl="$(var.DownloadUrl)"
+                        ForcePerMachine="yes"
+                        EnableFeatureSelection="yes"
+                        InstallCondition="InstallAllUsers and Include_launcher" />
+
+            <MsiPackage Id="launcher_JustForMe"
+                        SourceFile="launcher.msi"
+                        Compressed="$(var.CompressMSI)"
+                        DownloadUrl="$(var.DownloadUrl)"
+                        ForcePerMachine="no"
+                        EnableFeatureSelection="yes"
+                        InstallCondition="not InstallAllUsers and Include_launcher" />
+        </PackageGroup>
+    </Fragment>
+</Wix>
\ No newline at end of file
diff --git a/Tools/msi/bundle/packagegroups/lib.wxs b/Tools/msi/bundle/packagegroups/lib.wxs
new file mode 100644
--- /dev/null
+++ b/Tools/msi/bundle/packagegroups/lib.wxs
@@ -0,0 +1,56 @@
+<?xml version="1.0"?>
+<Wix xmlns="http://schemas.microsoft.com/wix/2006/wi">
+    <Fragment>
+        <PackageGroup Id="lib">
+            <MsiPackage Id="lib_AllUsers"
+                        SourceFile="lib.msi"
+                        Compressed="$(var.CompressMSI)"
+                        DownloadUrl="$(var.DownloadUrl)"
+                        ForcePerMachine="yes"
+                        InstallCondition="InstallAllUsers and Include_lib">
+                <MsiProperty Name="TARGETDIR" Value="[TargetDir]" />
+            </MsiPackage>
+            <MsiPackage Id="lib_AllUsers_pdb"
+                        SourceFile="lib_pdb.msi"
+                        Compressed="$(var.CompressPDB)"
+                        DownloadUrl="$(var.DownloadUrl)"
+                        ForcePerMachine="yes"
+                        InstallCondition="InstallAllUsers and Include_lib and Include_symbols">
+                <MsiProperty Name="TARGETDIR" Value="[TargetDir]" />
+            </MsiPackage>
+            <MsiPackage Id="lib_AllUsers_d"
+                        SourceFile="lib_d.msi"
+                        Compressed="$(var.CompressMSI_D)"
+                        DownloadUrl="$(var.DownloadUrl)"
+                        ForcePerMachine="yes"
+                        InstallCondition="InstallAllUsers and Include_lib and Include_debug">
+                <MsiProperty Name="TARGETDIR" Value="[TargetDir]" />
+            </MsiPackage>
+
+            <MsiPackage Id="lib_JustForMe"
+                        SourceFile="lib.msi"
+                        Compressed="$(var.CompressMSI)"
+                        DownloadUrl="$(var.DownloadUrl)"
+                        ForcePerMachine="no"
+                        InstallCondition="not InstallAllUsers and Include_lib">
+                <MsiProperty Name="TARGETDIR" Value="[TargetDir]" />
+            </MsiPackage>
+            <MsiPackage Id="lib_JustForMe_pdb"
+                        SourceFile="lib_pdb.msi"
+                        Compressed="$(var.CompressPDB)"
+                        DownloadUrl="$(var.DownloadUrl)"
+                        ForcePerMachine="no"
+                        InstallCondition="not InstallAllUsers and Include_lib and Include_symbols">
+                <MsiProperty Name="TARGETDIR" Value="[TargetDir]" />
+            </MsiPackage>
+            <MsiPackage Id="lib_JustForMe_d"
+                        SourceFile="lib_d.msi"
+                        Compressed="$(var.CompressMSI_D)"
+                        DownloadUrl="$(var.DownloadUrl)"
+                        ForcePerMachine="no"
+                        InstallCondition="not InstallAllUsers and Include_lib and Include_debug">
+                <MsiProperty Name="TARGETDIR" Value="[TargetDir]" />
+            </MsiPackage>
+        </PackageGroup>
+    </Fragment>
+</Wix>
\ No newline at end of file
diff --git a/Tools/msi/bundle/packagegroups/postinstall.wxs b/Tools/msi/bundle/packagegroups/postinstall.wxs
new file mode 100644
--- /dev/null
+++ b/Tools/msi/bundle/packagegroups/postinstall.wxs
@@ -0,0 +1,62 @@
+<?xml version="1.0"?>
+<Wix xmlns="http://schemas.microsoft.com/wix/2006/wi">
+    <Fragment>
+        <PackageGroup Id="postinstall">
+            <MsiPackage Id="pip_AllUsers"
+                        SourceFile="pip.msi"
+                        Compressed="$(var.CompressMSI)"
+                        DownloadUrl="$(var.DownloadUrl)"
+                        ForcePerMachine="yes"
+                        InstallCondition="InstallAllUsers and Include_pip">
+                <MsiProperty Name="TARGETDIR" Value="[TargetDir]" />
+            </MsiPackage>
+            <MsiPackage Id="pip_JustForMe"
+                        SourceFile="pip.msi"
+                        Compressed="$(var.CompressMSI)"
+                        DownloadUrl="$(var.DownloadUrl)"
+                        ForcePerMachine="no"
+                        InstallCondition="not InstallAllUsers and Include_pip">
+                <MsiProperty Name="TARGETDIR" Value="[TargetDir]" />
+            </MsiPackage>
+            
+            <MsiPackage Id="path_AllUsers"
+                        SourceFile="path.msi"
+                        Compressed="$(var.CompressMSI)"
+                        DownloadUrl="$(var.DownloadUrl)"
+                        ForcePerMachine="yes"
+                        InstallCondition="InstallAllUsers and PrependPath">
+                <MsiProperty Name="TARGETDIR" Value="[TargetDir]" />
+            </MsiPackage>
+            <MsiPackage Id="path_JustForMe"
+                        SourceFile="path.msi"
+                        Compressed="$(var.CompressMSI)"
+                        DownloadUrl="$(var.DownloadUrl)"
+                        ForcePerMachine="no"
+                        InstallCondition="not InstallAllUsers and PrependPath">
+                <MsiProperty Name="TARGETDIR" Value="[TargetDir]" />
+            </MsiPackage>
+            
+            <?define CompileAllCommand=-$(var.ShortVersion)$(var.Suffix32) -Wi "[TargetDir]\Lib\compileall.py" -f -x "bad_coding|badsyntax|site-packages|py2_|lib2to3\\tests|venv\\scripts" "[TargetDir]\Lib"?>
+            <ExePackage Id="compileall_AllUsers"
+                        SourceFile="py.exe"
+                        Compressed="yes"
+                        DisplayName="!(loc.CompileAllDescription)"
+                        InstallCommand='$(var.CompileAllCommand)'
+                        RepairCommand='$(var.CompileAllCommand)'
+                        Permanent="yes"
+                        PerMachine="yes"
+                        Vital="no"
+                        InstallCondition="InstallAllUsers and CompileAll" />
+            <ExePackage Id="compileall_JustForMe"
+                        SourceFile="py.exe"
+                        Compressed="yes"
+                        DisplayName="!(loc.CompileAllDescription)"
+                        InstallCommand='$(var.CompileAllCommand)'
+                        RepairCommand='$(var.CompileAllCommand)'
+                        Permanent="yes"
+                        PerMachine="no"
+                        Vital="no"
+                        InstallCondition="not InstallAllUsers and CompileAll" />
+        </PackageGroup>
+    </Fragment>
+</Wix>
\ No newline at end of file
diff --git a/Tools/msi/bundle/packagegroups/tcltk.wxs b/Tools/msi/bundle/packagegroups/tcltk.wxs
new file mode 100644
--- /dev/null
+++ b/Tools/msi/bundle/packagegroups/tcltk.wxs
@@ -0,0 +1,62 @@
+<?xml version="1.0"?>
+<Wix xmlns="http://schemas.microsoft.com/wix/2006/wi">
+    <Fragment>
+        <PackageGroup Id="tcltk">
+            <MsiPackage Id="tcltk_AllUsers"
+                        SourceFile="tcltk.msi"
+                        Compressed="$(var.CompressMSI)"
+                        DownloadUrl="$(var.DownloadUrl)"
+                        ForcePerMachine="yes"
+                        EnableFeatureSelection="yes"
+                        InstallCondition="InstallAllUsers and Include_tcltk">
+                <MsiProperty Name="TARGETDIR" Value="[TargetDir]" />
+            </MsiPackage>
+            <MsiPackage Id="tcltk_AllUsers_pdb"
+                        SourceFile="tcltk_pdb.msi"
+                        Compressed="$(var.CompressPDB)"
+                        DownloadUrl="$(var.DownloadUrl)"
+                        ForcePerMachine="yes"
+                        EnableFeatureSelection="yes"
+                        InstallCondition="InstallAllUsers and Include_tcltk and Include_symbols">
+                <MsiProperty Name="TARGETDIR" Value="[TargetDir]" />
+            </MsiPackage>
+            <MsiPackage Id="tcltk_AllUsers_d"
+                        SourceFile="tcltk_d.msi"
+                        Compressed="$(var.CompressMSI_D)"
+                        DownloadUrl="$(var.DownloadUrl)"
+                        ForcePerMachine="yes"
+                        EnableFeatureSelection="yes"
+                        InstallCondition="InstallAllUsers and Include_tcltk and Include_debug">
+                <MsiProperty Name="TARGETDIR" Value="[TargetDir]" />
+            </MsiPackage>
+
+            <MsiPackage Id="tcltk_JustForMe"
+                        SourceFile="tcltk.msi"
+                        Compressed="$(var.CompressMSI)"
+                        DownloadUrl="$(var.DownloadUrl)"
+                        ForcePerMachine="no"
+                        EnableFeatureSelection="yes"
+                        InstallCondition="not InstallAllUsers and Include_tcltk">
+                <MsiProperty Name="TARGETDIR" Value="[TargetDir]" />
+            </MsiPackage>
+            <MsiPackage Id="tcltk_JustForMe_pdb"
+                        SourceFile="tcltk_pdb.msi"
+                        Compressed="$(var.CompressPDB)"
+                        DownloadUrl="$(var.DownloadUrl)"
+                        ForcePerMachine="no"
+                        EnableFeatureSelection="yes"
+                        InstallCondition="not InstallAllUsers and Include_tcltk and Include_symbols">
+                <MsiProperty Name="TARGETDIR" Value="[TargetDir]" />
+            </MsiPackage>
+            <MsiPackage Id="tcltk_JustForMe_d"
+                        SourceFile="tcltk_d.msi"
+                        Compressed="$(var.CompressMSI_D)"
+                        DownloadUrl="$(var.DownloadUrl)"
+                        ForcePerMachine="no"
+                        EnableFeatureSelection="yes"
+                        InstallCondition="not InstallAllUsers and Include_tcltk and Include_debug">
+                <MsiProperty Name="TARGETDIR" Value="[TargetDir]" />
+            </MsiPackage>
+        </PackageGroup>
+    </Fragment>
+</Wix>
\ No newline at end of file
diff --git a/Tools/msi/bundle/packagegroups/test.wxs b/Tools/msi/bundle/packagegroups/test.wxs
new file mode 100644
--- /dev/null
+++ b/Tools/msi/bundle/packagegroups/test.wxs
@@ -0,0 +1,56 @@
+<?xml version="1.0"?>
+<Wix xmlns="http://schemas.microsoft.com/wix/2006/wi">
+    <Fragment>
+        <PackageGroup Id="test">
+            <MsiPackage Id="test_AllUsers"
+                        SourceFile="test.msi"
+                        Compressed="$(var.CompressMSI)"
+                        DownloadUrl="$(var.DownloadUrl)"
+                        ForcePerMachine="yes"
+                        InstallCondition="InstallAllUsers and Include_test">
+                <MsiProperty Name="TARGETDIR" Value="[TargetDir]" />
+            </MsiPackage>
+            <MsiPackage Id="test_AllUsers_pdb"
+                        SourceFile="test_pdb.msi"
+                        Compressed="$(var.CompressPDB)"
+                        DownloadUrl="$(var.DownloadUrl)"
+                        ForcePerMachine="yes"
+                        InstallCondition="InstallAllUsers and Include_test and Include_symbols">
+                <MsiProperty Name="TARGETDIR" Value="[TargetDir]" />
+            </MsiPackage>
+            <MsiPackage Id="test_AllUsers_d"
+                        SourceFile="test_d.msi"
+                        Compressed="$(var.CompressMSI_D)"
+                        DownloadUrl="$(var.DownloadUrl)"
+                        ForcePerMachine="yes"
+                        InstallCondition="InstallAllUsers and Include_test and Include_debug">
+                <MsiProperty Name="TARGETDIR" Value="[TargetDir]" />
+            </MsiPackage>
+
+            <MsiPackage Id="test_JustForMe"
+                        SourceFile="test.msi"
+                        Compressed="$(var.CompressMSI)"
+                        DownloadUrl="$(var.DownloadUrl)"
+                        ForcePerMachine="no"
+                        InstallCondition="not InstallAllUsers and Include_test">
+                <MsiProperty Name="TARGETDIR" Value="[TargetDir]" />
+            </MsiPackage>
+            <MsiPackage Id="test_JustForMe_pdb"
+                        SourceFile="test_pdb.msi"
+                        Compressed="$(var.CompressPDB)"
+                        DownloadUrl="$(var.DownloadUrl)"
+                        ForcePerMachine="no"
+                        InstallCondition="not InstallAllUsers and Include_test and Include_symbols">
+                <MsiProperty Name="TARGETDIR" Value="[TargetDir]" />
+            </MsiPackage>
+            <MsiPackage Id="test_JustForMe_d"
+                        SourceFile="test_d.msi"
+                        Compressed="$(var.CompressMSI_D)"
+                        DownloadUrl="$(var.DownloadUrl)"
+                        ForcePerMachine="no"
+                        InstallCondition="not InstallAllUsers and Include_test and Include_debug">
+                <MsiProperty Name="TARGETDIR" Value="[TargetDir]" />
+            </MsiPackage>
+        </PackageGroup>
+    </Fragment>
+</Wix>
\ No newline at end of file
diff --git a/Tools/msi/bundle/packagegroups/tools.wxs b/Tools/msi/bundle/packagegroups/tools.wxs
new file mode 100644
--- /dev/null
+++ b/Tools/msi/bundle/packagegroups/tools.wxs
@@ -0,0 +1,24 @@
+<?xml version="1.0"?>
+<Wix xmlns="http://schemas.microsoft.com/wix/2006/wi">
+    <Fragment>
+        <PackageGroup Id="tools">
+            <MsiPackage Id="tools_AllUsers"
+                        SourceFile="tools.msi"
+                        Compressed="$(var.CompressMSI)"
+                        DownloadUrl="$(var.DownloadUrl)"
+                        ForcePerMachine="yes"
+                        InstallCondition="InstallAllUsers and Include_tools">
+                <MsiProperty Name="TARGETDIR" Value="[TargetDir]" />
+            </MsiPackage>
+            
+            <MsiPackage Id="tools_JustForMe"
+                        SourceFile="tools.msi"
+                        Compressed="$(var.CompressMSI)"
+                        DownloadUrl="$(var.DownloadUrl)"
+                        ForcePerMachine="no"
+                        InstallCondition="not InstallAllUsers and Include_tools">
+                <MsiProperty Name="TARGETDIR" Value="[TargetDir]" />
+            </MsiPackage>
+        </PackageGroup>
+    </Fragment>
+</Wix>
\ No newline at end of file
diff --git a/Tools/msi/bundle/postinstall_en-US.wxl_template b/Tools/msi/bundle/postinstall_en-US.wxl_template
new file mode 100644
--- /dev/null
+++ b/Tools/msi/bundle/postinstall_en-US.wxl_template
@@ -0,0 +1,4 @@
+<?xml version="1.0" encoding="utf-8"?>
+<WixLocalization Culture="en-us" xmlns="http://schemas.microsoft.com/wix/2006/localization">
+    <String Id="CompileAllDescription">Precompiling standard library</String>
+</WixLocalization>
diff --git a/Tools/msi/bundle/releaselocal.wixproj b/Tools/msi/bundle/releaselocal.wixproj
new file mode 100644
--- /dev/null
+++ b/Tools/msi/bundle/releaselocal.wixproj
@@ -0,0 +1,21 @@
+<?xml version="1.0" encoding="utf-8"?>
+<Project ToolsVersion="4.0" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+    <PropertyGroup>
+        <ProjectGuid>{FCD43AC9-969F-49A1-8AC5-EDC27599D1EB}</ProjectGuid>
+        <OutputName>python</OutputName>
+        <OutputSuffix></OutputSuffix>
+    </PropertyGroup>
+
+    <Import Project="..\msi.props" />
+
+    <PropertyGroup>
+        <DefineConstants>
+            $(DefineConstants);
+            CompressMSI=yes;
+            CompressPDB=no;
+            CompressMSI_D=no
+        </DefineConstants>
+    </PropertyGroup>
+    
+    <Import Project="bundle.targets" />
+</Project>
\ No newline at end of file
diff --git a/Tools/msi/bundle/releaseweb.wixproj b/Tools/msi/bundle/releaseweb.wixproj
new file mode 100644
--- /dev/null
+++ b/Tools/msi/bundle/releaseweb.wixproj
@@ -0,0 +1,21 @@
+<?xml version="1.0" encoding="utf-8"?>
+<Project ToolsVersion="4.0" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+    <PropertyGroup>
+        <ProjectGuid>{71CDE213-CB39-4BD9-B89D-BBB878689144}</ProjectGuid>
+        <OutputName>python</OutputName>
+        <OutputSuffix>webinstall</OutputSuffix>
+    </PropertyGroup>
+
+    <Import Project="..\msi.props" />
+
+    <PropertyGroup>
+        <DefineConstants>
+            $(DefineConstants);
+            CompressMSI=no;
+            CompressPDB=no;
+            CompressMSI_D=no
+        </DefineConstants>
+    </PropertyGroup>
+    
+    <Import Project="bundle.targets" />
+</Project>
\ No newline at end of file
diff --git a/Tools/msi/bundle/snapshot.wixproj b/Tools/msi/bundle/snapshot.wixproj
new file mode 100644
--- /dev/null
+++ b/Tools/msi/bundle/snapshot.wixproj
@@ -0,0 +1,21 @@
+<?xml version="1.0" encoding="utf-8"?>
+<Project ToolsVersion="4.0" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+    <PropertyGroup>
+        <ProjectGuid>{8A4A1162-4BF9-4FF6-9A98-315F01E44932}</ProjectGuid>
+        <OutputName>python</OutputName>
+        <OutputSuffix></OutputSuffix>
+    </PropertyGroup>
+
+    <Import Project="..\msi.props" />
+
+    <PropertyGroup>
+        <DefineConstants>
+            $(DefineConstants);
+            CompressMSI=no;
+            CompressPDB=no;
+            CompressMSI_D=no;
+        </DefineConstants>
+    </PropertyGroup>
+    
+    <Import Project="bundle.targets" />
+</Project>
\ No newline at end of file
diff --git a/Tools/msi/common.wxs b/Tools/msi/common.wxs
new file mode 100644
--- /dev/null
+++ b/Tools/msi/common.wxs
@@ -0,0 +1,101 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<Wix xmlns="http://schemas.microsoft.com/wix/2006/wi">
+    <Fragment>
+        <Property Id="REGISTRYKEY" Value="Software\$(var.TestPrefix)Python\PythonCore\$(var.ShortVersion)$(var.Suffix32)" />
+    </Fragment>
+    
+    <Fragment>
+        <Property Id="UpgradeTable" Value="1" />
+        
+        <Upgrade Id="$(var.UpgradeCode)">
+            <UpgradeVersion Property="DOWNGRADE" Minimum="$(var.Version)" IncludeMinimum="no" OnlyDetect="yes" />
+            <UpgradeVersion Property="UPGRADE" Minimum="$(var.UpgradeMinimumVersion)" IncludeMinimum="yes" Maximum="$(var.Version)" IncludeMaximum="no" />
+        </Upgrade>
+        
+        <?if $(var.UpgradeCode)!=$(var.CoreUpgradeCode) ?>
+        <Upgrade Id="$(var.CoreUpgradeCode)">
+            <UpgradeVersion Property="MISSING_CORE" Minimum="$(var.Version)" Maximum="$(var.Version)" OnlyDetect="yes" />
+        </Upgrade>
+        <?endif ?>
+        
+        <Condition Message="!(loc.NoDowngrade)">Installed OR NOT DOWNGRADE</Condition>
+        <Condition Message="!(loc.IncorrectCore)">Installed OR NOT MISSING_CORE</Condition>
+        <Condition Message="!(loc.NoTargetDir)">Installed OR TARGETDIR OR Suppress_TARGETDIR_Check</Condition>
+
+        <InstallExecuteSequence>
+            <RemoveExistingProducts After="InstallInitialize">UPGRADE</RemoveExistingProducts>
+        </InstallExecuteSequence>
+    </Fragment>
+    
+    <Fragment>
+        <!-- Include an icon for the Programs and Features dialog -->
+        <Icon Id="ARPIcon" SourceFile="!(bindpath.src)PC\pycon.ico" />
+        <Property Id="ARPPRODUCTICON" Value="ARPIcon" />
+        <Property Id="ARPNOMODIFY" Value="1" />
+        <Property Id="DISABLEADVTSHORTCUTS" Value="1" />
+    </Fragment>
+    
+    <Fragment>
+        <Directory Id="TARGETDIR" Name="SourceDir">
+            <Directory Id="InstallDirectory" ComponentGuidGenerationSeed="$(var.InstallDirectoryGuidSeed)" />
+        </Directory>
+    </Fragment>
+    
+    <!-- Top-level directories -->
+    <Fragment>
+        <DirectoryRef Id="InstallDirectory">
+            <Directory Id="DLLs" Name="DLLs" />
+        </DirectoryRef>
+    </Fragment>
+
+    <Fragment>
+        <DirectoryRef Id="InstallDirectory">
+            <Directory Id="Doc" Name="Doc" />
+        </DirectoryRef>
+    </Fragment>
+
+    <Fragment>
+        <DirectoryRef Id="InstallDirectory">
+            <Directory Id="include" Name="include" />
+        </DirectoryRef>
+    </Fragment>
+
+    <Fragment>
+        <DirectoryRef Id="InstallDirectory">
+            <Directory Id="Lib" Name="Lib" />
+        </DirectoryRef>
+    </Fragment>
+
+    <Fragment>
+        <DirectoryRef Id="InstallDirectory">
+            <Directory Id="libs" Name="libs" />
+        </DirectoryRef>
+    </Fragment>
+
+    <Fragment>
+        <DirectoryRef Id="InstallDirectory">
+            <Directory Id="Scripts" Name="Scripts" />
+        </DirectoryRef>
+    </Fragment>
+
+    <Fragment>
+        <DirectoryRef Id="InstallDirectory">
+            <Directory Id="tcl" Name="tcl" />
+        </DirectoryRef>
+    </Fragment>
+
+    <Fragment>
+        <DirectoryRef Id="InstallDirectory">
+            <Directory Id="Tools" Name="Tools" />
+        </DirectoryRef>
+    </Fragment>
+    
+    <!-- Start Menu folder -->
+    <Fragment>
+        <DirectoryRef Id="TARGETDIR">
+            <Directory Id="ProgramMenuFolder">
+                <Directory Id="MenuDir" Name="!(loc.ProductName)" />
+            </Directory>
+        </DirectoryRef>
+    </Fragment>
+</Wix>
diff --git a/Tools/msi/common_en-US.wxl_template b/Tools/msi/common_en-US.wxl_template
new file mode 100644
--- /dev/null
+++ b/Tools/msi/common_en-US.wxl_template
@@ -0,0 +1,17 @@
+<?xml version="1.0" encoding="utf-8"?>
+<WixLocalization Culture="en-us" xmlns="http://schemas.microsoft.com/wix/2006/localization">
+    <String Id="LCID">1033</String>
+    <String Id="Culture">en-us</String>
+    <String Id="ProductName">Python {{ShortVersion}}</String>
+    <String Id="FullProductName">Python {{LongVersion}} ({{Bitness}})</String>
+    <String Id="Title">Python {{LongVersion}} !(loc.Descriptor) ({{Bitness}})</String>
+    <String Id="Description">Python {{LongVersion}} !(loc.Descriptor) ({{Bitness}})</String>
+    <String Id="TitlePdb">Python {{LongVersion}} !(loc.Descriptor) ({{Bitness}} symbols)</String>
+    <String Id="DescriptionPdb">Python {{LongVersion}} !(loc.Descriptor) ({{Bitness}} symbols)</String>
+    <String Id="Title_d">Python {{LongVersion}} !(loc.Descriptor) ({{Bitness}} debug)</String>
+    <String Id="Description_d">Python {{LongVersion}} !(loc.Descriptor) ({{Bitness}} debug)</String>
+    <String Id="Manufacturer">Python Software Foundation</String>
+    <String Id="NoDowngrade">A newer version of !(loc.ProductName) is already installed.</String>
+    <String Id="IncorrectCore">An incorrect version of a prerequisite package is installed. Please uninstall any other versions of !(loc.ProductName) and try installing this again.</String>
+    <String Id="NoTargetDir">The TARGETDIR variable must be provided when invoking this installer.</String>
+</WixLocalization>
diff --git a/Tools/msi/core/core.props b/Tools/msi/core/core.props
new file mode 100644
--- /dev/null
+++ b/Tools/msi/core/core.props
@@ -0,0 +1,12 @@
+<?xml version="1.0" encoding="utf-8"?>
+<Project ToolsVersion="4.0" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+    <Import Project="..\msi.props" />
+    <ItemGroup>
+        <Compile Include="*.wxs" />
+    </ItemGroup>
+    <ItemGroup>
+        <EmbeddedResource Include="*.wxl" />
+    </ItemGroup>
+
+    <Import Project="..\msi.targets" />
+</Project>
\ No newline at end of file
diff --git a/Tools/msi/core/core.wixproj b/Tools/msi/core/core.wixproj
new file mode 100644
--- /dev/null
+++ b/Tools/msi/core/core.wixproj
@@ -0,0 +1,11 @@
+<?xml version="1.0" encoding="utf-8"?>
+<Project ToolsVersion="4.0" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+    <PropertyGroup>
+        <ProjectGuid>{1B4502D5-B627-4F50-ABEA-4CC5A8E88265}</ProjectGuid>
+        <SchemaVersion>2.0</SchemaVersion>
+        <OutputName>core</OutputName>
+        <OutputType>Package</OutputType>
+        <DefineConstants>IncludeDefaultFeature=1;$(DefineConstants)</DefineConstants>
+    </PropertyGroup>
+    <Import Project="core.props" />
+</Project>
\ No newline at end of file
diff --git a/Tools/msi/core/core.wxs b/Tools/msi/core/core.wxs
new file mode 100644
--- /dev/null
+++ b/Tools/msi/core/core.wxs
@@ -0,0 +1,25 @@
+?<?xml version="1.0" encoding="UTF-8"?>
+<Wix xmlns="http://schemas.microsoft.com/wix/2006/wi">
+    <Product Id="*" Language="!(loc.LCID)" Name="!(loc.Title)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)">
+        <Package InstallerVersion="300" Compressed="yes" InstallScope="perUser" Platform="$(var.Platform)" />
+        <MediaTemplate EmbedCab="yes" CompressionLevel="high" />
+        
+        <PropertyRef Id="UpgradeTable" />
+        
+        <?ifdef IncludeDefaultFeature ?>
+        <Feature Id="DefaultFeature" AllowAdvertise="no" Title="!(loc.Title)" Description="!(loc.Description)">
+            <ComponentGroupRef Id="core_dll" />
+        </Feature>
+        <?endif ?>
+        <?ifdef IncludeSymbols ?>
+        <Feature Id="Symbols" AllowAdvertise="no" Title="!(loc.TitlePdb)" Description="!(loc.DescriptionPdb)">
+            <ComponentGroupRef Id="core_symbols" />
+        </Feature>
+        <?endif ?>
+        <?ifdef IncludeDebugBinaries ?>
+        <Feature Id="DebugBinaries" AllowAdvertise="no" Title="!(loc.Title_d)" Description="!(loc.Description_d)">
+            <ComponentGroupRef Id="core_dll_d" />
+        </Feature>
+        <?endif ?>
+    </Product>
+</Wix>
diff --git a/Tools/msi/core/core_d.wixproj b/Tools/msi/core/core_d.wixproj
new file mode 100644
--- /dev/null
+++ b/Tools/msi/core/core_d.wixproj
@@ -0,0 +1,11 @@
+<?xml version="1.0" encoding="utf-8"?>
+<Project ToolsVersion="4.0" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+    <PropertyGroup>
+        <ProjectGuid>{D3677DCF-098A-4398-9FA5-8E74AC37E0DF}</ProjectGuid>
+        <SchemaVersion>2.0</SchemaVersion>
+        <OutputName>core_d</OutputName>
+        <OutputType>Package</OutputType>
+        <DefineConstants>IncludeDebugBinaries=1;$(DefineConstants)</DefineConstants>
+    </PropertyGroup>
+    <Import Project="core.props" />
+</Project>
\ No newline at end of file
diff --git a/Tools/msi/core/core_en-US.wxl b/Tools/msi/core/core_en-US.wxl
new file mode 100644
--- /dev/null
+++ b/Tools/msi/core/core_en-US.wxl
@@ -0,0 +1,5 @@
+?<?xml version="1.0" encoding="utf-8"?>
+<WixLocalization Culture="en-us" xmlns="http://schemas.microsoft.com/wix/2006/localization">
+    <String Id="Descriptor">Core Interpreter</String>
+    <String Id="ShortDescriptor">core</String>
+</WixLocalization>
diff --git a/Tools/msi/core/core_files.wxs b/Tools/msi/core/core_files.wxs
new file mode 100644
--- /dev/null
+++ b/Tools/msi/core/core_files.wxs
@@ -0,0 +1,31 @@
+?<?xml version="1.0" encoding="UTF-8"?>
+<Wix xmlns="http://schemas.microsoft.com/wix/2006/wi">
+    <Fragment>
+        <ComponentGroup Id="core_dll">
+            <Component Id="python_stable.dll" Directory="InstallDirectory" Guid="*">
+                <File Id="python_stable.dll" Name="python$(var.MajorVersionNumber).dll" KeyPath="yes" />
+            </Component>
+            <Component Id="python.dll" Directory="InstallDirectory" Guid="*">
+                <File Id="python.dll" Name="python$(var.MajorVersionNumber)$(var.MinorVersionNumber).dll" KeyPath="yes" />
+            </Component>
+        </ComponentGroup>
+    </Fragment>
+    <Fragment>
+        <ComponentGroup Id="core_symbols">
+            <Component Id="python.pdb" Directory="InstallDirectory" Guid="*">
+                <File Id="python.pdb" Name="python$(var.MajorVersionNumber)$(var.MinorVersionNumber).pdb" KeyPath="yes" />
+            </Component>
+        </ComponentGroup>
+    </Fragment>
+    <Fragment>
+        <ComponentGroup Id="core_dll_d">
+            <Component Id="python_stable_d.dll" Directory="InstallDirectory" Guid="*">
+                <File Id="python_stable_d.dll" Name="python$(var.MajorVersionNumber)_d.dll" KeyPath="yes" />
+            </Component>
+            <Component Id="python_d.dll" Directory="InstallDirectory" Guid="*">
+                <File Id="python_d.dll" Name="python$(var.MajorVersionNumber)$(var.MinorVersionNumber)_d.dll" KeyPath="yes" />
+                <File Id="python_d.pdb" Name="python$(var.MajorVersionNumber)$(var.MinorVersionNumber)_d.pdb" KeyPath="no" />
+            </Component>
+        </ComponentGroup>
+    </Fragment>
+</Wix>
diff --git a/Tools/msi/core/core_pdb.wixproj b/Tools/msi/core/core_pdb.wixproj
new file mode 100644
--- /dev/null
+++ b/Tools/msi/core/core_pdb.wixproj
@@ -0,0 +1,11 @@
+<?xml version="1.0" encoding="utf-8"?>
+<Project ToolsVersion="4.0" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+    <PropertyGroup>
+        <ProjectGuid>{E98E7539-64E7-4DCE-AACD-01E3ADE40EFD}</ProjectGuid>
+        <SchemaVersion>2.0</SchemaVersion>
+        <OutputName>core_pdb</OutputName>
+        <OutputType>Package</OutputType>
+        <DefineConstants>IncludeSymbols=1;$(DefineConstants)</DefineConstants>
+    </PropertyGroup>
+    <Import Project="core.props" />
+</Project>
\ No newline at end of file
diff --git a/Tools/msi/crt/crt.wixproj b/Tools/msi/crt/crt.wixproj
new file mode 100644
--- /dev/null
+++ b/Tools/msi/crt/crt.wixproj
@@ -0,0 +1,20 @@
+<?xml version="1.0" encoding="utf-8"?>
+<Project ToolsVersion="4.0" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+    <PropertyGroup>
+        <ProjectGuid>{91C99298-8E2E-4422-A5AF-CC4FFF9A58D3}</ProjectGuid>
+        <SchemaVersion>2.0</SchemaVersion>
+        <OutputName>crt</OutputName>
+        <OutputType>Package</OutputType>
+        <SuppressIces>ICE71</SuppressIces>
+    </PropertyGroup>
+    <Import Project="..\msi.props" />
+    <ItemGroup>
+        <Compile Include="crt.wxs" />
+        <Compile Include="crt_files.$(VisualStudioVersion).wxs" />
+    </ItemGroup>
+    <ItemGroup>
+        <EmbeddedResource Include="*.wxl" />
+    </ItemGroup>
+    
+    <Import Project="..\msi.targets" />
+</Project>
\ No newline at end of file
diff --git a/Tools/msi/crt/crt.wxs b/Tools/msi/crt/crt.wxs
new file mode 100644
--- /dev/null
+++ b/Tools/msi/crt/crt.wxs
@@ -0,0 +1,26 @@
+?<?xml version="1.0" encoding="UTF-8"?>
+<Wix xmlns="http://schemas.microsoft.com/wix/2006/wi">
+    <Product Id="*" Language="!(loc.LCID)" Name="!(loc.Title)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)">
+        <Package InstallerVersion="300" Compressed="yes" InstallScope="perUser" Platform="$(var.Platform)" />
+        <MediaTemplate EmbedCab="yes" CompressionLevel="high" />
+        
+        <PropertyRef Id="UpgradeTable" />
+        
+        <?if $(var.Platform)~=x64 ?>
+        <?define DirName=System64?>
+        <?else ?>
+        <?define DirName=System?>
+        <?endif ?>
+        <DirectoryRef Id="TARGETDIR">
+            <Directory Id="$(var.DirName)Folder" Name="$(var.DirName)">
+                <Directory Id="SystemInstallDirectory" Name="." />
+            </Directory>
+        </DirectoryRef>
+        <?undef DirName ?>
+        
+        <Feature Id="DefaultFeature" AllowAdvertise="no" Title="!(loc.Title)" Description="!(loc.Description)">
+            <ComponentGroupRef Id="crt_files" />
+        </Feature>
+    </Product>
+</Wix>
+
diff --git a/Tools/msi/crt/crt_en-US.wxl b/Tools/msi/crt/crt_en-US.wxl
new file mode 100644
--- /dev/null
+++ b/Tools/msi/crt/crt_en-US.wxl
@@ -0,0 +1,5 @@
+?<?xml version="1.0" encoding="utf-8"?>
+<WixLocalization Culture="en-us" xmlns="http://schemas.microsoft.com/wix/2006/localization">
+    <String Id="Descriptor">C Runtime</String>
+    <String Id="ShortDescriptor">crt</String>
+</WixLocalization>
diff --git a/Tools/msi/crt/crt_files.12.0.wxs b/Tools/msi/crt/crt_files.12.0.wxs
new file mode 100644
--- /dev/null
+++ b/Tools/msi/crt/crt_files.12.0.wxs
@@ -0,0 +1,20 @@
+?<?xml version="1.0" encoding="UTF-8"?>
+<Wix xmlns="http://schemas.microsoft.com/wix/2006/wi">
+    <Fragment>
+        <?if $(var.Platform)~=x64 ?>
+        <?define msvcr120Guid={0835C947-D6D2-4E52-AF14-0231D04E88EA}?>
+        <?else ?>
+        <?define msvcr120Guid={E5B92048-5859-4AF1-AEAD-B97EBF00B087} ?>
+        <?endif ?>
+        <ComponentGroup Id="crt_files">
+            <Component Id="msvcr120.dll_LM" Directory="SystemInstallDirectory" Guid="$(var.msvcr120Guid)" Shared="yes" SharedDllRefCount="yes">
+                <Condition>ALLUSERS=1</Condition>
+                <File Id="msvcr120.dll_LM" Source="!(bindpath.crt)\msvcr120.dll" />
+            </Component>
+            <Component Id="msvcr120.dll_CU" Directory="InstallDirectory" Guid="*">
+                <Condition>NOT ALLUSERS=1</Condition>
+                <File Id="msvcr120.dll_CU" Source="!(bindpath.crt)\msvcr120.dll" />
+            </Component>
+        </ComponentGroup>
+    </Fragment>
+</Wix>
diff --git a/Tools/msi/crt/crt_files.14.0.wxs b/Tools/msi/crt/crt_files.14.0.wxs
new file mode 100644
--- /dev/null
+++ b/Tools/msi/crt/crt_files.14.0.wxs
@@ -0,0 +1,40 @@
+?<?xml version="1.0" encoding="UTF-8"?>
+<Wix xmlns="http://schemas.microsoft.com/wix/2006/wi">
+    <Fragment>
+        <?if $(var.Platform)~=x64 ?>
+        <?define appcrt140Guid={CC160FA9-B519-38EC-B358-B4146E8506C8}?>
+        <?define desktopcrt140Guid={4DB78A79-8D7F-35DD-B0E8-736DE44D25F3}?>
+        <?define vcruntime140Guid={B33258FD-750C-3B42-8BE4-535B48E97DB4}?>
+        <?else ?>
+        <?define appcrt140Guid={E3854F9C-4CFB-3B85-90BD-86AA22D82DC8}?>
+        <?define desktopcrt140Guid={46EAB1CD-C362-3139-BD7E-D8782E65253A}?>
+        <?define vcruntime140Guid={E8E39D3B-4F35-36D8-B892-4B28336FE041}?>
+        <?endif ?>
+        <ComponentGroup Id="crt_files">
+            <Component Id="appcrt140.dll_LM" Directory="SystemInstallDirectory" Guid="$(var.appcrt140Guid)" Shared="yes" SharedDllRefCount="yes">
+                <Condition>ALLUSERS=1</Condition>
+                <File Id="appcrt140.dll_LM" Source="!(bindpath.crt)\appcrt140.dll" />
+            </Component>
+            <Component Id="desktopcrt140.dll_LM" Directory="SystemInstallDirectory" Guid="$(var.desktopcrt140Guid)" Shared="yes" SharedDllRefCount="yes">
+                <Condition>ALLUSERS=1</Condition>
+                <File Id="desktopcrt140.dll_LM" Source="!(bindpath.crt)\desktopcrt140.dll" />
+            </Component>
+            <Component Id="vcruntime140.dll_LM" Directory="SystemInstallDirectory" Guid="$(var.vcruntime140Guid)" Shared="yes" SharedDllRefCount="yes">
+                <Condition>ALLUSERS=1</Condition>
+                <File Id="vcruntime140.dll_LM" Source="!(bindpath.crt)\vcruntime140.dll" />
+            </Component>
+            <Component Id="appcrt140.dll_CU" Directory="InstallDirectory" Guid="*">
+                <Condition>NOT ALLUSERS=1</Condition>
+                <File Id="appcrt140.dll_CU" Source="!(bindpath.crt)\appcrt140.dll" />
+            </Component>
+            <Component Id="desktopcrt140.dll_CU" Directory="InstallDirectory" Guid="*">
+                <Condition>NOT ALLUSERS=1</Condition>
+                <File Id="desktopcrt140.dll_CU" Source="!(bindpath.crt)\desktopcrt140.dll" />
+            </Component>
+            <Component Id="vcruntime140.dll_CU" Directory="InstallDirectory" Guid="*">
+                <Condition>NOT ALLUSERS=1</Condition>
+                <File Id="vcruntime140.dll_CU" Source="!(bindpath.crt)\vcruntime140.dll" />
+            </Component>
+        </ComponentGroup>
+    </Fragment>
+</Wix>
diff --git a/Tools/msi/crt/crtlicense.txt b/Tools/msi/crt/crtlicense.txt
new file mode 100644
--- /dev/null
+++ b/Tools/msi/crt/crtlicense.txt
@@ -0,0 +1,47 @@
+
+
+Additional Conditions for this Windows binary build
+---------------------------------------------------
+
+This program is linked with and uses Microsoft Distributable Code,
+copyrighted by Microsoft Corporation. The Microsoft Distributable Code
+includes the following files:
+
+appcrt140.dll
+desktopcrt140.dll
+vcruntime140.dll
+msvcp140.dll
+concrt140.dll
+vccorlib140.dll
+
+If you further distribute programs that include the Microsoft
+Distributable Code, you must comply with the restrictions on
+distribution specified by Microsoft. In particular, you must require
+distributors and external end users to agree to terms that protect the
+Microsoft Distributable Code at least as much as Microsoft's own
+requirements for the Distributable Code. See Microsoft's documentation
+(included in its developer tools and on its website at microsoft.com)
+for specific details.
+
+Redistribution of the Windows binary build of the Python interpreter
+complies with this agreement, provided that you do not:
+
+- alter any copyright, trademark or patent notice in Microsoft's
+Distributable Code;
+
+- use Microsoft's trademarks in your programs' names or in a way that
+suggests your programs come from or are endorsed by Microsoft;
+
+- distribute Microsoft's Distributable Code to run on a platform other
+than Microsoft operating systems, run-time technologies or application
+platforms; or
+
+- include Microsoft Distributable Code in malicious, deceptive or
+unlawful programs.
+
+These restrictions apply only to the Microsoft Distributable Code as
+defined above, not to Python itself or any programs running on the
+Python interpreter. The redistribution of the Python interpreter and
+libraries is governed by the Python Software License included with this
+file, or by other licenses as marked.
+
diff --git a/Tools/msi/crtlicense.txt b/Tools/msi/crtlicense.txt
deleted file mode 100644
--- a/Tools/msi/crtlicense.txt
+++ /dev/null
@@ -1,44 +0,0 @@
-
-
-Additional Conditions for this Windows binary build
----------------------------------------------------
-
-This program is linked with and uses Microsoft Distributable Code,
-copyrighted by Microsoft Corporation. The Microsoft Distributable Code
-includes the following files:
-
-msvcr90.dll
-msvcp90.dll
-msvcm90.dll
-
-If you further distribute programs that include the Microsoft
-Distributable Code, you must comply with the restrictions on
-distribution specified by Microsoft. In particular, you must require
-distributors and external end users to agree to terms that protect the
-Microsoft Distributable Code at least as much as Microsoft's own
-requirements for the Distributable Code. See Microsoft's documentation
-(included in its developer tools and on its website at microsoft.com)
-for specific details.
-
-Redistribution of the Windows binary build of the Python interpreter
-complies with this agreement, provided that you do not:
-
-- alter any copyright, trademark or patent notice in Microsoft's
-Distributable Code;
-
-- use Microsoft's trademarks in your programs' names or in a way that
-suggests your programs come from or are endorsed by Microsoft;
-
-- distribute Microsoft's Distributable Code to run on a platform other
-than Microsoft operating systems, run-time technologies or application
-platforms; or
-
-- include Microsoft Distributable Code in malicious, deceptive or
-unlawful programs.
-
-These restrictions apply only to the Microsoft Distributable Code as
-defined above, not to Python itself or any programs running on the
-Python interpreter. The redistribution of the Python interpreter and
-libraries is governed by the Python Software License included with this
-file, or by other licenses as marked.
-
diff --git a/Tools/msi/csv_to_wxs.py b/Tools/msi/csv_to_wxs.py
new file mode 100644
--- /dev/null
+++ b/Tools/msi/csv_to_wxs.py
@@ -0,0 +1,127 @@
+'''
+Processes a CSV file containing a list of files into a WXS file with
+components for each listed file.
+
+The CSV columns are:
+    source of file, target for file, group name
+
+Usage::
+    py txt_to_wxs.py [path to file list .csv] [path to destination .wxs]
+
+This is necessary to handle structures where some directories only
+contain other directories. MSBuild is not able to generate the
+Directory entries in the WXS file correctly, as it operates on files.
+Python, however, can easily fill in the gap.
+'''
+
+__author__ = "Steve Dower <steve.dower at microsoft.com>"
+
+import csv
+import re
+import sys
+
+from collections import defaultdict
+from itertools import chain, zip_longest
+from pathlib import PureWindowsPath
+from uuid import uuid1
+
+ID_CHAR_SUBS = {
+    '-': '_',
+    '+': '_P',
+}
+
+def make_id(path):
+    return re.sub(
+        r'[^A-Za-z0-9_.]',
+        lambda m: ID_CHAR_SUBS.get(m.group(0), '_'),
+        str(path).rstrip('/\\'),
+        flags=re.I
+    )
+
+DIRECTORIES = set()
+
+def main(file_source, install_target):
+    with open(file_source, 'r', newline='') as f:
+        files = list(csv.reader(f))
+
+    assert len(files) == len(set(make_id(f[1]) for f in files)), "Duplicate file IDs exist"
+
+    directories = defaultdict(set)
+    cache_directories = defaultdict(set)
+    groups = defaultdict(list)
+    for source, target, group, disk_id, condition in files:
+        target = PureWindowsPath(target)
+        groups[group].append((source, target, disk_id, condition))
+
+        if target.suffix.lower() in {".py", ".pyw"}:
+            cache_directories[group].add(target.parent)
+
+        for dirname in target.parents:
+            parent = make_id(dirname.parent)
+            if parent and parent != '.':
+                directories[parent].add(dirname.name)
+
+    lines = [
+        '<Wix xmlns="http://schemas.microsoft.com/wix/2006/wi">',
+        '    <Fragment>',
+    ]
+    for dir_parent in sorted(directories):
+        lines.append('        <DirectoryRef Id="{}">'.format(dir_parent))
+        for dir_name in sorted(directories[dir_parent]):
+            lines.append('            <Directory Id="{}_{}" Name="{}" />'.format(dir_parent, make_id(dir_name), dir_name))
+        lines.append('        </DirectoryRef>')
+    for dir_parent in (make_id(d) for group in cache_directories.values() for d in group):
+        lines.append('        <DirectoryRef Id="{}">'.format(dir_parent))
+        lines.append('            <Directory Id="{}___pycache__" Name="__pycache__" />'.format(dir_parent))
+        lines.append('        </DirectoryRef>')
+    lines.append('    </Fragment>')
+
+    for group in sorted(groups):
+        lines.extend([
+            '    <Fragment>',
+            '        <ComponentGroup Id="{}">'.format(group),
+        ])
+        for source, target, disk_id, condition in groups[group]:
+            lines.append('            <Component Id="{}" Directory="{}" Guid="*">'.format(make_id(target), make_id(target.parent)))
+            if condition:
+                lines.append('                <Condition>{}</Condition>'.format(condition))
+
+            if disk_id:
+                lines.append('                <File Id="{}" Name="{}" Source="{}" DiskId="{}" />'.format(make_id(target), target.name, source, disk_id))
+            else:
+                lines.append('                <File Id="{}" Name="{}" Source="{}" />'.format(make_id(target), target.name, source))
+            lines.append('            </Component>')
+
+        create_folders = {make_id(p) + "___pycache__" for p in cache_directories[group]}
+        remove_folders = {make_id(p2) for p1 in cache_directories[group] for p2 in chain((p1,), p1.parents)}
+        create_folders.discard(".")
+        remove_folders.discard(".")
+        if create_folders or remove_folders:
+            lines.append('            <Component Id="{}__pycache__folders" Directory="TARGETDIR" Guid="{}">'.format(group, uuid1()))
+            lines.extend('                <CreateFolder Directory="{}" />'.format(p) for p in create_folders)
+            lines.extend('                <RemoveFile Id="Remove_{0}_files" Name="*" On="uninstall" Directory="{0}" />'.format(p) for p in create_folders)
+            lines.extend('                <RemoveFolder Id="Remove_{0}_folder" On="uninstall" Directory="{0}" />'.format(p) for p in create_folders | remove_folders)
+            lines.append('            </Component>')
+
+        lines.extend([
+            '        </ComponentGroup>',
+            '    </Fragment>',
+        ])
+    lines.append('</Wix>')
+
+    # Check if the file matches. If so, we don't want to touch it so
+    # that we can skip rebuilding.
+    try:
+        with open(install_target, 'r') as f:
+            if all(x.rstrip('\r\n') == y for x, y in zip_longest(f, lines)):
+                print('File is up to date')
+                return
+    except IOError:
+        pass
+
+    with open(install_target, 'w') as f:
+        f.writelines(line + '\n' for line in lines)
+    print('Wrote {} lines to {}'.format(len(lines), install_target))
+
+if __name__ == '__main__':
+    main(sys.argv[1], sys.argv[2])
diff --git a/Tools/msi/dev/dev.props b/Tools/msi/dev/dev.props
new file mode 100644
--- /dev/null
+++ b/Tools/msi/dev/dev.props
@@ -0,0 +1,42 @@
+<?xml version="1.0" encoding="utf-8"?>
+<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+    <Import Project="..\msi.props" />
+    <PropertyGroup>
+        <DefineConstants Condition="$(BuildForRelease)">
+            $(DefineConstants);
+            IncludeMinGWLib=1;
+        </DefineConstants>
+    </PropertyGroup>
+    <ItemGroup>
+        <Compile Include="*.wxs" />
+    </ItemGroup>
+    <ItemGroup>
+        <EmbeddedResource Include="*.wxl" />
+    </ItemGroup>
+    <ItemGroup>
+        <InstallFiles Include="$(PySourcePath)include\*.h">
+            <SourceBase>$(PySourcePath)</SourceBase>
+            <Source>!(bindpath.src)</Source>
+            <TargetBase>$(PySourcePath)</TargetBase>
+            <Target_></Target_>
+            <Group>dev_include</Group>
+        </InstallFiles>
+    </ItemGroup>
+    
+    <Target Name="BuildMinGWLib"
+            Inputs="$(BuildPath)$(PyDllName).dll"
+            Outputs="$(BuildPath)lib$(PyDllName).a"
+            AfterTargets="PrepareForBuild"
+            Condition="$(BuildForRelease)">
+        <!-- Build libpython##.a as part of this project. This requires gendef and dlltool on the path. -->
+        <PropertyGroup>
+            <_GenDefPlatform>i386</_GenDefPlatform>
+            <_GenDefPlatform Condition="$(Platform) == 'x64'">i386:x86-64</_GenDefPlatform>
+        </PropertyGroup>
+
+        <Exec Command='gendef - "$(BuildPath)$(PyDllName).dll" &gt; "$(IntermediateOutputPath)mingwlib.def"' ContinueOnError="false" />
+        <Exec Command='dlltool --dllname $(PyDllName).dll --def "$(IntermediateOutputPath)mingwlib.def" --output-lib "$(BuildPath)lib$(PyDllName).a" -m $(_GenDefPlatform)' />
+    </Target>
+
+    <Import Project="..\msi.targets" />
+</Project>
\ No newline at end of file
diff --git a/Tools/msi/dev/dev.wixproj b/Tools/msi/dev/dev.wixproj
new file mode 100644
--- /dev/null
+++ b/Tools/msi/dev/dev.wixproj
@@ -0,0 +1,11 @@
+<?xml version="1.0" encoding="utf-8"?>
+<Project ToolsVersion="4.0" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+    <PropertyGroup>
+        <ProjectGuid>{5F23F608-D74B-4259-A0CE-8DC65CC7FE53}</ProjectGuid>
+        <SchemaVersion>2.0</SchemaVersion>
+        <OutputName Condition="'$(OutputName)' == ''">dev</OutputName>
+        <OutputType>Package</OutputType>
+        <DefineConstants>IncludeDefaultFeature=1;$(DefineConstants)</DefineConstants>
+    </PropertyGroup>
+    <Import Project="dev.props" />
+</Project>
\ No newline at end of file
diff --git a/Tools/msi/dev/dev.wxs b/Tools/msi/dev/dev.wxs
new file mode 100644
--- /dev/null
+++ b/Tools/msi/dev/dev.wxs
@@ -0,0 +1,25 @@
+?<?xml version="1.0" encoding="UTF-8"?>
+<Wix xmlns="http://schemas.microsoft.com/wix/2006/wi">
+    <Product Id="*" Language="!(loc.LCID)" Name="!(loc.Title)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)">
+        <Package InstallerVersion="300" Compressed="yes" InstallScope="perUser" Platform="$(var.Platform)" />
+        <MediaTemplate EmbedCab="yes" CompressionLevel="high" />
+        
+        <PropertyRef Id="UpgradeTable" />
+        
+        <?ifdef IncludeDefaultFeature ?>
+        <Feature Id="DefaultFeature" AllowAdvertise="no" Title="!(loc.Title)" Description="!(loc.Description)">
+            <ComponentGroupRef Id="dev_include" />
+            <ComponentGroupRef Id="dev_pyconfig" />
+            <ComponentGroupRef Id="dev_libs" />
+<?ifdef IncludeMinGWLib ?>
+            <ComponentGroupRef Id="dev_mingw" />
+<?endif ?>
+        </Feature>
+        <?endif ?>
+        <?ifdef IncludeDebugBinaries ?>
+        <Feature Id="DebugBinaries" AllowAdvertise="no" Title="!(loc.Title_d)" Description="!(loc.Description_d)">
+            <ComponentGroupRef Id="dev_libs_d" />
+        </Feature>
+        <?endif ?>
+    </Product>
+</Wix>
diff --git a/Tools/msi/dev/dev_d.wixproj b/Tools/msi/dev/dev_d.wixproj
new file mode 100644
--- /dev/null
+++ b/Tools/msi/dev/dev_d.wixproj
@@ -0,0 +1,11 @@
+<?xml version="1.0" encoding="utf-8"?>
+<Project ToolsVersion="4.0" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+    <PropertyGroup>
+        <ProjectGuid>{C11B4945-76BD-4137-B2E3-649460117A77}</ProjectGuid>
+        <SchemaVersion>2.0</SchemaVersion>
+        <OutputName>dev_d</OutputName>
+        <OutputType>Package</OutputType>
+        <DefineConstants>IncludeDebugBinaries=1;$(DefineConstants)</DefineConstants>
+    </PropertyGroup>
+    <Import Project="dev.props" />
+</Project>
\ No newline at end of file
diff --git a/Tools/msi/dev/dev_en-US.wxl b/Tools/msi/dev/dev_en-US.wxl
new file mode 100644
--- /dev/null
+++ b/Tools/msi/dev/dev_en-US.wxl
@@ -0,0 +1,5 @@
+?<?xml version="1.0" encoding="utf-8"?>
+<WixLocalization Culture="en-us" xmlns="http://schemas.microsoft.com/wix/2006/localization">
+    <String Id="Descriptor">Development Libraries</String>
+    <String Id="ShortDescriptor">dev</String>
+</WixLocalization>
diff --git a/Tools/msi/dev/dev_files.wxs b/Tools/msi/dev/dev_files.wxs
new file mode 100644
--- /dev/null
+++ b/Tools/msi/dev/dev_files.wxs
@@ -0,0 +1,42 @@
+?<?xml version="1.0" encoding="UTF-8"?>
+<Wix xmlns="http://schemas.microsoft.com/wix/2006/wi">
+    <Fragment>
+        <ComponentGroup Id="dev_pyconfig">
+            <Component Id="include_pyconfig.h" Directory="include" Guid="*">
+                <File Id="include_pyconfig.h" Name="pyconfig.h" Source="!(bindpath.src)PC\pyconfig.h" KeyPath="yes" />
+            </Component>
+        </ComponentGroup>
+    </Fragment>
+    
+    <Fragment>
+        <ComponentGroup Id="dev_libs">
+            <Component Id="libs_python3.lib" Directory="libs" Guid="*">
+                <File Id="libs_python_stable.lib" Name="python$(var.MajorVersionNumber).lib" KeyPath="yes" />
+            </Component>
+            <Component Id="libs_python.lib" Directory="libs" Guid="*">
+                <File Id="libs_python.lib" Name="python$(var.MajorVersionNumber)$(var.MinorVersionNumber).lib" KeyPath="yes" />
+            </Component>
+        </ComponentGroup>
+    </Fragment>
+    
+    <Fragment>
+        <ComponentGroup Id="dev_libs_d">
+            <Component Id="libs_python3_d.lib" Directory="libs" Guid="*">
+                <File Id="libs_python_stable_d.lib" Name="python$(var.MajorVersionNumber)_d.lib" />
+            </Component>
+            <Component Id="libs_python_d.lib" Directory="libs" Guid="*">
+                <File Id="libs_python_d.lib" Name="python$(var.MajorVersionNumber)$(var.MinorVersionNumber)_d.lib" />
+            </Component>
+        </ComponentGroup>
+    </Fragment>
+    
+    <?ifdef IncludeMinGWLib ?>
+    <Fragment>
+        <ComponentGroup Id="dev_mingw">
+            <Component Id="libs_libpython.a" Directory="libs" Guid="*">
+                <File Id="libs_libpython.a" Name="libpython$(var.MajorVersionNumber)$(var.MinorVersionNumber).a" KeyPath="yes" />
+            </Component>
+        </ComponentGroup>
+    </Fragment>
+    <?endif ?>
+</Wix>
diff --git a/Tools/msi/doc/doc.wixproj b/Tools/msi/doc/doc.wixproj
new file mode 100644
--- /dev/null
+++ b/Tools/msi/doc/doc.wixproj
@@ -0,0 +1,30 @@
+<?xml version="1.0" encoding="utf-8"?>
+<Project ToolsVersion="4.0" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+    <PropertyGroup>
+        <ProjectGuid>{0D62A2BB-5F71-4447-8C8C-9708407B3674}</ProjectGuid>
+        <SchemaVersion>2.0</SchemaVersion>
+        <OutputName>doc</OutputName>
+        <OutputType>Package</OutputType>
+        <!-- Shortcut validation is not necessary -->
+        <SuppressICEs>ICE43</SuppressICEs>
+    </PropertyGroup>
+    <Import Project="..\msi.props" />
+    <PropertyGroup>
+        <DocFilename>python$(MajorVersionNumber)$(MinorVersionNumber)$(MicroVersionNumber)$(ReleaseLevelName).chm</DocFilename>
+        <IncludeDocFile>false</IncludeDocFile>
+        <IncludeDocFile Condition="$(BuildForRelease) or Exists('$(PySourcePath)Doc\build\htmlhelp\$(DocFilename)')">true</IncludeDocFile>
+    </PropertyGroup>
+    <PropertyGroup Condition="$(IncludeDocFile)">
+        <DefineConstants>$(DefineConstants);DocFilename=$(DocFilename);</DefineConstants>
+    </PropertyGroup>
+    <ItemGroup>
+        <Compile Include="doc.wxs" />
+        <Compile Include="doc_files.wxs" Condition="$(IncludeDocFile)" />
+        <Compile Include="doc_no_files.wxs" Condition="!$(IncludeDocFile)" />
+    </ItemGroup>
+    <ItemGroup>
+        <WxlTemplate Include="*.wxl_template" />
+    </ItemGroup>
+    
+    <Import Project="..\msi.targets" />
+</Project>
\ No newline at end of file
diff --git a/Tools/msi/doc/doc.wxs b/Tools/msi/doc/doc.wxs
new file mode 100644
--- /dev/null
+++ b/Tools/msi/doc/doc.wxs
@@ -0,0 +1,26 @@
+?<?xml version="1.0" encoding="UTF-8"?>
+<Wix xmlns="http://schemas.microsoft.com/wix/2006/wi">
+    <Product Id="*" Language="!(loc.LCID)" Name="!(loc.Title)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)">
+        <Package InstallerVersion="300" Compressed="yes" InstallScope="perUser" Platform="$(var.Platform)" />
+        <MediaTemplate EmbedCab="yes" CompressionLevel="high" />
+        
+        <PropertyRef Id="UpgradeTable" />
+        <PropertyRef Id="REGISTRYKEY" />
+        
+        <Feature Id="DefaultFeature" AllowAdvertise="no" Title="!(loc.Title)" Description="!(loc.Description)">
+            <ComponentGroupRef Id="doc" />
+            
+            <Component Id="doc_shortcut" Directory="MenuDir" Guid="*">
+                <RegistryKey Root="HKMU" Key="[REGISTRYKEY]">
+                    <RegistryValue Key="Help\Main Python Documentation" Type="string" Value="[#python.chm]" KeyPath="yes" />
+                </RegistryKey>
+                <Shortcut Id="python.chm"
+                          Target="[#python.chm]"
+                          Name="!(loc.ShortcutName)"
+                          Description="!(loc.ShortcutDescription)" />
+                <RemoveFolder Id="Remove_MenuDir" On="uninstall" />
+            </Component>
+
+        </Feature>
+    </Product>
+</Wix>
diff --git a/Tools/msi/doc/doc_en-US.wxl_template b/Tools/msi/doc/doc_en-US.wxl_template
new file mode 100644
--- /dev/null
+++ b/Tools/msi/doc/doc_en-US.wxl_template
@@ -0,0 +1,7 @@
+?<?xml version="1.0" encoding="utf-8"?>
+<WixLocalization Culture="en-us" xmlns="http://schemas.microsoft.com/wix/2006/localization">
+    <String Id="ShortDescriptor">doc</String>
+    <String Id="Descriptor">Documentation</String>
+    <String Id="ShortcutName">Python {{ShortVersion}} {{Bitness}} Manuals</String>
+    <String Id="ShortcutDescription">View the !(loc.ProductName) documentation.</String>
+</WixLocalization>
diff --git a/Tools/msi/doc/doc_files.wxs b/Tools/msi/doc/doc_files.wxs
new file mode 100644
--- /dev/null
+++ b/Tools/msi/doc/doc_files.wxs
@@ -0,0 +1,12 @@
+?<?xml version="1.0" encoding="UTF-8"?>
+<Wix xmlns="http://schemas.microsoft.com/wix/2006/wi">
+    <Fragment>
+        <PropertyRef Id="REGISTRYKEY" />
+        
+        <ComponentGroup Id="doc">
+            <Component Id="python.chm" Directory="Doc" Guid="*">
+                <File Id="python.chm" Name="$(var.DocFilename)" KeyPath="yes" />
+            </Component>
+        </ComponentGroup>
+    </Fragment>
+</Wix>
diff --git a/Tools/msi/doc/doc_no_files.wxs b/Tools/msi/doc/doc_no_files.wxs
new file mode 100644
--- /dev/null
+++ b/Tools/msi/doc/doc_no_files.wxs
@@ -0,0 +1,17 @@
+?<?xml version="1.0" encoding="UTF-8"?>
+<Wix xmlns="http://schemas.microsoft.com/wix/2006/wi">
+    <Fragment>
+        <ComponentGroup Id="doc">
+            <!--
+            This file is included when the CHM is not available.
+            
+            This way, snapshot builds can succeed without having to
+            build the docs.
+            -->
+            <Component Id="EmptyDocFolder" Directory="Doc" Guid="{22FD42DB-EC66-4B1C-B1FC-44E0CF7B2462}">
+                <CreateFolder />
+                <RemoveFolder Id="Remove_EmptyDocFolder" On="uninstall" />
+            </Component>
+        </ComponentGroup>
+    </Fragment>
+</Wix>
diff --git a/Tools/msi/exe/crtlicense.txt b/Tools/msi/exe/crtlicense.txt
new file mode 100644
--- /dev/null
+++ b/Tools/msi/exe/crtlicense.txt
@@ -0,0 +1,44 @@
+
+
+Additional Conditions for this Windows binary build
+---------------------------------------------------
+
+This program is linked with and uses Microsoft Distributable Code,
+copyrighted by Microsoft Corporation. The Microsoft Distributable Code
+includes the following files:
+
+msvcr90.dll
+msvcp90.dll
+msvcm90.dll
+
+If you further distribute programs that include the Microsoft
+Distributable Code, you must comply with the restrictions on
+distribution specified by Microsoft. In particular, you must require
+distributors and external end users to agree to terms that protect the
+Microsoft Distributable Code at least as much as Microsoft's own
+requirements for the Distributable Code. See Microsoft's documentation
+(included in its developer tools and on its website at microsoft.com)
+for specific details.
+
+Redistribution of the Windows binary build of the Python interpreter
+complies with this agreement, provided that you do not:
+
+- alter any copyright, trademark or patent notice in Microsoft's
+Distributable Code;
+
+- use Microsoft's trademarks in your programs' names or in a way that
+suggests your programs come from or are endorsed by Microsoft;
+
+- distribute Microsoft's Distributable Code to run on a platform other
+than Microsoft operating systems, run-time technologies or application
+platforms; or
+
+- include Microsoft Distributable Code in malicious, deceptive or
+unlawful programs.
+
+These restrictions apply only to the Microsoft Distributable Code as
+defined above, not to Python itself or any programs running on the
+Python interpreter. The redistribution of the Python interpreter and
+libraries is governed by the Python Software License included with this
+file, or by other licenses as marked.
+
diff --git a/Tools/msi/exe/exe.props b/Tools/msi/exe/exe.props
new file mode 100644
--- /dev/null
+++ b/Tools/msi/exe/exe.props
@@ -0,0 +1,36 @@
+<?xml version="1.0" encoding="utf-8"?>
+<Project ToolsVersion="4.0" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+    <PropertyGroup>
+        <!-- Shortcut validation is not necessary -->
+        <SuppressICEs>ICE43</SuppressICEs>
+    </PropertyGroup>
+    <Import Project="..\msi.props" />
+    <ItemGroup>
+        <Compile Include="*.wxs" />
+    </ItemGroup>
+    <ItemGroup>
+        <EmbeddedResource Include="*.wxl" />
+        <WxlTemplate Include="*.wxl_template" />
+    </ItemGroup>
+    
+    <Target Name="_GenerateLicense" AfterTargets="PrepareForBuild">
+        <ItemGroup>
+            <LicenseFiles Include="$(PySourcePath)LICENSE;
+                                   crtlicense.txt;
+                                   $(bz2Dir)LICENSE;
+                                   $(opensslDir)LICENSE;
+                                   $(tclDir)license.terms;
+                                   $(tkDir)license.terms;
+                                   $(tixDir)license.terms" />
+            <_LicenseFiles Include="@(LicenseFiles)">
+                <Content>$([System.IO.File]::ReadAllText(%(FullPath)))</Content>
+            </_LicenseFiles>
+        </ItemGroup>
+        
+        <WriteLinesToFile File="$(BuildPath)LICENSE"
+                          Overwrite="true"
+                          Lines="@(_LicenseFiles->'%(Content)')" />
+    </Target>
+        
+    <Import Project="..\msi.targets" />
+</Project>
\ No newline at end of file
diff --git a/Tools/msi/exe/exe.wixproj b/Tools/msi/exe/exe.wixproj
new file mode 100644
--- /dev/null
+++ b/Tools/msi/exe/exe.wixproj
@@ -0,0 +1,11 @@
+<?xml version="1.0" encoding="utf-8"?>
+<Project ToolsVersion="4.0" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+    <PropertyGroup>
+        <ProjectGuid>{6BD53305-B03E-49DC-85FB-5551B8CCC843}</ProjectGuid>
+        <SchemaVersion>2.0</SchemaVersion>
+        <OutputName>exe</OutputName>
+        <OutputType>Package</OutputType>
+        <DefineConstants>IncludeDefaultFeature=1;$(DefineConstants)</DefineConstants>
+    </PropertyGroup>
+    <Import Project="exe.props" />
+</Project>
\ No newline at end of file
diff --git a/Tools/msi/exe/exe.wxs b/Tools/msi/exe/exe.wxs
new file mode 100644
--- /dev/null
+++ b/Tools/msi/exe/exe.wxs
@@ -0,0 +1,40 @@
+?<?xml version="1.0" encoding="UTF-8"?>
+<Wix xmlns="http://schemas.microsoft.com/wix/2006/wi">
+    <Product Id="*" Language="!(loc.LCID)" Name="!(loc.Title)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)">
+        <Package InstallerVersion="300" Compressed="yes" InstallScope="perUser" Platform="$(var.Platform)" />
+        <MediaTemplate EmbedCab="yes" CompressionLevel="high" />
+        
+        <PropertyRef Id="UpgradeTable" />
+        <PropertyRef Id="REGISTRYKEY" />
+        
+        <?ifdef IncludeDefaultFeature ?>
+        <Feature Id="DefaultFeature" AllowAdvertise="no" Title="!(loc.Title)" Description="!(loc.Description)">
+            <ComponentGroupRef Id="exe_python" />
+            <ComponentGroupRef Id="exe_txt" />
+            <ComponentGroupRef Id="exe_icons" />
+            
+            <Component Id="exe_shortcut" Directory="MenuDir" Guid="*">
+                <Shortcut Id="python.exe"
+                          Target="[#python.exe]"
+                          Name="!(loc.ShortcutName)"
+                          Description="!(loc.ShortcutDescription)" />
+                <RemoveFolder Id="Remove_MenuDir" Directory="MenuDir" On="uninstall" />
+                <RegistryKey Root="HKMU" Key="[REGISTRYKEY]">
+                    <RegistryValue Key="InstallPath" Type="string" Value="[InstallDirectory]" KeyPath="yes" />
+                    <RegistryValue Key="InstallPath\InstallGroup" Type="string" Value="Python $(var.ShortVersion)" />
+                </RegistryKey>
+            </Component>
+        </Feature>
+        <?endif ?>
+        <?ifdef IncludeSymbols ?>
+        <Feature Id="Symbols" AllowAdvertise="no" Title="!(loc.TitlePdb)" Description="!(loc.DescriptionPdb)">
+            <ComponentGroupRef Id="exe_python_symbols" />
+        </Feature>
+        <?endif ?>
+        <?ifdef IncludeDebugBinaries ?>
+        <Feature Id="DebugBinaries" AllowAdvertise="no" Title="!(loc.Title_d)" Description="!(loc.Description_d)">
+            <ComponentGroupRef Id="exe_python_d" />
+        </Feature>
+        <?endif ?>
+    </Product>
+</Wix>
diff --git a/Tools/msi/exe/exe_d.wixproj b/Tools/msi/exe/exe_d.wixproj
new file mode 100644
--- /dev/null
+++ b/Tools/msi/exe/exe_d.wixproj
@@ -0,0 +1,11 @@
+<?xml version="1.0" encoding="utf-8"?>
+<Project ToolsVersion="4.0" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+    <PropertyGroup>
+        <ProjectGuid>{B1CA739C-8DB0-403B-9010-D79507507CE9}</ProjectGuid>
+        <SchemaVersion>2.0</SchemaVersion>
+        <OutputName>exe_d</OutputName>
+        <OutputType>Package</OutputType>
+        <DefineConstants>IncludeDebugBinaries=1;$(DefineConstants)</DefineConstants>
+    </PropertyGroup>
+    <Import Project="exe.props" />
+</Project>
\ No newline at end of file
diff --git a/Tools/msi/exe/exe_en-US.wxl_template b/Tools/msi/exe/exe_en-US.wxl_template
new file mode 100644
--- /dev/null
+++ b/Tools/msi/exe/exe_en-US.wxl_template
@@ -0,0 +1,9 @@
+?<?xml version="1.0" encoding="utf-8"?>
+<WixLocalization Culture="en-us" xmlns="http://schemas.microsoft.com/wix/2006/localization">
+    <String Id="Descriptor">Executables</String>
+    <String Id="ShortDescriptor">executable</String>
+    <String Id="ShortcutName">Python {{ShortVersion}} ({{Bitness}})</String>
+    <String Id="ShortcutDescription">Launches the !(loc.ProductName) interpreter.</String>
+    <String Id="PathTitle">Add to PATH</String>
+    <String Id="PathDescription">Adds the install directory to PATH and .py to PATHEXT.</String>
+</WixLocalization>
diff --git a/Tools/msi/exe/exe_files.wxs b/Tools/msi/exe/exe_files.wxs
new file mode 100644
--- /dev/null
+++ b/Tools/msi/exe/exe_files.wxs
@@ -0,0 +1,68 @@
+?<?xml version="1.0" encoding="UTF-8"?>
+<Wix xmlns="http://schemas.microsoft.com/wix/2006/wi">
+    <Fragment>
+        <ComponentGroup Id="exe_txt">
+            <Component Id="LICENSE.txt" Directory="InstallDirectory" Guid="*">
+                <File Name="LICENSE.txt" Source="LICENSE" KeyPath="yes" />
+            </Component>
+            <Component Id="NEWS.txt" Directory="InstallDirectory" Guid="*">
+                <File Name="NEWS.txt" Source="!(bindpath.src)Misc\NEWS" KeyPath="yes" />
+            </Component>
+            <Component Id="README.txt" Directory="InstallDirectory" Guid="*">
+                <File Name="README.txt" Source="!(bindpath.src)README" KeyPath="yes" />
+            </Component>
+        </ComponentGroup>
+    </Fragment>
+
+    <Fragment>
+        <PropertyRef Id="REGISTRYKEY" />
+        
+        <ComponentGroup Id="exe_python">
+            <Component Id="python.exe" Directory="InstallDirectory" Guid="$(var.PythonExeComponentGuid)">
+                <File Name="python.exe" KeyPath="yes" />
+            </Component>
+            <Component Id="pythonw.exe" Directory="InstallDirectory" Guid="$(var.PythonwExeComponentGuid)">
+                <File Name="pythonw.exe" KeyPath="yes" />
+            </Component>
+        </ComponentGroup>
+    </Fragment>
+
+    <Fragment>
+        <ComponentGroup Id="exe_python_symbols">
+            <Component Id="python.pdb" Directory="InstallDirectory" Guid="*">
+                <File Name="python.pdb" />
+            </Component>
+            <Component Id="pythonw.pdb" Directory="InstallDirectory" Guid="*">
+                <File Name="pythonw.pdb" />
+            </Component>
+        </ComponentGroup>
+    </Fragment>
+        
+    <Fragment>
+        <ComponentGroup Id="exe_python_d">
+            <Component Id="python_d.exe" Directory="InstallDirectory" Guid="*">
+                <File Name="python_d.exe" />
+            </Component>
+            <Component Id="python_d.pdb" Directory="InstallDirectory" Guid="*">
+                <File Name="python_d.pdb" />
+            </Component>
+            <Component Id="pythonw_d.exe" Directory="InstallDirectory" Guid="*">
+                <File Name="pythonw_d.exe" />
+            </Component>
+            <Component Id="pythonw_d.pdb" Directory="InstallDirectory" Guid="*">
+                <File Name="pythonw_d.pdb" />
+            </Component>
+        </ComponentGroup>
+    </Fragment>
+    
+    <Fragment>
+        <ComponentGroup Id="exe_icons">
+            <Component Id="py.ico" Directory="DLLs" Guid="*">
+                <File Name="py.ico" Source="!(bindpath.src)PC\py.ico" KeyPath="yes" />
+            </Component>
+            <Component Id="pyc.ico" Directory="DLLs" Guid="*">
+                <File Name="pyc.ico" Source="!(bindpath.src)PC\pyc.ico" KeyPath="yes" />
+            </Component>
+        </ComponentGroup>
+    </Fragment>
+</Wix>
diff --git a/Tools/msi/exe/exe_pdb.wixproj b/Tools/msi/exe/exe_pdb.wixproj
new file mode 100644
--- /dev/null
+++ b/Tools/msi/exe/exe_pdb.wixproj
@@ -0,0 +1,11 @@
+<?xml version="1.0" encoding="utf-8"?>
+<Project ToolsVersion="4.0" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+    <PropertyGroup>
+        <ProjectGuid>{4A1F7045-8EE2-4276-ABB8-5E0C40E5F38B}</ProjectGuid>
+        <SchemaVersion>2.0</SchemaVersion>
+        <OutputName>exe_pdb</OutputName>
+        <OutputType>Package</OutputType>
+        <DefineConstants>IncludeSymbols=1;$(DefineConstants)</DefineConstants>
+    </PropertyGroup>
+    <Import Project="exe.props" />
+</Project>
\ No newline at end of file
diff --git a/Tools/msi/get_wix.py b/Tools/msi/get_wix.py
new file mode 100644
--- /dev/null
+++ b/Tools/msi/get_wix.py
@@ -0,0 +1,49 @@
+'''
+Downloads and extracts WiX to a local directory
+'''
+
+__author__ = 'Steve Dower <steve.dower at microsoft.com>'
+
+import io
+import os
+import sys
+
+from pathlib import Path
+from subprocess import Popen
+from zipfile import ZipFile
+
+EXTERNALS_DIR = None
+for p in Path(__file__).parents:
+    if any(p.glob("PCBuild/*.vcxproj")):
+        EXTERNALS_DIR = p / "externals"
+        break
+
+if not EXTERNALS_DIR:
+    print("Cannot find project root")
+    sys.exit(1)
+
+WIX_BINARIES_ZIP = 'http://wixtoolset.org/downloads/v3.10.1124.0/wix310-binaries.zip'
+TARGET_BIN_ZIP = EXTERNALS_DIR / "wix.zip"
+TARGET_BIN_DIR = EXTERNALS_DIR / "wix"
+
+POWERSHELL_COMMAND = "[IO.File]::WriteAllBytes('{}', (Invoke-WebRequest {} -UseBasicParsing).Content)"
+
+if __name__ == '__main__':
+    if TARGET_BIN_DIR.exists() and any(TARGET_BIN_DIR.glob("*")):
+        print('WiX is already installed')
+        sys.exit(0)
+
+    try:
+        TARGET_BIN_DIR.mkdir()
+    except FileExistsError:
+        pass
+
+    print('Downloading WiX to', TARGET_BIN_ZIP)
+    p = Popen(["powershell.exe", "-Command", POWERSHELL_COMMAND.format(TARGET_BIN_ZIP, WIX_BINARIES_ZIP)])
+    p.wait()
+    print('Extracting WiX to', TARGET_BIN_DIR)
+    with ZipFile(str(TARGET_BIN_ZIP)) as z:
+        z.extractall(str(TARGET_BIN_DIR))
+    TARGET_BIN_ZIP.unlink()
+
+    print('Extracted WiX')
diff --git a/Tools/msi/launcher/launcher.props b/Tools/msi/launcher/launcher.props
new file mode 100644
--- /dev/null
+++ b/Tools/msi/launcher/launcher.props
@@ -0,0 +1,12 @@
+<?xml version="1.0" encoding="utf-8"?>
+<Project ToolsVersion="4.0" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+    <Import Project="..\msi.props" />
+    <ItemGroup>
+        <Compile Include="*.wxs" />
+    </ItemGroup>
+    <ItemGroup>
+        <EmbeddedResource Include="*.wxl" />
+    </ItemGroup>
+    
+    <Import Project="..\msi.targets" />
+</Project>
\ No newline at end of file
diff --git a/Tools/msi/launcher/launcher.wixproj b/Tools/msi/launcher/launcher.wixproj
new file mode 100644
--- /dev/null
+++ b/Tools/msi/launcher/launcher.wixproj
@@ -0,0 +1,11 @@
+<?xml version="1.0" encoding="utf-8"?>
+<Project ToolsVersion="4.0" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+    <PropertyGroup>
+        <ProjectGuid>{921CF0E6-AEBC-4376-BA1D-CD46EBFE6DA5}</ProjectGuid>
+        <SchemaVersion>2.0</SchemaVersion>
+        <OutputName>launcher</OutputName>
+        <OutputType>Package</OutputType>
+        <DefineConstants>IncludeDefaultFeature=1;$(DefineConstants)</DefineConstants>
+    </PropertyGroup>
+    <Import Project="launcher.props" />
+</Project>
\ No newline at end of file
diff --git a/Tools/msi/launcher/launcher.wxs b/Tools/msi/launcher/launcher.wxs
new file mode 100644
--- /dev/null
+++ b/Tools/msi/launcher/launcher.wxs
@@ -0,0 +1,38 @@
+?<?xml version="1.0" encoding="UTF-8"?>
+<Wix xmlns="http://schemas.microsoft.com/wix/2006/wi">
+    <Product Id="*" Language="!(loc.LCID)" Name="!(loc.Title)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)">
+        <Package InstallerVersion="300" Compressed="yes" InstallScope="perUser" Platform="$(var.Platform)" />
+        <MediaTemplate EmbedCab="yes" CompressionLevel="high" />
+        
+        <Property Id="Suppress_TARGETDIR_Check" Value="1" />
+        <PropertyRef Id="UpgradeTable" />
+        <PropertyRef Id="ARPPRODUCTICON" />
+
+        <?ifdef IncludeDefaultFeature ?>
+        <Feature Id="DefaultFeature" AllowAdvertise="no" Title="!(loc.Title)" Description="!(loc.Description)">
+            <ComponentGroupRef Id="launcher_exe" Primary="yes" />
+        </Feature>
+        <Feature Id="AssociateFiles" AllowAdvertise="no" Title="!(loc.Title)" Description="!(loc.Description)">
+            <ComponentGroupRef Id="launcher_exe" />
+            <ComponentGroupRef Id="launcher_reg" />
+        </Feature>
+        <?endif ?>
+        <?ifdef IncludeSymbols ?>
+        <Feature Id="Symbols" AllowAdvertise="no" Title="!(loc.TitlePdb)" Description="!(loc.DescriptionPdb)">
+            <ComponentGroupRef Id="launcher_pdb" />
+        </Feature>
+        <?endif ?>
+        
+        <Directory Id="TARGETDIR" Name="SourceDir">
+            <Directory Id="LauncherInstallDirectory" />
+        </Directory>
+        
+        <CustomAction Id="SetLauncherInstallDirectoryLM" Property="LauncherInstallDirectory" Value="[WindowsFolder]" />
+        <CustomAction Id="SetLauncherInstallDirectoryCU" Property="LauncherInstallDirectory" Value="[LocalAppDataFolder]Programs\Python\Launcher" />
+        
+        <InstallExecuteSequence>
+            <Custom Before="SetLauncherInstallDirectoryLM" Action="SetLauncherInstallDirectoryCU">NOT Installed AND NOT ALLUSERS=1</Custom>
+            <Custom Before="CostFinalize" Action="SetLauncherInstallDirectoryLM">NOT Installed AND ALLUSERS=1</Custom>
+        </InstallExecuteSequence>
+    </Product>
+</Wix>
diff --git a/Tools/msi/launcher/launcher_en-US.wxl b/Tools/msi/launcher/launcher_en-US.wxl
new file mode 100644
--- /dev/null
+++ b/Tools/msi/launcher/launcher_en-US.wxl
@@ -0,0 +1,8 @@
+?<?xml version="1.0" encoding="utf-8"?>
+<WixLocalization Culture="en-us" xmlns="http://schemas.microsoft.com/wix/2006/localization">
+    <String Id="Descriptor">Launcher</String>
+    <String Id="ShortDescriptor">launcher</String>
+    <String Id="PythonFileDescription">Python File</String>
+    <String Id="PythonNoConFileDescription">Python File (no console)</String>
+    <String Id="PythonCompiledFileDescription">Compiled Python File</String>
+</WixLocalization>
diff --git a/Tools/msi/launcher/launcher_files.wxs b/Tools/msi/launcher/launcher_files.wxs
new file mode 100644
--- /dev/null
+++ b/Tools/msi/launcher/launcher_files.wxs
@@ -0,0 +1,35 @@
+?<?xml version="1.0" encoding="UTF-8"?>
+<Wix xmlns="http://schemas.microsoft.com/wix/2006/wi">
+    <Fragment>
+        <ComponentGroup Id="launcher_exe">
+            <Component Id="py.exe" Directory="LauncherInstallDirectory" Guid="{B5107402-6958-461B-8B0A-4037D3327160}">
+                <File Id="py.exe" Name="py.exe" Source="py.exe" KeyPath="yes" />
+            </Component>
+            <Component Id="pyw.exe" Directory="LauncherInstallDirectory" Guid="{8E52B8CD-48BB-4D74-84CD-6238BCD11F20}">
+                <File Id="pyw.exe" Name="pyw.exe" Source="pyw.exe" KeyPath="yes" />
+            </Component>
+
+            <Component Id="launcher_path_cu" Directory="LauncherInstallDirectory" Guid="{95AEB930-367C-475C-A17E-A89BFCD4C670}">
+                <Condition>NOT ALLUSERS=1</Condition>
+                
+                <RegistryValue KeyPath="yes" Root="HKMU" Key="Software\Python\PyLauncher\InstallDir" Value="[LauncherInstallDirectory]" Type="string" />
+                <Environment Id="PATH_CU" Action="set" Name="PATH" Part="first" Value="[LauncherInstallDirectory]" />
+            </Component>
+            <Component Id="launcher_path_lm" Directory="LauncherInstallDirectory" Guid="{4A41C365-4E27-4D38-A6D1-4A01B4A6500C}">
+                <Condition>ALLUSERS=1</Condition>
+                <RegistryValue KeyPath="yes" Root="HKMU" Key="Software\Python\PyLauncher\InstallDir" Value="[LauncherInstallDirectory]" Type="string" />
+            </Component>
+        </ComponentGroup>
+    </Fragment>
+    
+    <Fragment>
+        <ComponentGroup Id="launcher_pdb">
+            <Component Id="py.pdb" Directory="LauncherInstallDirectory" Guid="*">
+                <File Id="py.pdb" Name="py.pdb" Source="py.pdb" />
+            </Component>
+            <Component Id="pyw.pdb" Directory="LauncherInstallDirectory" Guid="*">
+                <File Id="pyw.pdb" Name="pyw.pdb" Source="pyw.pdb" />
+            </Component>
+        </ComponentGroup>
+    </Fragment>
+</Wix>
diff --git a/Tools/msi/launcher/launcher_pdb.wixproj b/Tools/msi/launcher/launcher_pdb.wixproj
new file mode 100644
--- /dev/null
+++ b/Tools/msi/launcher/launcher_pdb.wixproj
@@ -0,0 +1,11 @@
+<?xml version="1.0" encoding="utf-8"?>
+<Project ToolsVersion="4.0" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+    <PropertyGroup>
+        <ProjectGuid>{A21D4A23-483F-4822-A0B1-FCB14D8CEBA7}</ProjectGuid>
+        <SchemaVersion>2.0</SchemaVersion>
+        <OutputName>launcher_pdb</OutputName>
+        <OutputType>Package</OutputType>
+        <DefineConstants>IncludeSymbols=1;$(DefineConstants)</DefineConstants>
+    </PropertyGroup>
+    <Import Project="launcher.props" />
+</Project>
\ No newline at end of file
diff --git a/Tools/msi/launcher/launcher_reg.wxs b/Tools/msi/launcher/launcher_reg.wxs
new file mode 100644
--- /dev/null
+++ b/Tools/msi/launcher/launcher_reg.wxs
@@ -0,0 +1,32 @@
+?<?xml version="1.0" encoding="UTF-8"?>
+<Wix xmlns="http://schemas.microsoft.com/wix/2006/wi">
+    <Fragment>
+        <ComponentGroup Id="launcher_reg">
+            <Component Id="file_association" Directory="LauncherInstallDirectory" Guid="{5AF84D9A-D820-456B-B230-6E0105A50276}">
+                <RegistryValue KeyPath="yes" Root="HKMU" Key="Software\Python\PyLauncher\AssociateFiles" Value="1" Type="integer" />
+                
+                <ProgId Id="$(var.TestPrefix)Python.File" Description="!(loc.PythonFileDescription)" Advertise="no" Icon="py.exe" IconIndex="1">
+                    <Extension Id="$(var.FileExtension)" ContentType="text/plain">
+                        <Verb Id="open" TargetFile="py.exe" Argument="&quot;%L&quot; %*" />
+                    </Extension>
+                </ProgId>
+                <RegistryValue Root="HKCR" Key="$(var.TestPrefix)Python.File\shellex\DropHandler" Value="{60254CA5-953B-11CF-8C96-00AA00B8708C}" Type="string" />
+                
+                <ProgId Id="$(var.TestPrefix)Python.NoConFile" Description="!(loc.PythonNoConFileDescription)" Advertise="no" Icon="py.exe" IconIndex="1">
+                    <Extension Id="$(var.FileExtension)w" ContentType="text/plain">
+                        <Verb Id="open" TargetFile="pyw.exe" Argument="&quot;%L&quot; %*" />
+                    </Extension>
+                </ProgId>
+                <RegistryValue Root="HKCR" Key="$(var.TestPrefix)Python.NoConFile\shellex\DropHandler" Value="{60254CA5-953B-11CF-8C96-00AA00B8708C}" Type="string" />
+                
+                <ProgId Id="$(var.TestPrefix)Python.CompiledFile" Description="!(loc.PythonCompiledFileDescription)" Advertise="no" Icon="py.exe" IconIndex="2">
+                    <Extension Id="$(var.FileExtension)c">
+                        <Verb Id="open" TargetFile="py.exe" Argument="&quot;%L&quot; %*" />
+                    </Extension>
+                    <Extension Id="$(var.FileExtension)o" />
+                </ProgId>
+                <RegistryValue Root="HKCR" Key="$(var.TestPrefix)Python.CompiledFile\shellex\DropHandler" Value="{60254CA5-953B-11CF-8C96-00AA00B8708C}" Type="string" />
+            </Component>
+        </ComponentGroup>
+    </Fragment>
+</Wix>
diff --git a/Tools/msi/lib/lib.props b/Tools/msi/lib/lib.props
new file mode 100644
--- /dev/null
+++ b/Tools/msi/lib/lib.props
@@ -0,0 +1,27 @@
+<?xml version="1.0" encoding="utf-8"?>
+<Project ToolsVersion="4.0" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+    <Import Project="..\msi.props" />
+    <ItemGroup>
+        <Compile Include="*.wxs" />
+    </ItemGroup>
+    <ItemGroup>
+        <EmbeddedResource Include="*.wxl" />
+    </ItemGroup>
+    <ItemGroup>
+        <ExcludeFolders Include="Lib\test;Lib\tests;Lib\tkinter;Lib\idlelib;Lib\turtledemo" />
+        <InstallFiles Include="$(PySourcePath)Lib\**\*"
+                      Exclude="$(PySourcePath)Lib\**\*.pyc;
+                               $(PySourcePath)Lib\**\*.pyo;
+                               $(PySourcePath)Lib\site-packages\README;
+                               @(ExcludeFolders->'$(PySourcePath)%(Identity)\*');
+                               @(ExcludeFolders->'$(PySourcePath)%(Identity)\**\*')">
+            <SourceBase>$(PySourcePath)Lib</SourceBase>
+            <Source>!(bindpath.src)Lib\</Source>
+            <TargetBase>$(PySourcePath)Lib</TargetBase>
+            <Target_>Lib\</Target_>
+            <Group>lib_py</Group>
+        </InstallFiles>
+    </ItemGroup>
+    
+    <Import Project="..\msi.targets" />
+</Project>
\ No newline at end of file
diff --git a/Tools/msi/lib/lib.wixproj b/Tools/msi/lib/lib.wixproj
new file mode 100644
--- /dev/null
+++ b/Tools/msi/lib/lib.wixproj
@@ -0,0 +1,11 @@
+<?xml version="1.0" encoding="utf-8"?>
+<Project ToolsVersion="4.0" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+    <PropertyGroup>
+        <ProjectGuid>{11367E76-3337-4602-8F1E-77DB4F370D7E}</ProjectGuid>
+        <SchemaVersion>2.0</SchemaVersion>
+        <OutputName>lib</OutputName>
+        <OutputType>Package</OutputType>
+        <DefineConstants>IncludeDefaultFeature=1;$(DefineConstants)</DefineConstants>
+    </PropertyGroup>
+    <Import Project="lib.props" />
+</Project>
\ No newline at end of file
diff --git a/Tools/msi/lib/lib.wxs b/Tools/msi/lib/lib.wxs
new file mode 100644
--- /dev/null
+++ b/Tools/msi/lib/lib.wxs
@@ -0,0 +1,28 @@
+?<?xml version="1.0" encoding="UTF-8"?>
+<Wix xmlns="http://schemas.microsoft.com/wix/2006/wi">
+    <Product Id="*" Language="!(loc.LCID)" Name="!(loc.Title)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)">
+        <Package InstallerVersion="300" Compressed="yes" InstallScope="perUser" Platform="$(var.Platform)" />
+        <MediaTemplate EmbedCab="yes" CompressionLevel="high" />
+        
+        <PropertyRef Id="UpgradeTable" />
+        <PropertyRef Id="REGISTRYKEY" />
+        
+        <?ifdef IncludeDefaultFeature ?>
+        <Feature Id="DefaultFeature" AllowAdvertise="no" Title="!(loc.Title)" Description="!(loc.Description)">
+            <ComponentGroupRef Id="lib_py" />
+            <ComponentGroupRef Id="lib_files" />
+            <ComponentGroupRef Id="lib_extensions" />
+        </Feature>
+        <?endif ?>
+        <?ifdef IncludeSymbols ?>
+        <Feature Id="Symbols" AllowAdvertise="no" Title="!(loc.TitlePdb)" Description="!(loc.DescriptionPdb)">
+            <ComponentGroupRef Id="lib_extensions_symbols" />
+        </Feature>
+        <?endif ?>
+        <?ifdef IncludeDebugBinaries ?>
+        <Feature Id="DebugBinaries" AllowAdvertise="no" Title="!(loc.Title_d)" Description="!(loc.Description_d)">
+            <ComponentGroupRef Id="lib_extensions_d" />
+        </Feature>
+        <?endif ?>
+    </Product>
+</Wix>
diff --git a/Tools/msi/lib/lib_d.wixproj b/Tools/msi/lib/lib_d.wixproj
new file mode 100644
--- /dev/null
+++ b/Tools/msi/lib/lib_d.wixproj
@@ -0,0 +1,11 @@
+<?xml version="1.0" encoding="utf-8"?>
+<Project ToolsVersion="4.0" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+    <PropertyGroup>
+        <ProjectGuid>{6C443CD3-8258-4335-BA03-49DA9C34CE4D}</ProjectGuid>
+        <SchemaVersion>2.0</SchemaVersion>
+        <OutputName>lib_d</OutputName>
+        <OutputType>Package</OutputType>
+        <DefineConstants>IncludeDebugBinaries=1;$(DefineConstants)</DefineConstants>
+    </PropertyGroup>
+    <Import Project="lib.props" />
+</Project>
\ No newline at end of file
diff --git a/Tools/msi/lib/lib_en-US.wxl b/Tools/msi/lib/lib_en-US.wxl
new file mode 100644
--- /dev/null
+++ b/Tools/msi/lib/lib_en-US.wxl
@@ -0,0 +1,5 @@
+?<?xml version="1.0" encoding="utf-8"?>
+<WixLocalization Culture="en-us" xmlns="http://schemas.microsoft.com/wix/2006/localization">
+    <String Id="Descriptor">Standard Library</String>
+    <String Id="ShortDescriptor">lib</String>
+</WixLocalization>
diff --git a/Tools/msi/lib/lib_files.wxs b/Tools/msi/lib/lib_files.wxs
new file mode 100644
--- /dev/null
+++ b/Tools/msi/lib/lib_files.wxs
@@ -0,0 +1,72 @@
+?<?xml version="1.0" encoding="UTF-8"?>
+<Wix xmlns="http://schemas.microsoft.com/wix/2006/wi">
+    <?define exts=pyexpat;select;unicodedata;winsound;_bz2;_elementtree;_socket;_ssl;_msi;_ctypes;_hashlib;_multiprocessing;_lzma;_decimal;_overlapped;_sqlite3 ?>
+    <Fragment>
+        <ComponentGroup Id="lib_extensions">
+            <?foreach ext in $(var.exts)?>
+        
+            <Component Id="$(var.ext).pyd" Directory="DLLs" Guid="*">
+                <File Name="$(var.ext).pyd" KeyPath="yes" />
+            </Component>
+            
+            <?endforeach ?>
+            
+            <Component Id="sqlite3.dll" Directory="DLLs" Guid="*">
+                <File Name="sqlite3.dll" KeyPath="yes" />
+            </Component>
+        </ComponentGroup>
+    </Fragment>
+    
+    <Fragment>
+        <ComponentGroup Id="lib_extensions_symbols">
+            <?foreach ext in $(var.exts)?>
+            
+            <Component Id="$(var.ext).pdb" Directory="DLLs" Guid="*">
+                <Condition>SYMBOLS=1</Condition>
+                <File Name="$(var.ext).pdb" />
+            </Component>
+            
+            <?endforeach ?>
+            
+            <Component Id="sqlite3.pdb" Directory="DLLs" Guid="*">
+                <File Name="sqlite3.pdb" />
+            </Component>
+        </ComponentGroup>
+    </Fragment>
+    
+    <Fragment>
+        <ComponentGroup Id="lib_extensions_d">
+            <?foreach ext in $(var.exts)?>
+            
+            <Component Id="$(var.ext)_d.pyd" Directory="DLLs" Guid="*">
+                <File Name="$(var.ext)_d.pyd" />
+            </Component>
+            <Component Id="$(var.ext)_d.pdb" Directory="DLLs" Guid="*">
+                <File Name="$(var.ext)_d.pdb" />
+            </Component>
+            
+            <?endforeach ?>
+        </ComponentGroup>
+    </Fragment>
+    <Fragment>
+        <PropertyRef Id="REGISTRYKEY" />
+        
+        <ComponentGroup Id="lib_files">
+            <Component Id="PythonPathRegistry" Directory="Lib" Guid="*">
+                <RegistryKey Root="HKMU" Key="[REGISTRYKEY]">
+                    <RegistryValue Key="PythonPath" Type="string" Value="[Lib];[DLLs]" />
+                </RegistryKey>
+            </Component>
+            <Component Id="Lib_site_packages_README" Directory="Lib_site_packages" Guid="*">
+                <File Id="Lib_site_packages_README" Name="README.txt" Source="!(bindpath.src)Lib\site-packages\README" KeyPath="yes" />
+            </Component>
+            <Component Id="Lib2to3_pickle_remove" Directory="Lib_lib2to3" Guid="$(var.RemoveLib2to3PickleComponentGuid)">
+                <RemoveFile Id="Lib2to3_pickle_remove_files" Name="*.pickle" On="uninstall" />
+                <RemoveFolder Id="Lib2to3_pickle_remove_folder" On="uninstall" />
+            </Component>
+        </ComponentGroup>
+        <DirectoryRef Id="Lib">
+            <Directory Id="Lib_site_packages" Name="site-packages" />
+        </DirectoryRef>
+    </Fragment>
+</Wix>
diff --git a/Tools/msi/lib/lib_pdb.wixproj b/Tools/msi/lib/lib_pdb.wixproj
new file mode 100644
--- /dev/null
+++ b/Tools/msi/lib/lib_pdb.wixproj
@@ -0,0 +1,11 @@
+<?xml version="1.0" encoding="utf-8"?>
+<Project ToolsVersion="4.0" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+    <PropertyGroup>
+        <ProjectGuid>{5E0BCE93-D1AC-4591-BBCB-3A2BE5A4B3D1}</ProjectGuid>
+        <SchemaVersion>2.0</SchemaVersion>
+        <OutputName>lib_pdb</OutputName>
+        <OutputType>Package</OutputType>
+        <DefineConstants>IncludeSymbols=1;$(DefineConstants)</DefineConstants>
+    </PropertyGroup>
+    <Import Project="lib.props" />
+</Project>
\ No newline at end of file
diff --git a/Tools/msi/msi.props b/Tools/msi/msi.props
new file mode 100644
--- /dev/null
+++ b/Tools/msi/msi.props
@@ -0,0 +1,161 @@
+<?xml version="1.0" encoding="utf-8"?>
+<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003" TreatAsLocalProperty="ReleaseUri">
+    <PropertyGroup>
+        <DefineSolutionProperties>false</DefineSolutionProperties>
+        <TreatWarningsAsErrors>false</TreatWarningsAsErrors>
+        <SuppressIces>$(SuppressIces);ICE03;ICE57;ICE61</SuppressIces>
+        <CompilerSuppressSpecificWarnings>1026</CompilerSuppressSpecificWarnings>
+        <BuildForRelease Condition="'$(BuildForRelease)' == ''">false</BuildForRelease>
+        <SignOutput Condition="'$(SigningCertificate)' != ''">true</SignOutput>
+        <Configuration Condition="'$(Configuration)' == ''">Release</Configuration>
+        <Platform Condition="'$(Platform)' == ''">x86</Platform>
+        <InstallScope Condition="'$(InstallScope)' != 'perMachine'">perUser</InstallScope>
+        <!--
+        This URI is used to generate the various GUIDs used by the installer.
+        Installers built with the same URI will upgrade each other or block
+        when attempting to downgrade.
+        
+        By default, this is the local computer name, which will produce
+        installers that do not interfere with other installers. Products
+        that intend to bundle Python should rebuild these modules with their
+        own URI to avoid conflicting with the official releases.
+        
+        The official releases are built with http://www.python.org/.
+        
+        This is not the same as the DownloadUrl property used in the bundle
+        projects.
+        -->
+        <ReleaseUri Condition="'$(ReleaseUri)' == ''">$(ComputerName)</ReleaseUri>
+        <ReleaseUri Condition="!$(ReleaseUri.EndsWith(`/`))">$(ReleaseUri)/</ReleaseUri>
+    </PropertyGroup>
+
+    <Import Project="wix.props" />
+    <Import Project="..\..\PCBuild\tcltk.props" />
+    
+    <ItemGroup>
+        <Compile Include="$(MSBuildThisFileDirectory)common.wxs" />
+        <WxlTemplate Include="$(MSBuildThisFileDirectory)\*.wxl_template" />
+        <WixExtension Include="WixUtilExtension">
+            <HintPath>WixUtilExtension</HintPath>
+            <Name>WixUtilExtension</Name>
+        </WixExtension>
+    </ItemGroup>
+
+    <PropertyGroup>
+        <IntermediateOutputPath>$(MSBuildThisFileDirectory)\obj\$(Configuration)_$(Platform)\$(OutputName)</IntermediateOutputPath>
+        <IntermediateOutputPath Condition="'$(OutputSuffix)' != ''">$(IntermediateOutputPath)_$(OutputSuffix)</IntermediateOutputPath>
+        <OutputPath Condition="'$(OutputPath)' == ''">$(BuildPath)</OutputPath>
+        <OutputPath Condition="!HasTrailingSlash($(OutputPath))">$(OutputPath)\</OutputPath>
+        <OutDir>$(OutputPath)</OutDir>
+        <ReuseCabinetCache>true</ReuseCabinetCache>
+        <CRTModule Condition="'$(VisualStudioVersion)' == '12.0'">$(CommonProgramFiles)\Merge Modules\Microsoft_VC120_CRT_$(Platform).msm</CRTModule>
+        <CRTModule Condition="'$(VisualStudioVersion)' == '14.0'">$(CommonProgramFiles)\Merge Modules\Microsoft_VC140_CRT_$(Platform).msm</CRTModule>
+        <CRTRedist Condition="'$(VisualStudioVersion)' == '12.0'">$([System.IO.Path]::GetFullPath(`$(VS120COMNTOOLS)\..\..\VC\redist\$(Platform)\Microsoft.VC120.CRT`))</CRTRedist>
+        <CRTRedist Condition="'$(VisualStudioVersion)' == '14.0'">$([System.IO.Path]::GetFullPath(`$(VS140COMNTOOLS)\..\..\VC\redist\$(Platform)\Microsoft.VC140.CRT`))</CRTRedist>
+        <CRTRedist Condition="'$(CRTRedist)' != '' and !Exists($(CRTRedist))"></CRTRedist>
+
+        <RevisionNumber>$(ReleaseLevelNumber)</RevisionNumber>
+        <RevisionNumber Condition="!$(BuildForRelease)">$([System.Math]::Floor($([System.DateTime]::Now.Subtract($([System.DateTime]::new(2001, 1, 1))).TotalDays)))</RevisionNumber>
+    </PropertyGroup>
+    
+    <PropertyGroup>
+        <Bitness>32-bit</Bitness>
+        <Bitness Condition="$(Platform) == 'x64'">64-bit</Bitness>
+        <DefineConstants>
+            $(DefineConstants);
+            Version=$(MajorVersionNumber).$(MinorVersionNumber).$(MicroVersionNumber).$(RevisionNumber);
+            ShortVersion=$(MajorVersionNumber).$(MinorVersionNumber);
+            LongVersion=$(PythonVersion);
+            MajorVersionNumber=$(MajorVersionNumber);
+            MinorVersionNumber=$(MinorVersionNumber);
+            UpgradeMinimumVersion=$(MajorVersionNumber).$(MinorVersionNumber).0.0;
+            UpgradeMaximumVersion=$(MajorVersionNumber).$(MinorVersionNumber).150.0;
+            NextMajorVersionNumber=$(MajorVersionNumber).$([msbuild]::Add($(MinorVersionNumber), 1)).0.0;
+            PyDebugExt=$(PyDebugExt);
+        </DefineConstants>
+        <DefineConstants Condition="'$(CRTModule)' != '' and Exists($(CRTModule))">
+            $(DefineConstants);CRTModule=$(CRTModule);
+        </DefineConstants>
+        <DefineConstants Condition="'$(CRTRedist)' != ''">
+            $(DefineConstants);CRTRedist=$(CRTRedist);
+        </DefineConstants>
+        <DefineConstants Condition="$(Configuration) != 'Debug'">
+            $(DefineConstants);TestPrefix=;FileExtension=py;
+        </DefineConstants>
+        <DefineConstants Condition="$(Configuration) == 'Debug'">
+            $(DefineConstants);TestPrefix=x;FileExtension=px;
+        </DefineConstants>
+        <DefineConstants Condition="$(Platform) != 'x64'">
+            $(DefineConstants);Suffix32=-32;
+        </DefineConstants>
+        <DefineConstants Condition="$(Platform) == 'x64'">
+            $(DefineConstants);Suffix32=;
+        </DefineConstants>
+    </PropertyGroup>
+
+    <ItemDefinitionGroup>
+        <InstallFiles>
+            <Group>generated_filelist</Group>
+            <Condition></Condition>
+            <DiskId></DiskId>
+        </InstallFiles>
+        <LinkerBindInputPaths>
+            <Visible>false</Visible>
+        </LinkerBindInputPaths>
+    </ItemDefinitionGroup>
+    <ItemGroup>
+        <LinkerBindInputPaths Include="$(BuildPath);$(PySourcePath)Doc\build\htmlhelp">
+            <BindName></BindName>
+        </LinkerBindInputPaths>
+        <LinkerBindInputPaths Include="$(PySourcePath)">
+            <BindName>src</BindName>
+        </LinkerBindInputPaths>
+        <LinkerBindInputPaths Include="$(tcltkDir)">
+            <BindName>tcltk</BindName>
+        </LinkerBindInputPaths>
+        <LinkerBindInputPaths Include="$(CRTRedist)" Condition="'$(CRTRedist)' != ''">
+            <BindName>crt</BindName>
+        </LinkerBindInputPaths>
+    </ItemGroup>
+
+    <Target Name="_ValidateMsiProps" BeforeTargets="PrepareForBuild">
+        <Error Text="Platform '$(Platform)' is not supported. Use 'x86' or 'x64'." Condition="$(Platform) != 'x86' and $(Platform) != 'x64'" />
+    </Target>
+    
+    <ItemGroup>
+        <_Uuid Include="CoreUpgradeCode">
+            <Uri>upgradecode</Uri>
+        </_Uuid>
+        <_Uuid Include="UpgradeCode">
+            <Uri>upgradecode/$(OutputName)</Uri>
+        </_Uuid>
+        <_Uuid Include="InstallDirectoryGuidSeed">
+            <Uri>installdirectoryseed</Uri>
+        </_Uuid>
+        <_Uuid Include="PythonExeComponentGuid">
+            <Uri>python.exe</Uri>
+        </_Uuid>
+        <_Uuid Include="PythonwExeComponentGuid">
+            <Uri>pythonw.exe</Uri>
+        </_Uuid>
+        <_Uuid Include="RemoveLib2to3PickleComponentGuid">
+            <Uri>lib2to3/pickles</Uri>
+        </_Uuid>
+    </ItemGroup>
+    <Target Name="_GenerateGuids" AfterTargets="PrepareForBuild">
+        <PropertyGroup>
+            <_Uuids>@(_Uuid->'("%(Identity)", "%(Uri)")',',')</_Uuids>
+            <_GenerateCommand>import uuid; print('\n'.join('{}={}'.format(i, uuid.uuid5(uuid.UUID('c8d9733e-a70c-43ff-ab0c-e26456f11083'), '$(ReleaseUri)' + j)) for i,j in [$(_Uuids.Replace(`"`,`'`))]))</_GenerateCommand>
+        </PropertyGroup>
+        
+        <Exec Command='"$(PythonExe)" -c "$(_GenerateCommand)" &gt; "$(IntermediateOutputPath)$(OutputName)guids.txt"' IgnoreExitCode="false" />
+        
+        <ReadLinesFromFile File="$(IntermediateOutputPath)$(OutputName)guids.txt">
+            <Output TaskParameter="Lines" ItemName="_UuidValue" />
+        </ReadLinesFromFile>
+        
+        <PropertyGroup>
+            <DefineConstants>$(DefineConstants);@(_UuidValue,';');</DefineConstants>
+        </PropertyGroup>
+    </Target>
+</Project>
\ No newline at end of file
diff --git a/Tools/msi/msi.py b/Tools/msi/msi.py
deleted file mode 100644
--- a/Tools/msi/msi.py
+++ /dev/null
@@ -1,1454 +0,0 @@
-# Python MSI Generator
-# (C) 2003 Martin v. Loewis
-# See "FOO" in comments refers to MSDN sections with the title FOO.
-import msilib, schema, sequence, os, glob, time, re, shutil, zipfile
-import subprocess, tempfile
-from msilib import Feature, CAB, Directory, Dialog, Binary, add_data
-import uisample
-from win32com.client import constants
-from distutils.spawn import find_executable
-
-# Settings can be overridden in config.py below
-# 0 for official python.org releases
-# 1 for intermediate releases by anybody, with
-# a new product code for every package.
-snapshot = 1
-# 1 means that file extension is px, not py,
-# and binaries start with x
-testpackage = 0
-# Location of build tree
-srcdir = os.path.abspath("../..")
-# Text to be displayed as the version in dialogs etc.
-# goes into file name and ProductCode. Defaults to
-# current_version.day for Snapshot, current_version otherwise
-full_current_version = None
-# Is Tcl available at all?
-have_tcl = True
-# path to PCbuild directory
-PCBUILD="PCbuild"
-# msvcrt version
-MSVCR = "100"
-# Name of certificate in default store to sign MSI with
-certname = None
-# Make a zip file containing the PDB files for this build?
-pdbzip = True
-
-try:
-    from config import *
-except ImportError:
-    pass
-
-# Extract current version from Include/patchlevel.h
-lines = open(srcdir + "/Include/patchlevel.h").readlines()
-major = minor = micro = level = serial = None
-levels = {
-    'PY_RELEASE_LEVEL_ALPHA':0xA,
-    'PY_RELEASE_LEVEL_BETA': 0xB,
-    'PY_RELEASE_LEVEL_GAMMA':0xC,
-    'PY_RELEASE_LEVEL_FINAL':0xF
-    }
-for l in lines:
-    if not l.startswith("#define"):
-        continue
-    l = l.split()
-    if len(l) != 3:
-        continue
-    _, name, value = l
-    if name == 'PY_MAJOR_VERSION': major = value
-    if name == 'PY_MINOR_VERSION': minor = value
-    if name == 'PY_MICRO_VERSION': micro = value
-    if name == 'PY_RELEASE_LEVEL': level = levels[value]
-    if name == 'PY_RELEASE_SERIAL': serial = value
-
-short_version = major+"."+minor
-# See PC/make_versioninfo.c
-FIELD3 = 1000*int(micro) + 10*level + int(serial)
-current_version = "%s.%d" % (short_version, FIELD3)
-
-# This should never change. The UpgradeCode of this package can be
-# used in the Upgrade table of future packages to make the future
-# package replace this one. See "UpgradeCode Property".
-# upgrade_code gets set to upgrade_code_64 when we have determined
-# that the target is Win64.
-upgrade_code_snapshot='{92A24481-3ECB-40FC-8836-04B7966EC0D5}'
-upgrade_code='{65E6DE48-A358-434D-AA4F-4AF72DB4718F}'
-upgrade_code_64='{6A965A0C-6EE6-4E3A-9983-3263F56311EC}'
-
-if snapshot:
-    current_version = "%s.%s.%s" % (major, minor, int(time.time()/3600/24))
-
-if full_current_version is None:
-    full_current_version = current_version
-
-extensions = [
-    'pyexpat.pyd',
-    'select.pyd',
-    'unicodedata.pyd',
-    'winsound.pyd',
-    '_bz2.pyd',
-    '_elementtree.pyd',
-    '_socket.pyd',
-    '_ssl.pyd',
-    '_testcapi.pyd',
-    '_tkinter.pyd',
-    '_msi.pyd',
-    '_ctypes.pyd',
-    '_ctypes_test.pyd',
-    '_sqlite3.pyd',
-    '_hashlib.pyd',
-    '_multiprocessing.pyd',
-    '_lzma.pyd',
-    '_decimal.pyd',
-    '_testbuffer.pyd',
-    '_testimportmultiple.pyd',
-    '_overlapped.pyd',
-]
-
-# Well-known component UUIDs
-# These are needed for SharedDLLs reference counter; if
-# a different UUID was used for each incarnation of, say,
-# python24.dll, an upgrade would set the reference counter
-# from 1 to 2 (due to what I consider a bug in MSI)
-# Using the same UUID is fine since these files are versioned,
-# so Installer will always keep the newest version.
-# NOTE: All uuids are self generated.
-pythondll_uuid = {
-    "24":"{9B81E618-2301-4035-AC77-75D9ABEB7301}",
-    "25":"{2e41b118-38bd-4c1b-a840-6977efd1b911}",
-    "26":"{34ebecac-f046-4e1c-b0e3-9bac3cdaacfa}",
-    "27":"{4fe21c76-1760-437b-a2f2-99909130a175}",
-    "30":"{6953bc3b-6768-4291-8410-7914ce6e2ca8}",
-    "31":"{4afcba0b-13e4-47c3-bebe-477428b46913}",
-    "32":"{3ff95315-1096-4d31-bd86-601d5438ad5e}",
-    "33":"{f7581ca4-d368-4eea-8f82-d48c64c4f047}",
-    "34":"{7A0C5812-2583-40D9-BCBB-CD7485F11377}",
-    } [major+minor]
-
-# Compute the name that Sphinx gives to the docfile
-docfile = micro
-if level < 0xf:
-    if level == 0xC:
-        docfile += "rc%s" % (serial,)
-    else:
-        docfile += '%x%s' % (level, serial)
-docfile = 'python%s%s%s.chm' % (major, minor, docfile)
-
-# Build the mingw import library, libpythonXY.a
-# This requires 'nm' and 'dlltool' executables on your PATH
-def build_mingw_lib(lib_file, def_file, dll_file, mingw_lib):
-    warning = "WARNING: %s - libpythonXX.a not built"
-    nm = find_executable('nm')
-    dlltool = find_executable('dlltool')
-
-    if not nm or not dlltool:
-        print(warning % "nm and/or dlltool were not found")
-        return False
-
-    nm_command = '%s -Cs %s' % (nm, lib_file)
-    dlltool_command = "%s --dllname %s --def %s --output-lib %s" % \
-        (dlltool, dll_file, def_file, mingw_lib)
-    export_match = re.compile(r"^_imp__(.*) in python\d+\.dll").match
-
-    f = open(def_file,'w')
-    f.write("LIBRARY %s\n" % dll_file)
-    f.write("EXPORTS\n")
-
-    nm_pipe = os.popen(nm_command)
-    for line in nm_pipe.readlines():
-        m = export_match(line)
-        if m:
-            f.write(m.group(1)+"\n")
-    f.close()
-    exit = nm_pipe.close()
-
-    if exit:
-        print(warning % "nm did not run successfully")
-        return False
-
-    if os.system(dlltool_command) != 0:
-        print(warning % "dlltool did not run successfully")
-        return False
-
-    return True
-
-# Target files (.def and .a) go in PCBuild directory
-lib_file = os.path.join(srcdir, PCBUILD, "python%s%s.lib" % (major, minor))
-def_file = os.path.join(srcdir, PCBUILD, "python%s%s.def" % (major, minor))
-dll_file = "python%s%s.dll" % (major, minor)
-mingw_lib = os.path.join(srcdir, PCBUILD, "libpython%s%s.a" % (major, minor))
-
-have_mingw = build_mingw_lib(lib_file, def_file, dll_file, mingw_lib)
-
-# Determine the target architecture
-if os.system("nmake /nologo /c /f msisupport.mak") != 0:
-    raise RuntimeError("'nmake /f msisupport.mak' failed")
-dll_path = os.path.join(srcdir, PCBUILD, dll_file)
-msilib.set_arch_from_file(dll_path)
-if msilib.pe_type(dll_path) != msilib.pe_type("msisupport.dll"):
-    raise SystemError("msisupport.dll for incorrect architecture")
-
-if msilib.Win64:
-    upgrade_code = upgrade_code_64
-
-if snapshot:
-    product_code = msilib.gen_uuid()
-else:
-    # official release: generate UUID from the download link that the file will have
-    import uuid
-    product_code = uuid.uuid3(uuid.NAMESPACE_URL,
-                    'http://www.python.org/ftp/python/%s.%s.%s/python-%s%s.msi' %
-                    (major, minor, micro, full_current_version, msilib.arch_ext))
-    product_code = '{%s}' % product_code
-
-if testpackage:
-    ext = 'px'
-    testprefix = 'x'
-else:
-    ext = 'py'
-    testprefix = ''
-
-if msilib.Win64:
-    SystemFolderName = "[System64Folder]"
-    registry_component = 4|256
-else:
-    SystemFolderName = "[SystemFolder]"
-    registry_component = 4
-
-msilib.reset()
-
-# condition in which to install pythonxy.dll in system32:
-# a) it is Windows 9x or
-# b) it is NT, the user is privileged, and has chosen per-machine installation
-sys32cond = "(Windows9x or (Privileged and ALLUSERS))"
-
-def build_database():
-    """Generate an empty database, with just the schema and the
-    Summary information stream."""
-    if snapshot:
-        uc = upgrade_code_snapshot
-    else:
-        uc = upgrade_code
-    if msilib.Win64:
-        productsuffix = " (64-bit)"
-    else:
-        productsuffix = ""
-    # schema represents the installer 2.0 database schema.
-    # sequence is the set of standard sequences
-    # (ui/execute, admin/advt/install)
-    msiname = "python-%s%s.msi" % (full_current_version, msilib.arch_ext)
-    db = msilib.init_database(msiname,
-                  schema, ProductName="Python "+full_current_version+productsuffix,
-                  ProductCode=product_code,
-                  ProductVersion=current_version,
-                  Manufacturer=u"Python Software Foundation",
-                  request_uac = True)
-    # The default sequencing of the RemoveExistingProducts action causes
-    # removal of files that got just installed. Place it after
-    # InstallInitialize, so we first uninstall everything, but still roll
-    # back in case the installation is interrupted
-    msilib.change_sequence(sequence.InstallExecuteSequence,
-                           "RemoveExistingProducts", 1510)
-    msilib.add_tables(db, sequence)
-    # We cannot set ALLUSERS in the property table, as this cannot be
-    # reset if the user choses a per-user installation. Instead, we
-    # maintain WhichUsers, which can be "ALL" or "JUSTME". The UI manages
-    # this property, and when the execution starts, ALLUSERS is set
-    # accordingly.
-    add_data(db, "Property", [("UpgradeCode", uc),
-                              ("WhichUsers", "ALL"),
-                              ("ProductLine", "Python%s%s" % (major, minor)),
-                             ])
-    db.Commit()
-    return db, msiname
-
-def remove_old_versions(db):
-    "Fill the upgrade table."
-    start = "%s.%s.0" % (major, minor)
-    # This requests that feature selection states of an older
-    # installation should be forwarded into this one. Upgrading
-    # requires that both the old and the new installation are
-    # either both per-machine or per-user.
-    migrate_features = 1
-    # See "Upgrade Table". We remove releases with the same major and
-    # minor version. For an snapshot, we remove all earlier snapshots. For
-    # a release, we remove all snapshots, and all earlier releases.
-    if snapshot:
-        add_data(db, "Upgrade",
-            [(upgrade_code_snapshot, start,
-              current_version,
-              None,                     # Ignore language
-              migrate_features,
-              None,                     # Migrate ALL features
-              "REMOVEOLDSNAPSHOT")])
-        props = "REMOVEOLDSNAPSHOT"
-    else:
-        add_data(db, "Upgrade",
-            [(upgrade_code, start, current_version,
-              None, migrate_features, None, "REMOVEOLDVERSION"),
-             (upgrade_code_snapshot, start, "%s.%d.0" % (major, int(minor)+1),
-              None, migrate_features, None, "REMOVEOLDSNAPSHOT")])
-        props = "REMOVEOLDSNAPSHOT;REMOVEOLDVERSION"
-
-    props += ";TARGETDIR;DLLDIR;LAUNCHERDIR"
-    # Installer collects the product codes of the earlier releases in
-    # these properties. In order to allow modification of the properties,
-    # they must be declared as secure. See "SecureCustomProperties Property"
-    add_data(db, "Property", [("SecureCustomProperties", props)])
-
-class PyDialog(Dialog):
-    """Dialog class with a fixed layout: controls at the top, then a ruler,
-    then a list of buttons: back, next, cancel. Optionally a bitmap at the
-    left."""
-    def __init__(self, *args, **kw):
-        """Dialog(database, name, x, y, w, h, attributes, title, first,
-        default, cancel, bitmap=true)"""
-        Dialog.__init__(self, *args)
-        ruler = self.h - 36
-        bmwidth = 152*ruler/328
-        if kw.get("bitmap", True):
-            self.bitmap("Bitmap", 0, 0, bmwidth, ruler, "PythonWin")
-        self.line("BottomLine", 0, ruler, self.w, 0)
-
-    def title(self, title):
-        "Set the title text of the dialog at the top."
-        # name, x, y, w, h, flags=Visible|Enabled|Transparent|NoPrefix,
-        # text, in VerdanaBold10
-        self.text("Title", 135, 10, 220, 60, 0x30003,
-                  r"{\VerdanaBold10}%s" % title)
-
-    def back(self, title, next, name = "Back", active = 1):
-        """Add a back button with a given title, the tab-next button,
-        its name in the Control table, possibly initially disabled.
-
-        Return the button, so that events can be associated"""
-        if active:
-            flags = 3 # Visible|Enabled
-        else:
-            flags = 1 # Visible
-        return self.pushbutton(name, 180, self.h-27 , 56, 17, flags, title, next)
-
-    def cancel(self, title, next, name = "Cancel", active = 1):
-        """Add a cancel button with a given title, the tab-next button,
-        its name in the Control table, possibly initially disabled.
-
-        Return the button, so that events can be associated"""
-        if active:
-            flags = 3 # Visible|Enabled
-        else:
-            flags = 1 # Visible
-        return self.pushbutton(name, 304, self.h-27, 56, 17, flags, title, next)
-
-    def next(self, title, next, name = "Next", active = 1):
-        """Add a Next button with a given title, the tab-next button,
-        its name in the Control table, possibly initially disabled.
-
-        Return the button, so that events can be associated"""
-        if active:
-            flags = 3 # Visible|Enabled
-        else:
-            flags = 1 # Visible
-        return self.pushbutton(name, 236, self.h-27, 56, 17, flags, title, next)
-
-    def xbutton(self, name, title, next, xpos):
-        """Add a button with a given title, the tab-next button,
-        its name in the Control table, giving its x position; the
-        y-position is aligned with the other buttons.
-
-        Return the button, so that events can be associated"""
-        return self.pushbutton(name, int(self.w*xpos - 28), self.h-27, 56, 17, 3, title, next)
-
-def add_ui(db):
-    x = y = 50
-    w = 370
-    h = 300
-    title = "[ProductName] Setup"
-
-    # see "Dialog Style Bits"
-    modal = 3      # visible | modal
-    modeless = 1   # visible
-    track_disk_space = 32
-
-    add_data(db, 'ActionText', uisample.ActionText)
-    add_data(db, 'UIText', uisample.UIText)
-
-    # Bitmaps
-    if not os.path.exists(srcdir+r"\PC\python_icon.exe"):
-        raise RuntimeError("Run icons.mak in PC directory")
-    add_data(db, "Binary",
-             [("PythonWin", msilib.Binary(r"%s\PCbuild\installer.bmp" % srcdir)), # 152x328 pixels
-              ("py.ico",msilib.Binary(srcdir+r"\PC\py.ico")),
-             ])
-    add_data(db, "Icon",
-             [("python_icon.exe", msilib.Binary(srcdir+r"\PC\python_icon.exe"))])
-
-    # Scripts
-    # CheckDir sets TargetExists if TARGETDIR exists.
-    # UpdateEditIDLE sets the REGISTRY.tcl component into
-    # the installed/uninstalled state according to both the
-    # Extensions and TclTk features.
-    add_data(db, "Binary", [("Script", msilib.Binary("msisupport.dll"))])
-    # See "Custom Action Type 1"
-    if msilib.Win64:
-        CheckDir = "CheckDir"
-        UpdateEditIDLE = "UpdateEditIDLE"
-    else:
-        CheckDir =  "_CheckDir at 4"
-        UpdateEditIDLE = "_UpdateEditIDLE at 4"
-    add_data(db, "CustomAction",
-        [("CheckDir", 1, "Script", CheckDir)])
-    if have_tcl:
-        add_data(db, "CustomAction",
-        [("UpdateEditIDLE", 1, "Script", UpdateEditIDLE)])
-
-    # UI customization properties
-    add_data(db, "Property",
-             # See "DefaultUIFont Property"
-             [("DefaultUIFont", "DlgFont8"),
-              # See "ErrorDialog Style Bit"
-              ("ErrorDialog", "ErrorDlg"),
-              ("Progress1", "Install"),   # modified in maintenance type dlg
-              ("Progress2", "installs"),
-              ("MaintenanceForm_Action", "Repair")])
-
-    # Fonts, see "TextStyle Table"
-    add_data(db, "TextStyle",
-             [("DlgFont8", "Tahoma", 9, None, 0),
-              ("DlgFontBold8", "Tahoma", 8, None, 1), #bold
-              ("VerdanaBold10", "Verdana", 10, None, 1),
-              ("VerdanaRed9", "Verdana", 9, 255, 0),
-             ])
-
-    compileargs = r'-Wi "[TARGETDIR]Lib\compileall.py" -f -x "bad_coding|badsyntax|site-packages|py2_|lib2to3\\tests|venv\\scripts" "[TARGETDIR]Lib"'
-    lib2to3args = r'-c "import lib2to3.pygram, lib2to3.patcomp;lib2to3.patcomp.PatternCompiler()"'
-    updatepipargs = r'-m ensurepip -U --default-pip'
-    removepipargs = r'-B -m ensurepip._uninstall'
-    # See "CustomAction Table"
-    add_data(db, "CustomAction", [
-        # msidbCustomActionTypeFirstSequence + msidbCustomActionTypeTextData + msidbCustomActionTypeProperty
-        # See "Custom Action Type 51",
-        # "Custom Action Execution Scheduling Options"
-        ("InitialTargetDir", 307, "TARGETDIR",
-         "[WindowsVolume]Python%s%s" % (major, minor)),
-        ("SetDLLDirToTarget", 307, "DLLDIR", "[TARGETDIR]"),
-        ("SetDLLDirToSystem32", 307, "DLLDIR", SystemFolderName),
-        ("SetLauncherDirToTarget", 307, "LAUNCHERDIR", "[TARGETDIR]"),
-        ("SetLauncherDirToWindows", 307, "LAUNCHERDIR", "[WindowsFolder]"),
-        # msidbCustomActionTypeExe + msidbCustomActionTypeSourceFile
-        # See "Custom Action Type 18"
-        # msidbCustomActionTypeInScript (1024); run during actual installation
-        # msidbCustomActionTypeNoImpersonate (2048); run action in system account, not user account
-        ("CompilePyc", 18+1024+2048, "python.exe", compileargs),
-        ("CompilePyo", 18+1024+2048, "python.exe", "-O "+compileargs),
-        ("CompileGrammar", 18+1024+2048, "python.exe", lib2to3args),
-        ("UpdatePip", 18+1024+2048, "python.exe", updatepipargs),
-        ("RemovePip", 18+1024+2048, "python.exe", removepipargs),
-        ])
-
-    # UI Sequences, see "InstallUISequence Table", "Using a Sequence Table"
-    # Numbers indicate sequence; see sequence.py for how these action integrate
-    add_data(db, "InstallUISequence",
-             [("PrepareDlg", "Not Privileged or Windows9x or Installed", 140),
-              ("WhichUsersDlg", "Privileged and not Windows9x and not Installed", 141),
-              ("InitialTargetDir", 'TARGETDIR=""', 750),
-              # In the user interface, assume all-users installation if privileged.
-              ("SetDLLDirToSystem32", 'DLLDIR="" and ' + sys32cond, 751),
-              ("SetDLLDirToTarget", 'DLLDIR="" and not ' + sys32cond, 752),
-              ("SetLauncherDirToWindows", 'LAUNCHERDIR="" and ' + sys32cond, 753),
-              ("SetLauncherDirToTarget", 'LAUNCHERDIR="" and not ' + sys32cond, 754),
-              ("SelectDirectoryDlg", "Not Installed", 1230),
-              # XXX no support for resume installations yet
-              #("ResumeDlg", "Installed AND (RESUME OR Preselected)", 1240),
-              ("MaintenanceTypeDlg", "Installed AND NOT RESUME AND NOT Preselected", 1250),
-              ("ProgressDlg", None, 1280)])
-    add_data(db, "AdminUISequence",
-             [("InitialTargetDir", 'TARGETDIR=""', 750),
-              ("SetDLLDirToTarget", 'DLLDIR=""', 751),
-              ("SetLauncherDirToTarget", 'LAUNCHERDIR=""', 752),
-             ])
-
-    # Prepend TARGETDIR to the system path, and remove it on uninstall.
-    add_data(db, "Environment",
-             [("PathAddition", "=-*Path", "[TARGETDIR];[TARGETDIR]Scripts;[~]", "REGISTRY.path")])
-
-    # Execute Sequences
-    add_data(db, "InstallExecuteSequence",
-            [("InitialTargetDir", 'TARGETDIR=""', 750),
-             ("SetDLLDirToSystem32", 'DLLDIR="" and ' + sys32cond, 751),
-             ("SetDLLDirToTarget", 'DLLDIR="" and not ' + sys32cond, 752),
-             ("SetLauncherDirToWindows", 'LAUNCHERDIR="" and ' + sys32cond, 753),
-             ("SetLauncherDirToTarget", 'LAUNCHERDIR="" and not ' + sys32cond, 754),
-             ("UpdateEditIDLE", None, 1050),
-             # run command if install state of pip changes to INSTALLSTATE_LOCAL
-             # run after InstallFiles
-             ("UpdatePip", "&pip_feature=3", 4001),
-             # remove pip when state changes to INSTALLSTATE_ABSENT
-             # run before RemoveFiles
-             ("RemovePip", "&pip_feature=2", 3499),
-             ("CompilePyc", "COMPILEALL", 4002),
-             ("CompilePyo", "COMPILEALL", 4003),
-             ("CompileGrammar", "COMPILEALL", 4004),
-            ])
-    add_data(db, "AdminExecuteSequence",
-            [("InitialTargetDir", 'TARGETDIR=""', 750),
-             ("SetDLLDirToTarget", 'DLLDIR=""', 751),
-             ("SetLauncherDirToTarget", 'LAUNCHERDIR=""', 752),
-            ])
-
-    #####################################################################
-    # Standard dialogs: FatalError, UserExit, ExitDialog
-    fatal=PyDialog(db, "FatalError", x, y, w, h, modal, title,
-                 "Finish", "Finish", "Finish")
-    fatal.title("[ProductName] Installer ended prematurely")
-    fatal.back("< Back", "Finish", active = 0)
-    fatal.cancel("Cancel", "Back", active = 0)
-    fatal.text("Description1", 135, 70, 220, 80, 0x30003,
-               "[ProductName] setup ended prematurely because of an error.  Your system has not been modified.  To install this program at a later time, please run the installation again.")
-    fatal.text("Description2", 135, 155, 220, 20, 0x30003,
-               "Click the Finish button to exit the Installer.")
-    c=fatal.next("Finish", "Cancel", name="Finish")
-    # See "ControlEvent Table". Parameters are the event, the parameter
-    # to the action, and optionally the condition for the event, and the order
-    # of events.
-    c.event("EndDialog", "Exit")
-
-    user_exit=PyDialog(db, "UserExit", x, y, w, h, modal, title,
-                 "Finish", "Finish", "Finish")
-    user_exit.title("[ProductName] Installer was interrupted")
-    user_exit.back("< Back", "Finish", active = 0)
-    user_exit.cancel("Cancel", "Back", active = 0)
-    user_exit.text("Description1", 135, 70, 220, 80, 0x30003,
-               "[ProductName] setup was interrupted.  Your system has not been modified.  "
-               "To install this program at a later time, please run the installation again.")
-    user_exit.text("Description2", 135, 155, 220, 20, 0x30003,
-               "Click the Finish button to exit the Installer.")
-    c = user_exit.next("Finish", "Cancel", name="Finish")
-    c.event("EndDialog", "Exit")
-
-    exit_dialog = PyDialog(db, "ExitDialog", x, y, w, h, modal, title,
-                         "Finish", "Finish", "Finish")
-    exit_dialog.title("Complete the [ProductName] Installer")
-    exit_dialog.back("< Back", "Finish", active = 0)
-    exit_dialog.cancel("Cancel", "Back", active = 0)
-    exit_dialog.text("Acknowledgements", 135, 95, 220, 120, 0x30003,
-      "Special Windows thanks to:\n"
-      "    Mark Hammond, without whose years of freely \n"
-      "    shared Windows expertise, Python for Windows \n"
-      "    would still be Python for DOS.")
-
-    c = exit_dialog.text("warning", 135, 200, 220, 40, 0x30003,
-            "{\\VerdanaRed9}Warning: Python 2.5.x is the last "
-            "Python release for Windows 9x.")
-    c.condition("Hide", "NOT Version9X")
-
-    exit_dialog.text("Description", 135, 235, 220, 20, 0x30003,
-               "Click the Finish button to exit the Installer.")
-    c = exit_dialog.next("Finish", "Cancel", name="Finish")
-    c.event("EndDialog", "Return")
-
-    #####################################################################
-    # Required dialog: FilesInUse, ErrorDlg
-    inuse = PyDialog(db, "FilesInUse",
-                     x, y, w, h,
-                     19,                # KeepModeless|Modal|Visible
-                     title,
-                     "Retry", "Retry", "Retry", bitmap=False)
-    inuse.text("Title", 15, 6, 200, 15, 0x30003,
-               r"{\DlgFontBold8}Files in Use")
-    inuse.text("Description", 20, 23, 280, 20, 0x30003,
-               "Some files that need to be updated are currently in use.")
-    inuse.text("Text", 20, 55, 330, 50, 3,
-               "The following applications are using files that need to be updated by this setup. Close these applications and then click Retry to continue the installation or Cancel to exit it.")
-    inuse.control("List", "ListBox", 20, 107, 330, 130, 7, "FileInUseProcess",
-                  None, None, None)
-    c=inuse.back("Exit", "Ignore", name="Exit")
-    c.event("EndDialog", "Exit")
-    c=inuse.next("Ignore", "Retry", name="Ignore")
-    c.event("EndDialog", "Ignore")
-    c=inuse.cancel("Retry", "Exit", name="Retry")
-    c.event("EndDialog","Retry")
-
-
-    # See "Error Dialog". See "ICE20" for the required names of the controls.
-    error = Dialog(db, "ErrorDlg",
-                   50, 10, 330, 101,
-                   65543,       # Error|Minimize|Modal|Visible
-                   title,
-                   "ErrorText", None, None)
-    error.text("ErrorText", 50,9,280,48,3, "")
-    error.control("ErrorIcon", "Icon", 15, 9, 24, 24, 5242881, None, "py.ico", None, None)
-    error.pushbutton("N",120,72,81,21,3,"No",None).event("EndDialog","ErrorNo")
-    error.pushbutton("Y",240,72,81,21,3,"Yes",None).event("EndDialog","ErrorYes")
-    error.pushbutton("A",0,72,81,21,3,"Abort",None).event("EndDialog","ErrorAbort")
-    error.pushbutton("C",42,72,81,21,3,"Cancel",None).event("EndDialog","ErrorCancel")
-    error.pushbutton("I",81,72,81,21,3,"Ignore",None).event("EndDialog","ErrorIgnore")
-    error.pushbutton("O",159,72,81,21,3,"Ok",None).event("EndDialog","ErrorOk")
-    error.pushbutton("R",198,72,81,21,3,"Retry",None).event("EndDialog","ErrorRetry")
-
-    #####################################################################
-    # Global "Query Cancel" dialog
-    cancel = Dialog(db, "CancelDlg", 50, 10, 260, 85, 3, title,
-                    "No", "No", "No")
-    cancel.text("Text", 48, 15, 194, 30, 3,
-                "Are you sure you want to cancel [ProductName] installation?")
-    cancel.control("Icon", "Icon", 15, 15, 24, 24, 5242881, None,
-                   "py.ico", None, None)
-    c=cancel.pushbutton("Yes", 72, 57, 56, 17, 3, "Yes", "No")
-    c.event("EndDialog", "Exit")
-
-    c=cancel.pushbutton("No", 132, 57, 56, 17, 3, "No", "Yes")
-    c.event("EndDialog", "Return")
-
-    #####################################################################
-    # Global "Wait for costing" dialog
-    costing = Dialog(db, "WaitForCostingDlg", 50, 10, 260, 85, modal, title,
-                     "Return", "Return", "Return")
-    costing.text("Text", 48, 15, 194, 30, 3,
-                 "Please wait while the installer finishes determining your disk space requirements.")
-    costing.control("Icon", "Icon", 15, 15, 24, 24, 5242881, None,
-                    "py.ico", None, None)
-    c = costing.pushbutton("Return", 102, 57, 56, 17, 3, "Return", None)
-    c.event("EndDialog", "Exit")
-
-    #####################################################################
-    # Preparation dialog: no user input except cancellation
-    prep = PyDialog(db, "PrepareDlg", x, y, w, h, modeless, title,
-                    "Cancel", "Cancel", "Cancel")
-    prep.text("Description", 135, 70, 220, 40, 0x30003,
-              "Please wait while the Installer prepares to guide you through the installation.")
-    prep.title("Welcome to the [ProductName] Installer")
-    c=prep.text("ActionText", 135, 110, 220, 20, 0x30003, "Pondering...")
-    c.mapping("ActionText", "Text")
-    c=prep.text("ActionData", 135, 135, 220, 30, 0x30003, None)
-    c.mapping("ActionData", "Text")
-    prep.back("Back", None, active=0)
-    prep.next("Next", None, active=0)
-    c=prep.cancel("Cancel", None)
-    c.event("SpawnDialog", "CancelDlg")
-
-    #####################################################################
-    # Target directory selection
-    seldlg = PyDialog(db, "SelectDirectoryDlg", x, y, w, h, modal, title,
-                    "Next", "Next", "Cancel")
-    seldlg.title("Select Destination Directory")
-    c = seldlg.text("Existing", 135, 25, 235, 30, 0x30003,
-                    "{\VerdanaRed9}This update will replace your existing [ProductLine] installation.")
-    c.condition("Hide", 'REMOVEOLDVERSION="" and REMOVEOLDSNAPSHOT=""')
-    seldlg.text("Description", 135, 50, 220, 40, 0x30003,
-               "Please select a directory for the [ProductName] files.")
-
-    seldlg.back("< Back", None, active=0)
-    c = seldlg.next("Next >", "Cancel")
-    c.event("DoAction", "CheckDir", "TargetExistsOk<>1", order=1)
-    # If the target exists, but we found that we are going to remove old versions, don't bother
-    # confirming that the target directory exists. Strictly speaking, we should determine that
-    # the target directory is indeed the target of the product that we are going to remove, but
-    # I don't know how to do that.
-    c.event("SpawnDialog", "ExistingDirectoryDlg", 'TargetExists=1 and REMOVEOLDVERSION="" and REMOVEOLDSNAPSHOT=""', 2)
-    c.event("SetTargetPath", "TARGETDIR", 'TargetExists=0 or REMOVEOLDVERSION<>"" or REMOVEOLDSNAPSHOT<>""', 3)
-    c.event("SpawnWaitDialog", "WaitForCostingDlg", "CostingComplete=1", 4)
-    c.event("NewDialog", "SelectFeaturesDlg", 'TargetExists=0 or REMOVEOLDVERSION<>"" or REMOVEOLDSNAPSHOT<>""', 5)
-
-    c = seldlg.cancel("Cancel", "DirectoryCombo")
-    c.event("SpawnDialog", "CancelDlg")
-
-    seldlg.control("DirectoryCombo", "DirectoryCombo", 135, 70, 172, 80, 393219,
-                   "TARGETDIR", None, "DirectoryList", None)
-    seldlg.control("DirectoryList", "DirectoryList", 135, 90, 208, 136, 3, "TARGETDIR",
-                   None, "PathEdit", None)
-    seldlg.control("PathEdit", "PathEdit", 135, 230, 206, 16, 3, "TARGETDIR", None, "Next", None)
-    c = seldlg.pushbutton("Up", 306, 70, 18, 18, 3, "Up", None)
-    c.event("DirectoryListUp", "0")
-    c = seldlg.pushbutton("NewDir", 324, 70, 30, 18, 3, "New", None)
-    c.event("DirectoryListNew", "0")
-
-    #####################################################################
-    # SelectFeaturesDlg
-    features = PyDialog(db, "SelectFeaturesDlg", x, y, w, h, modal|track_disk_space,
-                        title, "Tree", "Next", "Cancel")
-    features.title("Customize [ProductName]")
-    features.text("Description", 135, 35, 220, 15, 0x30003,
-                  "Select the way you want features to be installed.")
-    features.text("Text", 135,45,220,30, 3,
-                  "Click on the icons in the tree below to change the way features will be installed.")
-
-    c=features.back("< Back", "Next")
-    c.event("NewDialog", "SelectDirectoryDlg")
-
-    c=features.next("Next >", "Cancel")
-    c.mapping("SelectionNoItems", "Enabled")
-    c.event("SpawnDialog", "DiskCostDlg", "OutOfDiskSpace=1", order=1)
-    c.event("EndDialog", "Return", "OutOfDiskSpace<>1", order=2)
-
-    c=features.cancel("Cancel", "Tree")
-    c.event("SpawnDialog", "CancelDlg")
-
-    # The browse property is not used, since we have only a single target path (selected already)
-    features.control("Tree", "SelectionTree", 135, 75, 220, 95, 7, "_BrowseProperty",
-                     "Tree of selections", "Back", None)
-
-    #c=features.pushbutton("Reset", 42, 243, 56, 17, 3, "Reset", "DiskCost")
-    #c.mapping("SelectionNoItems", "Enabled")
-    #c.event("Reset", "0")
-
-    features.control("Box", "GroupBox", 135, 170, 225, 90, 1, None, None, None, None)
-
-    c=features.xbutton("DiskCost", "Disk &Usage", None, 0.10)
-    c.mapping("SelectionNoItems","Enabled")
-    c.event("SpawnDialog", "DiskCostDlg")
-
-    c=features.xbutton("Advanced", "Advanced", None, 0.30)
-    c.event("SpawnDialog", "AdvancedDlg")
-
-    c=features.text("ItemDescription", 140, 180, 210, 40, 3,
-                  "Multiline description of the currently selected item.")
-    c.mapping("SelectionDescription","Text")
-
-    c=features.text("ItemSize", 140, 225, 210, 33, 3,
-                    "The size of the currently selected item.")
-    c.mapping("SelectionSize", "Text")
-
-    #####################################################################
-    # Disk cost
-    cost = PyDialog(db, "DiskCostDlg", x, y, w, h, modal, title,
-                    "OK", "OK", "OK", bitmap=False)
-    cost.text("Title", 15, 6, 200, 15, 0x30003,
-              "{\DlgFontBold8}Disk Space Requirements")
-    cost.text("Description", 20, 20, 280, 20, 0x30003,
-              "The disk space required for the installation of the selected features.")
-    cost.text("Text", 20, 53, 330, 60, 3,
-              "The highlighted volumes (if any) do not have enough disk space "
-              "available for the currently selected features.  You can either "
-              "remove some files from the highlighted volumes, or choose to "
-              "install less features onto local drive(s), or select different "
-              "destination drive(s).")
-    cost.control("VolumeList", "VolumeCostList", 20, 100, 330, 150, 393223,
-                 None, "{120}{70}{70}{70}{70}", None, None)
-    cost.xbutton("OK", "Ok", None, 0.5).event("EndDialog", "Return")
-
-    #####################################################################
-    # WhichUsers Dialog. Only available on NT, and for privileged users.
-    # This must be run before FindRelatedProducts, because that will
-    # take into account whether the previous installation was per-user
-    # or per-machine. We currently don't support going back to this
-    # dialog after "Next" was selected; to support this, we would need to
-    # find how to reset the ALLUSERS property, and how to re-run
-    # FindRelatedProducts.
-    # On Windows9x, the ALLUSERS property is ignored on the command line
-    # and in the Property table, but installer fails according to the documentation
-    # if a dialog attempts to set ALLUSERS.
-    whichusers = PyDialog(db, "WhichUsersDlg", x, y, w, h, modal, title,
-                        "AdminInstall", "Next", "Cancel")
-    whichusers.title("Select whether to install [ProductName] for all users of this computer.")
-    # A radio group with two options: allusers, justme
-    g = whichusers.radiogroup("AdminInstall", 135, 60, 235, 80, 3,
-                              "WhichUsers", "", "Next")
-    g.condition("Disable", "VersionNT=600") # Not available on Vista and Windows 2008
-    g.add("ALL", 0, 5, 150, 20, "Install for all users")
-    g.add("JUSTME", 0, 25, 235, 20, "Install just for me (not available on Windows Vista)")
-
-    whichusers.back("Back", None, active=0)
-
-    c = whichusers.next("Next >", "Cancel")
-    c.event("[ALLUSERS]", "1", 'WhichUsers="ALL"', 1)
-    c.event("EndDialog", "Return", order = 2)
-
-    c = whichusers.cancel("Cancel", "AdminInstall")
-    c.event("SpawnDialog", "CancelDlg")
-
-    #####################################################################
-    # Advanced Dialog.
-    advanced = PyDialog(db, "AdvancedDlg", x, y, w, h, modal, title,
-                        "CompilePyc", "Ok", "Ok")
-    advanced.title("Advanced Options for [ProductName]")
-
-    # A checkbox whether to build pyc files
-    advanced.checkbox("CompilePyc", 135, 60, 230, 50, 3,
-                      "COMPILEALL", "Compile .py files to byte code after installation", "Ok")
-
-    c = advanced.cancel("Ok", "CompilePyc", name="Ok") # Button just has location of cancel button.
-    c.event("EndDialog", "Return")
-
-    #####################################################################
-    # Existing Directory dialog
-    dlg = Dialog(db, "ExistingDirectoryDlg", 50, 30, 200, 80, modal, title,
-                   "No", "No", "No")
-    dlg.text("Title", 10, 20, 180, 40, 3,
-             "[TARGETDIR] exists. Are you sure you want to overwrite existing files?")
-    c=dlg.pushbutton("Yes", 30, 60, 55, 17, 3, "Yes", "No")
-    c.event("[TargetExists]", "0", order=1)
-    c.event("[TargetExistsOk]", "1", order=2)
-    c.event("EndDialog", "Return", order=3)
-    c=dlg.pushbutton("No", 115, 60, 55, 17, 3, "No", "Yes")
-    c.event("EndDialog", "Return")
-
-    #####################################################################
-    # Installation Progress dialog (modeless)
-    progress = PyDialog(db, "ProgressDlg", x, y, w, h, modeless, title,
-                        "Cancel", "Cancel", "Cancel", bitmap=False)
-    progress.text("Title", 20, 15, 200, 15, 0x30003,
-                  "{\DlgFontBold8}[Progress1] [ProductName]")
-    progress.text("Text", 35, 65, 300, 30, 3,
-                  "Please wait while the Installer [Progress2] [ProductName]. "
-                  "This may take several minutes.")
-    progress.text("StatusLabel", 35, 100, 35, 20, 3, "Status:")
-
-    c=progress.text("ActionText", 70, 100, w-70, 20, 3, "Pondering...")
-    c.mapping("ActionText", "Text")
-
-    #c=progress.text("ActionData", 35, 140, 300, 20, 3, None)
-    #c.mapping("ActionData", "Text")
-
-    c=progress.control("ProgressBar", "ProgressBar", 35, 120, 300, 10, 65537,
-                       None, "Progress done", None, None)
-    c.mapping("SetProgress", "Progress")
-
-    progress.back("< Back", "Next", active=False)
-    progress.next("Next >", "Cancel", active=False)
-    progress.cancel("Cancel", "Back").event("SpawnDialog", "CancelDlg")
-
-    # Maintenance type: repair/uninstall
-    maint = PyDialog(db, "MaintenanceTypeDlg", x, y, w, h, modal, title,
-                     "Next", "Next", "Cancel")
-    maint.title("Welcome to the [ProductName] Setup Wizard")
-    maint.text("BodyText", 135, 63, 230, 42, 3,
-               "Select whether you want to repair or remove [ProductName].")
-    g=maint.radiogroup("RepairRadioGroup", 135, 108, 230, 60, 3,
-                        "MaintenanceForm_Action", "", "Next")
-    g.add("Change", 0, 0, 200, 17, "&Change [ProductName]")
-    g.add("Repair", 0, 18, 200, 17, "&Repair [ProductName]")
-    g.add("Remove", 0, 36, 200, 17, "Re&move [ProductName]")
-
-    maint.back("< Back", None, active=False)
-    c=maint.next("Finish", "Cancel")
-    # Change installation: Change progress dialog to "Change", then ask
-    # for feature selection
-    c.event("[Progress1]", "Change", 'MaintenanceForm_Action="Change"', 1)
-    c.event("[Progress2]", "changes", 'MaintenanceForm_Action="Change"', 2)
-
-    # Reinstall: Change progress dialog to "Repair", then invoke reinstall
-    # Also set list of reinstalled features to "ALL"
-    c.event("[REINSTALL]", "ALL", 'MaintenanceForm_Action="Repair"', 5)
-    c.event("[Progress1]", "Repairing", 'MaintenanceForm_Action="Repair"', 6)
-    c.event("[Progress2]", "repairs", 'MaintenanceForm_Action="Repair"', 7)
-    c.event("Reinstall", "ALL", 'MaintenanceForm_Action="Repair"', 8)
-
-    # Uninstall: Change progress to "Remove", then invoke uninstall
-    # Also set list of removed features to "ALL"
-    c.event("[REMOVE]", "ALL", 'MaintenanceForm_Action="Remove"', 11)
-    c.event("[Progress1]", "Removing", 'MaintenanceForm_Action="Remove"', 12)
-    c.event("[Progress2]", "removes", 'MaintenanceForm_Action="Remove"', 13)
-    c.event("Remove", "ALL", 'MaintenanceForm_Action="Remove"', 14)
-
-    # Close dialog when maintenance action scheduled
-    c.event("EndDialog", "Return", 'MaintenanceForm_Action<>"Change"', 20)
-    c.event("NewDialog", "SelectFeaturesDlg", 'MaintenanceForm_Action="Change"', 21)
-
-    maint.cancel("Cancel", "RepairRadioGroup").event("SpawnDialog", "CancelDlg")
-
-
-# See "Feature Table". The feature level is 1 for all features,
-# and the feature attributes are 0 for the DefaultFeature, and
-# FollowParent for all other features. The numbers are the Display
-# column.
-def add_features(db):
-    # feature attributes:
-    # msidbFeatureAttributesFollowParent == 2
-    # msidbFeatureAttributesDisallowAdvertise == 8
-    # Features that need to be installed with together with the main feature
-    # (i.e. additional Python libraries) need to follow the parent feature.
-    # Features that have no advertisement trigger (e.g. the test suite)
-    # must not support advertisement
-    global default_feature, tcltk, htmlfiles, tools, testsuite
-    global ext_feature, private_crt, prepend_path, pip_feature
-    default_feature = Feature(db, "DefaultFeature", "Python",
-                              "Python Interpreter and Libraries",
-                              1, directory = "TARGETDIR")
-    shared_crt = Feature(db, "SharedCRT", "MSVCRT", "C Run-Time (system-wide)", 0,
-                         level=0)
-    private_crt = Feature(db, "PrivateCRT", "MSVCRT", "C Run-Time (private)", 0,
-                          level=0)
-    add_data(db, "Condition", [("SharedCRT", 1, sys32cond),
-                               ("PrivateCRT", 1, "not "+sys32cond)])
-    # We don't support advertisement of extensions
-    ext_feature = Feature(db, "Extensions", "Register Extensions",
-                          "Make this Python installation the default Python installation", 3,
-                         parent = default_feature, attributes=2|8)
-    if have_tcl:
-        tcltk = Feature(db, "TclTk", "Tcl/Tk", "Tkinter, IDLE, pydoc", 5,
-                    parent = default_feature, attributes=2)
-    htmlfiles = Feature(db, "Documentation", "Documentation",
-                        "Python HTMLHelp File", 7, parent = default_feature)
-    tools = Feature(db, "Tools", "Utility Scripts",
-                    "Python utility scripts (Tools/)", 9,
-                    parent = default_feature, attributes=2)
-    # pip installation isn't enabled by default until a clean uninstall procedure
-    # becomes possible
-    pip_feature = Feature(db, "pip_feature", "pip",
-                    "Install (or upgrade from an earlier version) pip, "
-                    "a tool for installing and managing Python packages.", 11,
-                    parent = default_feature, attributes=2|8)
-    testsuite = Feature(db, "Testsuite", "Test suite",
-                        "Python test suite (Lib/test/)", 13,
-                        parent = default_feature, attributes=2|8)
-    # prepend_path is an additional feature which is to be off by default.
-    # Since the default level for the above features is 1, this needs to be
-    # at least level higher.
-    prepend_path = Feature(db, "PrependPath", "Add python.exe to Path",
-                        "Prepend [TARGETDIR] to the system Path variable. "
-                        "This allows you to type 'python' into a command "
-                        "prompt without needing the full path.", 15,
-                        parent = default_feature, attributes=2|8,
-                        level=2)
-
-def extract_msvcr100():
-    # Find the redistributable files
-    if msilib.Win64:
-        arch = "x64"
-    else:
-        arch = "x86"
-    dir = os.path.join(os.environ['VS100COMNTOOLS'], r"..\..\VC\redist\%s\Microsoft.VC100.CRT" % arch)
-
-    result = []
-    installer = msilib.MakeInstaller()
-    # At least for VS2010, manifests are no longer provided
-    name = "msvcr100.dll"
-    path = os.path.join(dir, name)
-    kw = {'src':path}
-    kw['version'] = installer.FileVersion(path, 0)
-    kw['language'] = installer.FileVersion(path, 1)
-    return name, kw
-
-def generate_license():
-    import shutil, glob
-    out = open("LICENSE.txt", "w")
-    shutil.copyfileobj(open(os.path.join(srcdir, "LICENSE")), out)
-    shutil.copyfileobj(open("crtlicense.txt"), out)
-    for name, pat, file in (("bzip2","bzip2-*", "LICENSE"),
-                      ("openssl", "openssl-*", "LICENSE"),
-                      ("Tcl", "tcl-8*", "license.terms"),
-                      ("Tk", "tk-8*", "license.terms"),
-                      ("Tix", "tix-*", "license.terms")):
-        out.write("\nThis copy of Python includes a copy of %s, which is licensed under the following terms:\n\n" % name)
-        dirs = glob.glob(srcdir+"/../"+pat)
-        if not dirs:
-            raise ValueError, "Could not find "+srcdir+"/../"+pat
-        if len(dirs) > 2 and not snapshot:
-            raise ValueError, "Multiple copies of "+pat
-        dir = dirs[0]
-        shutil.copyfileobj(open(os.path.join(dir, file)), out)
-    out.close()
-
-
-class PyDirectory(Directory):
-    """By default, all components in the Python installer
-    can run from source."""
-    def __init__(self, *args, **kw):
-        if "componentflags" not in kw:
-            kw['componentflags'] = 2 #msidbComponentAttributesOptional
-        Directory.__init__(self, *args, **kw)
-
-def hgmanifest():
-    # Fetch file list from Mercurial
-    process = subprocess.Popen(['hg', 'manifest'], stdout=subprocess.PIPE)
-    stdout, stderr = process.communicate()
-    # Create nested directories for file tree
-    result = {}
-    for line in stdout.splitlines():
-        components = line.split('/')
-        d = result
-        while len(components) > 1:
-            d1 = d.setdefault(components[0], {})
-            d = d1
-            del components[0]
-        d[components[0]] = None
-    return result
-
-
-# See "File Table", "Component Table", "Directory Table",
-# "FeatureComponents Table"
-def add_files(db):
-    installer = msilib.MakeInstaller()
-    hgfiles = hgmanifest()
-    cab = CAB("python")
-    tmpfiles = []
-    # Add all executables, icons, text files into the TARGETDIR component
-    root = PyDirectory(db, cab, None, srcdir, "TARGETDIR", "SourceDir")
-    default_feature.set_current()
-    root.add_file("README.txt", src="README")
-    root.add_file("NEWS.txt", src="Misc/NEWS")
-    generate_license()
-    root.add_file("LICENSE.txt", src=os.path.abspath("LICENSE.txt"))
-    root.start_component("python.exe", keyfile="python.exe")
-    root.add_file("%s/python.exe" % PCBUILD)
-    root.start_component("pythonw.exe", keyfile="pythonw.exe")
-    root.add_file("%s/pythonw.exe" % PCBUILD)
-
-    # msidbComponentAttributesSharedDllRefCount = 8, see "Component Table"
-    dlldir = PyDirectory(db, cab, root, srcdir, "DLLDIR", ".")
-    launcherdir = PyDirectory(db, cab, root, srcdir, "LAUNCHERDIR", ".")
-
-    # msidbComponentAttributes64bit = 256; this disables registry redirection
-    # to allow setting the SharedDLLs key in the 64-bit portion even for a
-    # 32-bit installer.
-    # XXX does this still allow to install the component on a 32-bit system?
-    # Pick up 32-bit binary always
-    launchersrc = PCBUILD
-    if launchersrc.lower() == 'pcbuild\\x64-pgo':
-        launchersrc = 'PCBuild\\win32-pgo'
-    if launchersrc.lower() == 'pcbuild\\amd64':
-        launchersrc = 'PCBuild'
-    launcher = os.path.join(srcdir, launchersrc, "py.exe")
-    launcherdir.start_component("launcher", flags = 8+256, keyfile="py.exe")
-    launcherdir.add_file(launcher,
-                         version=installer.FileVersion(launcher, 0),
-                         language=installer.FileVersion(launcher, 1))
-    launcherw = os.path.join(srcdir, launchersrc, "pyw.exe")
-    launcherdir.start_component("launcherw", flags = 8+256, keyfile="pyw.exe")
-    launcherdir.add_file(launcherw,
-                         version=installer.FileVersion(launcherw, 0),
-                         language=installer.FileVersion(launcherw, 1))
-
-    pydll = "python%s%s.dll" % (major, minor)
-    pydllsrc = os.path.join(srcdir, PCBUILD, pydll)
-    dlldir.start_component("DLLDIR", flags = 8, keyfile = pydll, uuid = pythondll_uuid)
-    pyversion = installer.FileVersion(pydllsrc, 0)
-    if not snapshot:
-        # For releases, the Python DLL has the same version as the
-        # installer package.
-        assert pyversion.split(".")[:3] == current_version.split(".")
-    dlldir.add_file("%s/python%s%s.dll" % (PCBUILD, major, minor),
-                    version=pyversion,
-                    language=installer.FileVersion(pydllsrc, 1))
-    DLLs = PyDirectory(db, cab, root, srcdir + "/" + PCBUILD, "DLLs", "DLLS|DLLs")
-
-    # msvcr90.dll: Need to place the DLL and the manifest into the root directory,
-    # plus another copy of the manifest in the DLLs directory, with the manifest
-    # pointing to the root directory
-    root.start_component("msvcr90", feature=private_crt)
-    # Results are ID,keyword pairs
-    crtdll, kwds = extract_msvcr100()
-    root.add_file(crtdll, **kwds)
-    # Copy the manifest
-    # Actually, don't do that anymore - no DLL in DLLs should have a manifest
-    # dependency on msvcr90.dll anymore, so this should not be necessary
-    #manifest_dlls = manifest[0]+".root"
-    #open(manifest_dlls, "w").write(open(manifest[1]['src']).read().replace("msvcr","../msvcr"))
-    #DLLs.start_component("msvcr90_dlls", feature=private_crt)
-    #DLLs.add_file(manifest[0], src=os.path.abspath(manifest_dlls))
-
-    # Now start the main component for the DLLs directory;
-    # no regular files have been added to the directory yet.
-    DLLs.start_component()
-
-    # Check if _ctypes.pyd exists
-    have_ctypes = os.path.exists(srcdir+"/%s/_ctypes.pyd" % PCBUILD)
-    if not have_ctypes:
-        print("WARNING: _ctypes.pyd not found, ctypes will not be included")
-        extensions.remove("_ctypes.pyd")
-
-    # Add all .py files in Lib, except tkinter, test
-    dirs = []
-    pydirs = [(root, "Lib", hgfiles["Lib"], default_feature)]
-    while pydirs:
-        # Commit every now and then, or else installer will complain
-        db.Commit()
-        parent, dir, files, feature = pydirs.pop()
-        if dir.startswith("plat-"):
-            continue
-        if dir in ["tkinter", "idlelib", "turtledemo"]:
-            if not have_tcl:
-                continue
-            feature = tcltk
-            tcltk.set_current()
-        elif dir in ('test', 'tests'):
-            feature = testsuite
-        elif not have_ctypes and dir == "ctypes":
-            continue
-        feature.set_current()
-        lib = PyDirectory(db, cab, parent, dir, dir, "%s|%s" % (parent.make_short(dir), dir))
-        dirs.append(lib)
-        has_py = False
-        for name, subdir in files.items():
-            if subdir is None:
-                assert os.path.isfile(os.path.join(lib.absolute, name))
-                if name == 'README':
-                    lib.add_file("README.txt", src="README")
-                else:
-                    lib.add_file(name)
-                    has_py = has_py or name.endswith(".py") or name.endswith(".pyw")
-            else:
-                assert os.path.isdir(os.path.join(lib.absolute, name))
-                pydirs.append((lib, name, subdir, feature))
-
-        if has_py:
-            lib.remove_pyc()
-    # Add DLLs
-    default_feature.set_current()
-    lib = DLLs
-    lib.add_file("py.ico", src=srcdir+"/PC/py.ico")
-    lib.add_file("pyc.ico", src=srcdir+"/PC/pyc.ico")
-    dlls = []
-    tclfiles = []
-    for f in extensions:
-        if f=="_tkinter.pyd":
-            continue
-        if not os.path.exists(srcdir + "/" + PCBUILD + "/" + f):
-            print("WARNING: Missing extension", f)
-            continue
-        dlls.append(f)
-        lib.add_file(f)
-    lib.add_file('python3.dll')
-    # Add sqlite
-    if msilib.msi_type=="Intel64;1033":
-        sqlite_arch = "/ia64"
-    elif msilib.msi_type=="x64;1033":
-        sqlite_arch = "/amd64"
-        tclsuffix = "64"
-    else:
-        sqlite_arch = ""
-        tclsuffix = ""
-    lib.add_file("sqlite3.dll")
-    if have_tcl:
-        if not os.path.exists("%s/%s/_tkinter.pyd" % (srcdir, PCBUILD)):
-            print("WARNING: Missing _tkinter.pyd")
-        else:
-            lib.start_component("TkDLLs", tcltk)
-            lib.add_file("_tkinter.pyd")
-            dlls.append("_tkinter.pyd")
-            tcldir = os.path.normpath(srcdir+("/../tcltk%s/bin" % tclsuffix))
-            for f in glob.glob1(tcldir, "*.dll"):
-                lib.add_file(f, src=os.path.join(tcldir, f))
-    # check whether there are any unknown extensions
-    for f in glob.glob1(srcdir+"/"+PCBUILD, "*.pyd"):
-        if f.endswith("_d.pyd"): continue # debug version
-        if f in dlls: continue
-        print("WARNING: Unknown extension", f)
-
-    # Add headers
-    default_feature.set_current()
-    lib = PyDirectory(db, cab, root, "include", "include", "INCLUDE|include")
-    lib.glob("*.h")
-    lib.add_file("pyconfig.h", src="../PC/pyconfig.h")
-    # Add import libraries
-    lib = PyDirectory(db, cab, root, PCBUILD, "libs", "LIBS|libs")
-    for f in dlls:
-        lib.add_file(f.replace('pyd','lib'))
-    lib.add_file('python%s%s.lib' % (major, minor))
-    lib.add_file('python3.lib')
-    # Add the mingw-format library
-    if have_mingw:
-        lib.add_file('libpython%s%s.a' % (major, minor))
-    if have_tcl:
-        # Add Tcl/Tk
-        tcldirs = [(root, '../tcltk%s/lib' % tclsuffix, 'tcl')]
-        tcltk.set_current()
-        while tcldirs:
-            parent, phys, dir = tcldirs.pop()
-            lib = PyDirectory(db, cab, parent, phys, dir, "%s|%s" % (parent.make_short(dir), dir))
-            if not os.path.exists(lib.absolute):
-                continue
-            for f in os.listdir(lib.absolute):
-                if os.path.isdir(os.path.join(lib.absolute, f)):
-                    tcldirs.append((lib, f, f))
-                else:
-                    lib.add_file(f)
-    # Add tools
-    tools.set_current()
-    tooldir = PyDirectory(db, cab, root, "Tools", "Tools", "TOOLS|Tools")
-    for f in ['i18n', 'pynche', 'Scripts']:
-        lib = PyDirectory(db, cab, tooldir, f, f, "%s|%s" % (tooldir.make_short(f), f))
-        lib.glob("*.py")
-        lib.glob("*.pyw")
-        lib.remove_pyc()
-        lib.glob("*.txt")
-        if f == "pynche":
-            x = PyDirectory(db, cab, lib, "X", "X", "X|X")
-            x.glob("*.txt")
-        if os.path.exists(os.path.join(lib.absolute, "README")):
-            lib.add_file("README.txt", src="README")
-        if f == 'Scripts':
-            lib.add_file("2to3.py", src="2to3")
-            lib.add_file("pydoc3.py", src="pydoc3")
-            lib.add_file("pyvenv.py", src="pyvenv")
-    # Add documentation
-    htmlfiles.set_current()
-    lib = PyDirectory(db, cab, root, "Doc", "Doc", "DOC|Doc")
-    lib.start_component("documentation", keyfile=docfile)
-    lib.add_file(docfile, src="build/htmlhelp/"+docfile)
-
-    cab.commit(db)
-
-    for f in tmpfiles:
-        os.unlink(f)
-
-# See "Registry Table", "Component Table"
-def add_registry(db):
-    # File extensions, associated with the REGISTRY.def component
-    # IDLE verbs depend on the tcltk feature.
-    # msidbComponentAttributesRegistryKeyPath = 4
-    # -1 for Root specifies "dependent on ALLUSERS property"
-    tcldata = []
-    if have_tcl:
-        tcldata = [
-            ("REGISTRY.tcl", msilib.gen_uuid(), "TARGETDIR", registry_component, None,
-             "py.IDLE")]
-    add_data(db, "Component",
-             # msidbComponentAttributesRegistryKeyPath = 4
-             [("REGISTRY", msilib.gen_uuid(), "TARGETDIR", registry_component, None,
-               "InstallPath"),
-              ("REGISTRY.doc", msilib.gen_uuid(), "TARGETDIR", registry_component, None,
-               "Documentation"),
-              ("REGISTRY.path", msilib.gen_uuid(), "TARGETDIR", registry_component, None,
-              None),
-              ("REGISTRY.ensurepip", msilib.gen_uuid(), "TARGETDIR", registry_component, "EnsurePipRun",
-              None),
-              ("REGISTRY.def", msilib.gen_uuid(), "TARGETDIR", registry_component,
-               None, None)] + tcldata)
-    # See "FeatureComponents Table".
-    # The association between TclTk and pythonw.exe is necessary to make ICE59
-    # happy, because the installer otherwise believes that the IDLE and PyDoc
-    # shortcuts might get installed without pythonw.exe being install. This
-    # is not true, since installing TclTk will install the default feature, which
-    # will cause pythonw.exe to be installed.
-    # REGISTRY.tcl is not associated with any feature, as it will be requested
-    # through a custom action
-    tcldata = []
-    if have_tcl:
-        tcldata = [(tcltk.id, "pythonw.exe")]
-    add_data(db, "FeatureComponents",
-             [(default_feature.id, "REGISTRY"),
-              (htmlfiles.id, "REGISTRY.doc"),
-              (prepend_path.id, "REGISTRY.path"),
-              (pip_feature.id, "REGISTRY.ensurepip"),
-              (ext_feature.id, "REGISTRY.def")] +
-              tcldata
-              )
-    # Extensions are not advertised. For advertised extensions,
-    # we would need separate binaries that install along with the
-    # extension.
-    pat = r"Software\Classes\%sPython.%sFile\shell\%s\command"
-    ewi = "Edit with IDLE"
-    pat2 = r"Software\Classes\%sPython.%sFile\DefaultIcon"
-    pat3 = r"Software\Classes\%sPython.%sFile"
-    pat4 = r"Software\Classes\%sPython.%sFile\shellex\DropHandler"
-    tcl_verbs = []
-    if have_tcl:
-        tcl_verbs=[
-             ("py.IDLE", -1, pat % (testprefix, "", ewi), "",
-              r'"[TARGETDIR]pythonw.exe" "[TARGETDIR]Lib\idlelib\idle.pyw" -e "%1"',
-              "REGISTRY.tcl"),
-             ("pyw.IDLE", -1, pat % (testprefix, "NoCon", ewi), "",
-              r'"[TARGETDIR]pythonw.exe" "[TARGETDIR]Lib\idlelib\idle.pyw" -e "%1"',
-              "REGISTRY.tcl"),
-        ]
-    add_data(db, "Registry",
-            [# Extensions
-             ("py.ext", -1, r"Software\Classes\."+ext, "",
-              "Python.File", "REGISTRY.def"),
-             ("pyw.ext", -1, r"Software\Classes\."+ext+'w', "",
-              "Python.NoConFile", "REGISTRY.def"),
-             ("pyc.ext", -1, r"Software\Classes\."+ext+'c', "",
-              "Python.CompiledFile", "REGISTRY.def"),
-             ("pyo.ext", -1, r"Software\Classes\."+ext+'o', "",
-              "Python.CompiledFile", "REGISTRY.def"),
-             # MIME types
-             ("py.mime", -1, r"Software\Classes\."+ext, "Content Type",
-              "text/plain", "REGISTRY.def"),
-             ("pyw.mime", -1, r"Software\Classes\."+ext+'w', "Content Type",
-              "text/plain", "REGISTRY.def"),
-             #Verbs
-             ("py.open", -1, pat % (testprefix, "", "open"), "",
-              r'"[LAUNCHERDIR]py.exe" "%1" %*', "REGISTRY.def"),
-             ("pyw.open", -1, pat % (testprefix, "NoCon", "open"), "",
-              r'"[LAUNCHERDIR]pyw.exe" "%1" %*', "REGISTRY.def"),
-             ("pyc.open", -1, pat % (testprefix, "Compiled", "open"), "",
-              r'"[LAUNCHERDIR]py.exe" "%1" %*', "REGISTRY.def"),
-             ] + tcl_verbs + [
-             #Icons
-             ("py.icon", -1, pat2 % (testprefix, ""), "",
-              r'[DLLs]py.ico', "REGISTRY.def"),
-             ("pyw.icon", -1, pat2 % (testprefix, "NoCon"), "",
-              r'[DLLs]py.ico', "REGISTRY.def"),
-             ("pyc.icon", -1, pat2 % (testprefix, "Compiled"), "",
-              r'[DLLs]pyc.ico', "REGISTRY.def"),
-             # Descriptions
-             ("py.txt", -1, pat3 % (testprefix, ""), "",
-              "Python File", "REGISTRY.def"),
-             ("pyw.txt", -1, pat3 % (testprefix, "NoCon"), "",
-              "Python File (no console)", "REGISTRY.def"),
-             ("pyc.txt", -1, pat3 % (testprefix, "Compiled"), "",
-              "Compiled Python File", "REGISTRY.def"),
-             # Drop Handler
-             ("py.drop", -1, pat4 % (testprefix, ""), "",
-              "{60254CA5-953B-11CF-8C96-00AA00B8708C}", "REGISTRY.def"),
-             ("pyw.drop", -1, pat4 % (testprefix, "NoCon"), "",
-              "{60254CA5-953B-11CF-8C96-00AA00B8708C}", "REGISTRY.def"),
-             ("pyc.drop", -1, pat4 % (testprefix, "Compiled"), "",
-              "{60254CA5-953B-11CF-8C96-00AA00B8708C}", "REGISTRY.def"),
-            ])
-
-    # PATHEXT
-    add_data(db, "Environment",
-             [("PathExtAddition", "=-*PathExt", "[~];.PY", "REGISTRY.def")])
-
-    # Registry keys
-    prefix = r"Software\%sPython\PythonCore\%s" % (testprefix, short_version)
-    add_data(db, "Registry",
-             [("InstallPath", -1, prefix+r"\InstallPath", "", "[TARGETDIR]", "REGISTRY"),
-              ("InstallGroup", -1, prefix+r"\InstallPath\InstallGroup", "",
-               "Python %s" % short_version, "REGISTRY"),
-              ("PythonPath", -1, prefix+r"\PythonPath", "",
-               r"[TARGETDIR]Lib;[TARGETDIR]DLLs", "REGISTRY"),
-              ("Documentation", -1, prefix+r"\Help\Main Python Documentation", "",
-               "[TARGETDIR]Doc\\"+docfile , "REGISTRY.doc"),
-              ("Modules", -1, prefix+r"\Modules", "+", None, "REGISTRY"),
-              ("AppPaths", -1, r"Software\Microsoft\Windows\CurrentVersion\App Paths\Python.exe",
-               "", r"[TARGETDIR]Python.exe", "REGISTRY.def"),
-              ("DisplayIcon", -1,
-               r"Software\Microsoft\Windows\CurrentVersion\Uninstall\%s" % product_code,
-               "DisplayIcon", "[TARGETDIR]python.exe", "REGISTRY"),
-              # Fake registry entry to allow installer to track whether ensurepip has been run
-              ("EnsurePipRun", -1, prefix+r"\EnsurePipRun", "", "#1", "REGISTRY.ensurepip"),
-              ])
-    # Shortcuts, see "Shortcut Table"
-    add_data(db, "Directory",
-             [("ProgramMenuFolder", "TARGETDIR", "."),
-              ("MenuDir", "ProgramMenuFolder", "PY%s%s|%sPython %s.%s" % (major,minor,testprefix,major,minor))])
-    add_data(db, "RemoveFile",
-             [("MenuDir", "TARGETDIR", None, "MenuDir", 2)])
-    tcltkshortcuts = []
-    if msilib.Win64:
-        bitted = "64 bit"
-    else:
-        bitted = "32 bit"
-    if have_tcl:
-        tcltkshortcuts = [
-              ("IDLE", "MenuDir",
-               "IDLE|IDLE (Python "+short_version+" GUI - "+bitted+")",
-               "pythonw.exe", tcltk.id, r'"[TARGETDIR]Lib\idlelib\idle.pyw"',
-               None, None, "python_icon.exe", 0, None, "TARGETDIR"),
-              ]
-    add_data(db, "Shortcut",
-             tcltkshortcuts +
-             [# Advertised shortcuts: targets are features, not files
-              ("Python", "MenuDir",
-               "PYTHON|Python "+short_version+" (command line - "+bitted+")",
-               "python.exe", default_feature.id, None, None, None,
-               "python_icon.exe", 2, None, "TARGETDIR"),
-              # Advertising the Manual breaks on (some?) Win98, and the shortcut lacks an
-              # icon first.
-              #("Manual", "MenuDir", "MANUAL|Python Manuals", "documentation",
-              # htmlfiles.id, None, None, None, None, None, None, None),
-              ## Non-advertised shortcuts: must be associated with a registry component
-              ("Manual", "MenuDir", "MANUAL|Python "+short_version+" Manuals",
-               "REGISTRY.doc", "[#%s]" % docfile,
-               None, None, None, None, None, None, None),
-              ("PyDoc", "MenuDir",
-               "MODDOCS|Python "+short_version+" Docs Server (pydoc - "+
-               bitted+")", "python.exe", default_feature.id, r'-m pydoc -b',
-               None, None, "python_icon.exe", 0, None, "TARGETDIR"),
-              ("Uninstall", "MenuDir", "UNINST|Uninstall Python "+
-               short_version+" ("+bitted+")", "REGISTRY",
-               SystemFolderName+"msiexec",  "/x%s" % product_code,
-               None, None, None, None, None, None),
-              ])
-    db.Commit()
-
-def build_pdbzip():
-    pdbexclude = ['kill_python.pdb', 'make_buildinfo.pdb',
-                  'make_versioninfo.pdb']
-    path = "python-%s%s-pdb.zip" % (full_current_version, msilib.arch_ext)
-    pdbzip = zipfile.ZipFile(path, 'w')
-    for f in glob.glob1(os.path.join(srcdir, PCBUILD), "*.pdb"):
-        if f not in pdbexclude and not f.endswith('_d.pdb'):
-            pdbzip.write(os.path.join(srcdir, PCBUILD, f), f)
-    pdbzip.close()
-
-db,msiname = build_database()
-try:
-    add_features(db)
-    add_ui(db)
-    add_files(db)
-    add_registry(db)
-    remove_old_versions(db)
-    db.Commit()
-finally:
-    del db
-
-# Merge CRT into MSI file. This requires the database to be closed.
-mod_dir = os.path.join(os.environ["ProgramFiles"], "Common Files", "Merge Modules")
-if msilib.Win64:
-    modules = ["Microsoft_VC100_CRT_x64.msm"]
-else:
-    modules = ["Microsoft_VC100_CRT_x86.msm"]
-
-for i, n in enumerate(modules):
-    modules[i] = os.path.join(mod_dir, n)
-
-def merge(msi, feature, rootdir, modules):
-    cab_and_filecount = []
-    # Step 1: Merge databases, extract cabfiles
-    m = msilib.MakeMerge2()
-    m.OpenLog("merge.log")
-    m.OpenDatabase(msi)
-    for module in modules:
-        print module
-        m.OpenModule(module,0)
-        m.Merge(feature, rootdir)
-        print "Errors:"
-        for e in m.Errors:
-            print e.Type, e.ModuleTable, e.DatabaseTable
-            print "   Modkeys:",
-            for s in e.ModuleKeys: print s,
-            print
-            print "   DBKeys:",
-            for s in e.DatabaseKeys: print s,
-            print
-        cabname = tempfile.mktemp(suffix=".cab")
-        m.ExtractCAB(cabname)
-        cab_and_filecount.append((cabname, len(m.ModuleFiles)))
-        m.CloseModule()
-    m.CloseDatabase(True)
-    m.CloseLog()
-
-    # Step 2: Add CAB files
-    i = msilib.MakeInstaller()
-    db = i.OpenDatabase(msi, constants.msiOpenDatabaseModeTransact)
-
-    v = db.OpenView("SELECT LastSequence FROM Media")
-    v.Execute(None)
-    maxmedia = -1
-    while 1:
-        r = v.Fetch()
-        if not r: break
-        seq = r.IntegerData(1)
-        if seq > maxmedia:
-            maxmedia = seq
-    print "Start of Media", maxmedia
-
-    for cabname, count in cab_and_filecount:
-        stream = "merged%d" % maxmedia
-        msilib.add_data(db, "Media",
-                [(maxmedia+1, maxmedia+count, None, "#"+stream, None, None)])
-        msilib.add_stream(db, stream,  cabname)
-        os.unlink(cabname)
-        maxmedia += count
-    # The merge module sets ALLUSERS to 1 in the property table.
-    # This is undesired; delete that
-    v = db.OpenView("DELETE FROM Property WHERE Property='ALLUSERS'")
-    v.Execute(None)
-    v.Close()
-    db.Commit()
-
-merge(msiname, "SharedCRT", "TARGETDIR", modules)
-
-# certname (from config.py) should be (a substring of)
-# the certificate subject, e.g. "Python Software Foundation"
-if certname:
-    os.system('signtool sign /n "%s" '
-      '/t http://timestamp.verisign.com/scripts/timestamp.dll '
-      '/d "Python %s" '
-      '%s' % (certname, full_current_version, msiname))
-
-if pdbzip:
-    build_pdbzip()
diff --git a/Tools/msi/msi.targets b/Tools/msi/msi.targets
new file mode 100644
--- /dev/null
+++ b/Tools/msi/msi.targets
@@ -0,0 +1,62 @@
+<?xml version="1.0" encoding="utf-8"?>
+<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+
+    <Target Name="ProcessInstallFiles" AfterTargets="PrepareForBuild" Condition="@(InstallFiles) != ''">
+        <PropertyGroup>
+            <_FileListTarget>$(IntermediateOutputPath)$(MSBuildProjectName).g.csv</_FileListTarget>
+            <_InstallFilesTarget>$(IntermediateOutputPath)$(MSBuildProjectName).g.wxs</_InstallFilesTarget>
+        </PropertyGroup>
+        
+        <ItemGroup>
+            <InstallFiles>
+                <_Source>%(Source)$([msbuild]::MakeRelative(%(SourceBase), %(FullPath)))</_Source>
+                <_Target>%(Target_)$([msbuild]::MakeRelative(%(TargetBase), %(FullPath)))</_Target>
+            </InstallFiles>
+        </ItemGroup>
+        
+        <WriteLinesToFile File="$(_FileListTarget)" Lines="@(InstallFiles->'&quot;%(_Source)&quot;,&quot;%(_Target)&quot;,&quot;%(Group)&quot;,&quot;%(DiskId)&quot;,&quot;%(Condition)&quot;')" Overwrite="true" />
+        <Exec Command='"$(PythonExe)" csv_to_wxs.py "$(_FileListTarget)" "$(_InstallFilesTarget)"'
+              WorkingDirectory="$(MSBuildThisFileDirectory)" />
+
+        <ItemGroup>
+            <FileWrites Include="$(_FileListTarget);$(_InstallFilesTarget)" />
+            <Compile Include="$(_InstallFilesTarget)" />
+        </ItemGroup>
+    </Target>
+
+    <Target Name="_TransformWxlTemplates" AfterTargets="PrepareForBuild" Inputs="@(WxlTemplate)" Outputs="$(IntermediateOutputPath)%(Filename).wxl">
+        <PropertyGroup>
+            <_Content>$([System.IO.File]::ReadAllText(%(WxlTemplate.FullPath)).Replace(`{{ShortVersion}}`, `$(MajorVersionNumber).$(MinorVersionNumber)`).Replace(`{{LongVersion}}`, `$(PythonVersion)`).Replace(`{{Bitness}}`, `$(Bitness)`))</_Content>
+            <_ExistingContent Condition="Exists('$(IntermediateOutputPath)%(WxlTemplate.Filename).wxl')">$([System.IO.File]::ReadAllText($(IntermediateOutputPath)%(WxlTemplate.Filename).wxl))</_ExistingContent>
+        </PropertyGroup>
+        
+        <WriteLinesToFile File="$(IntermediateOutputPath)%(WxlTemplate.Filename).wxl"
+                          Lines="$(_Content)"
+                          Overwrite="true"
+                          Condition="$(_Content) != $(_ExistingContent)" />
+        
+        <ItemGroup>
+            <EmbeddedResource Include="$(IntermediateOutputPath)%(WxlTemplate.Filename).wxl" />
+            <FileWrites Include="$(IntermediateOutputPath)%(WxlTemplate.Filename).wxl" />
+        </ItemGroup>
+    </Target>
+
+    <Import Project="$(WixTargetsPath)" />
+
+    <Target Name="SignCabs">
+        <Error Text="Unable to locate signtool.exe. Set /p:SignToolPath and rebuild" Condition="'$(_SignCommand)' == ''" />
+        <Exec Command="$(_SignCommand) @(SignCabs->'&quot;%(FullPath)&quot;',' ')" ContinueOnError="false" />
+    </Target>
+    <Target Name="SignMsi">
+        <Error Text="Unable to locate signtool.exe. Set /p:SignToolPath and rebuild" Condition="'$(_SignCommand)' == ''" />
+        <Exec Command="$(_SignCommand) @(SignMsi->'&quot;%(FullPath)&quot;',' ')" ContinueOnError="false" />
+    </Target>
+    <Target Name="SignBundleEngine">
+        <Error Text="Unable to locate signtool.exe. Set /p:SignToolPath and rebuild" Condition="'$(_SignCommand)' == ''" />
+        <Exec Command="$(_SignCommand) @(SignBundleEngine->'&quot;%(FullPath)&quot;',' ')" ContinueOnError="false" />
+    </Target>
+    <Target Name="SignBundle">
+        <Error Text="Unable to locate signtool.exe. Set /p:SignToolPath and rebuild" Condition="'$(_SignCommand)' == ''" />
+        <Exec Command="$(_SignCommand) @(SignBundle->'&quot;%(FullPath)&quot;',' ')" ContinueOnError="false" />
+    </Target>
+</Project>
\ No newline at end of file
diff --git a/Tools/msi/msilib.py b/Tools/msi/msilib.py
deleted file mode 100644
--- a/Tools/msi/msilib.py
+++ /dev/null
@@ -1,679 +0,0 @@
-# Microsoft Installer Library
-# (C) 2003 Martin v. Loewis
-
-import win32com.client.gencache
-import win32com.client
-import pythoncom, pywintypes
-from win32com.client import constants
-import re, string, os, sets, glob, subprocess, sys, _winreg, struct, _msi
-
-try:
-    basestring
-except NameError:
-    basestring = (str, unicode)
-
-# Partially taken from Wine
-datasizemask=      0x00ff
-type_valid=        0x0100
-type_localizable=  0x0200
-
-typemask=          0x0c00
-type_long=         0x0000
-type_short=        0x0400
-type_string=       0x0c00
-type_binary=       0x0800
-
-type_nullable=     0x1000
-type_key=          0x2000
-# XXX temporary, localizable?
-knownbits = datasizemask | type_valid | type_localizable | \
-            typemask | type_nullable | type_key
-
-# Summary Info Property IDs
-PID_CODEPAGE=1
-PID_TITLE=2
-PID_SUBJECT=3
-PID_AUTHOR=4
-PID_KEYWORDS=5
-PID_COMMENTS=6
-PID_TEMPLATE=7
-PID_LASTAUTHOR=8
-PID_REVNUMBER=9
-PID_LASTPRINTED=11
-PID_CREATE_DTM=12
-PID_LASTSAVE_DTM=13
-PID_PAGECOUNT=14
-PID_WORDCOUNT=15
-PID_CHARCOUNT=16
-PID_APPNAME=18
-PID_SECURITY=19
-
-def reset():
-    global _directories
-    _directories = sets.Set()
-
-def EnsureMSI():
-    win32com.client.gencache.EnsureModule('{000C1092-0000-0000-C000-000000000046}', 1033, 1, 0)
-
-def EnsureMSM():
-    try:
-        win32com.client.gencache.EnsureModule('{0ADDA82F-2C26-11D2-AD65-00A0C9AF11A6}', 0, 1, 0)
-    except pywintypes.com_error:
-        win32com.client.gencache.EnsureModule('{0ADDA82F-2C26-11D2-AD65-00A0C9AF11A6}', 0, 2, 0)
-
-_Installer=None
-def MakeInstaller():
-    global _Installer
-    if _Installer is None:
-        EnsureMSI()
-        _Installer = win32com.client.Dispatch('WindowsInstaller.Installer',
-                     resultCLSID='{000C1090-0000-0000-C000-000000000046}')
-    return _Installer
-
-_Merge=None
-def MakeMerge2():
-    global _Merge
-    if _Merge is None:
-        EnsureMSM()
-        _Merge = win32com.client.Dispatch("Msm.Merge2.1")
-    return _Merge
-
-class Table:
-    def __init__(self, name):
-        self.name = name
-        self.fields = []
-
-    def add_field(self, index, name, type):
-        self.fields.append((index,name,type))
-
-    def sql(self):
-        fields = []
-        keys = []
-        self.fields.sort()
-        fields = [None]*len(self.fields)
-        for index, name, type in self.fields:
-            index -= 1
-            unk = type & ~knownbits
-            if unk:
-                print "%s.%s unknown bits %x" % (self.name, name, unk)
-            size = type & datasizemask
-            dtype = type & typemask
-            if dtype == type_string:
-                if size:
-                    tname="CHAR(%d)" % size
-                else:
-                    tname="CHAR"
-            elif dtype == type_short:
-                assert size==2
-                tname = "SHORT"
-            elif dtype == type_long:
-                assert size==4
-                tname="LONG"
-            elif dtype == type_binary:
-                assert size==0
-                tname="OBJECT"
-            else:
-                tname="unknown"
-                print "%s.%sunknown integer type %d" % (self.name, name, size)
-            if type & type_nullable:
-                flags = ""
-            else:
-                flags = " NOT NULL"
-            if type & type_localizable:
-                flags += " LOCALIZABLE"
-            fields[index] = "`%s` %s%s" % (name, tname, flags)
-            if type & type_key:
-                keys.append("`%s`" % name)
-        fields = ", ".join(fields)
-        keys = ", ".join(keys)
-        return "CREATE TABLE %s (%s PRIMARY KEY %s)" % (self.name, fields, keys)
-
-    def create(self, db):
-        v = db.OpenView(self.sql())
-        v.Execute(None)
-        v.Close()
-
-class Binary:
-    def __init__(self, fname):
-        self.name = fname
-    def __repr__(self):
-        return 'msilib.Binary(os.path.join(dirname,"%s"))' % self.name
-
-def gen_schema(destpath, schemapath):
-    d = MakeInstaller()
-    schema = d.OpenDatabase(schemapath,
-            win32com.client.constants.msiOpenDatabaseModeReadOnly)
-
-    # XXX ORBER BY
-    v=schema.OpenView("SELECT * FROM _Columns")
-    curtable=None
-    tables = []
-    v.Execute(None)
-    f = open(destpath, "wt")
-    f.write("from msilib import Table\n")
-    while 1:
-        r=v.Fetch()
-        if not r:break
-        name=r.StringData(1)
-        if curtable != name:
-            f.write("\n%s = Table('%s')\n" % (name,name))
-            curtable = name
-            tables.append(name)
-        f.write("%s.add_field(%d,'%s',%d)\n" %
-                (name, r.IntegerData(2), r.StringData(3), r.IntegerData(4)))
-    v.Close()
-
-    f.write("\ntables=[%s]\n\n" % (", ".join(tables)))
-
-    # Fill the _Validation table
-    f.write("_Validation_records = [\n")
-    v = schema.OpenView("SELECT * FROM _Validation")
-    v.Execute(None)
-    while 1:
-        r = v.Fetch()
-        if not r:break
-        # Table, Column, Nullable
-        f.write("(%s,%s,%s," %
-                (`r.StringData(1)`, `r.StringData(2)`, `r.StringData(3)`))
-        def put_int(i):
-            if r.IsNull(i):f.write("None, ")
-            else:f.write("%d," % r.IntegerData(i))
-        def put_str(i):
-            if r.IsNull(i):f.write("None, ")
-            else:f.write("%s," % `r.StringData(i)`)
-        put_int(4) # MinValue
-        put_int(5) # MaxValue
-        put_str(6) # KeyTable
-        put_int(7) # KeyColumn
-        put_str(8) # Category
-        put_str(9) # Set
-        put_str(10)# Description
-        f.write("),\n")
-    f.write("]\n\n")
-
-    f.close()
-
-def gen_sequence(destpath, msipath):
-    dir = os.path.dirname(destpath)
-    d = MakeInstaller()
-    seqmsi = d.OpenDatabase(msipath,
-            win32com.client.constants.msiOpenDatabaseModeReadOnly)
-
-    v = seqmsi.OpenView("SELECT * FROM _Tables");
-    v.Execute(None)
-    f = open(destpath, "w")
-    print >>f, "import msilib,os;dirname=os.path.dirname(__file__)"
-    tables = []
-    while 1:
-        r = v.Fetch()
-        if not r:break
-        table = r.StringData(1)
-        tables.append(table)
-        f.write("%s = [\n" % table)
-        v1 = seqmsi.OpenView("SELECT * FROM `%s`" % table)
-        v1.Execute(None)
-        info = v1.ColumnInfo(constants.msiColumnInfoTypes)
-        while 1:
-            r = v1.Fetch()
-            if not r:break
-            rec = []
-            for i in range(1,r.FieldCount+1):
-                if r.IsNull(i):
-                    rec.append(None)
-                elif info.StringData(i)[0] in "iI":
-                    rec.append(r.IntegerData(i))
-                elif info.StringData(i)[0] in "slSL":
-                    rec.append(r.StringData(i))
-                elif info.StringData(i)[0]=="v":
-                    size = r.DataSize(i)
-                    bytes = r.ReadStream(i, size, constants.msiReadStreamBytes)
-                    bytes = bytes.encode("latin-1") # binary data represented "as-is"
-                    if table == "Binary":
-                        fname = rec[0]+".bin"
-                        open(os.path.join(dir,fname),"wb").write(bytes)
-                        rec.append(Binary(fname))
-                    else:
-                        rec.append(bytes)
-                else:
-                    raise "Unsupported column type", info.StringData(i)
-            f.write(repr(tuple(rec))+",\n")
-        v1.Close()
-        f.write("]\n\n")
-    v.Close()
-    f.write("tables=%s\n" % repr(map(str,tables)))
-    f.close()
-
-class _Unspecified:pass
-def change_sequence(seq, action, seqno=_Unspecified, cond = _Unspecified):
-    "Change the sequence number of an action in a sequence list"
-    for i in range(len(seq)):
-        if seq[i][0] == action:
-            if cond is _Unspecified:
-                cond = seq[i][1]
-            if seqno is _Unspecified:
-                seqno = seq[i][2]
-            seq[i] = (action, cond, seqno)
-            return
-    raise ValueError, "Action not found in sequence"
-
-def add_data(db, table, values):
-    d = MakeInstaller()
-    v = db.OpenView("SELECT * FROM `%s`" % table)
-    count = v.ColumnInfo(0).FieldCount
-    r = d.CreateRecord(count)
-    for value in values:
-        assert len(value) == count, value
-        for i in range(count):
-            field = value[i]
-            if isinstance(field, (int, long)):
-                r.SetIntegerData(i+1,field)
-            elif isinstance(field, basestring):
-                r.SetStringData(i+1,field)
-            elif field is None:
-                pass
-            elif isinstance(field, Binary):
-                r.SetStream(i+1, field.name)
-            else:
-                raise TypeError, "Unsupported type %s" % field.__class__.__name__
-        v.Modify(win32com.client.constants.msiViewModifyInsert, r)
-        r.ClearData()
-    v.Close()
-
-def add_stream(db, name, path):
-    d = MakeInstaller()
-    v = db.OpenView("INSERT INTO _Streams (Name, Data) VALUES ('%s', ?)" % name)
-    r = d.CreateRecord(1)
-    r.SetStream(1, path)
-    v.Execute(r)
-    v.Close()
-
-def init_database(name, schema,
-                  ProductName, ProductCode, ProductVersion,
-                  Manufacturer,
-                  request_uac = False):
-    try:
-        os.unlink(name)
-    except OSError:
-        pass
-    ProductCode = ProductCode.upper()
-    d = MakeInstaller()
-    # Create the database
-    db = d.OpenDatabase(name,
-         win32com.client.constants.msiOpenDatabaseModeCreate)
-    # Create the tables
-    for t in schema.tables:
-        t.create(db)
-    # Fill the validation table
-    add_data(db, "_Validation", schema._Validation_records)
-    # Initialize the summary information, allowing at most 20 properties
-    si = db.GetSummaryInformation(20)
-    si.SetProperty(PID_TITLE, "Installation Database")
-    si.SetProperty(PID_SUBJECT, ProductName)
-    si.SetProperty(PID_AUTHOR, Manufacturer)
-    si.SetProperty(PID_TEMPLATE, msi_type)
-    si.SetProperty(PID_REVNUMBER, gen_uuid())
-    if request_uac:
-        wc = 2 # long file names, compressed, original media
-    else:
-        wc = 2 | 8 # +never invoke UAC
-    si.SetProperty(PID_WORDCOUNT, wc)
-    si.SetProperty(PID_PAGECOUNT, 200)
-    si.SetProperty(PID_APPNAME, "Python MSI Library")
-    # XXX more properties
-    si.Persist()
-    add_data(db, "Property", [
-        ("ProductName", ProductName),
-        ("ProductCode", ProductCode),
-        ("ProductVersion", ProductVersion),
-        ("Manufacturer", Manufacturer),
-        ("ProductLanguage", "1033")])
-    db.Commit()
-    return db
-
-def add_tables(db, module):
-    for table in module.tables:
-        add_data(db, table, getattr(module, table))
-
-def make_id(str):
-    #str = str.replace(".", "_") # colons are allowed
-    str = str.replace(" ", "_")
-    str = str.replace("-", "_")
-    str = str.replace("+", "_")
-    if str[0] in string.digits:
-        str = "_"+str
-    assert re.match("^[A-Za-z_][A-Za-z0-9_.]*$", str), "FILE"+str
-    return str
-
-def gen_uuid():
-    return str(pythoncom.CreateGuid())
-
-class CAB:
-    def __init__(self, name):
-        self.name = name
-        self.files = []
-        self.filenames = sets.Set()
-        self.index = 0
-
-    def gen_id(self, dir, file):
-        logical = _logical = make_id(file)
-        pos = 1
-        while logical in self.filenames:
-            logical = "%s.%d" % (_logical, pos)
-            pos += 1
-        self.filenames.add(logical)
-        return logical
-
-    def append(self, full, file, logical = None):
-        if os.path.isdir(full):
-            return
-        if not logical:
-            logical = self.gen_id(dir, file)
-        self.index += 1
-        self.files.append((full, logical))
-        return self.index, logical
-
-    def commit(self, db):
-        try:
-            os.unlink(self.name+".cab")
-        except OSError:
-            pass
-        _msi.FCICreate(self.name+".cab", self.files)
-        add_data(db, "Media",
-                [(1, self.index, None, "#"+self.name, None, None)])
-        add_stream(db, self.name, self.name+".cab")
-        os.unlink(self.name+".cab")
-        db.Commit()
-
-_directories = sets.Set()
-class Directory:
-    def __init__(self, db, cab, basedir, physical, _logical, default, componentflags=None):
-        """Create a new directory in the Directory table. There is a current component
-        at each point in time for the directory, which is either explicitly created
-        through start_component, or implicitly when files are added for the first
-        time. Files are added into the current component, and into the cab file.
-        To create a directory, a base directory object needs to be specified (can be
-        None), the path to the physical directory, and a logical directory name.
-        Default specifies the DefaultDir slot in the directory table. componentflags
-        specifies the default flags that new components get."""
-        index = 1
-        _logical = make_id(_logical)
-        logical = _logical
-        while logical in _directories:
-            logical = "%s%d" % (_logical, index)
-            index += 1
-        _directories.add(logical)
-        self.db = db
-        self.cab = cab
-        self.basedir = basedir
-        self.physical = physical
-        self.logical = logical
-        self.component = None
-        self.short_names = {}
-        self.ids = sets.Set()
-        self.keyfiles = {}
-        self.componentflags = componentflags
-        if basedir:
-            self.absolute = os.path.join(basedir.absolute, physical)
-            blogical = basedir.logical
-        else:
-            self.absolute = physical
-            blogical = None
-        # initially assume that all files in this directory are unpackaged
-        # as files from self.absolute get added, this set is reduced
-        self.unpackaged_files = set()
-        for f in os.listdir(self.absolute):
-            if os.path.isfile(os.path.join(self.absolute, f)):
-                self.unpackaged_files.add(f)
-        add_data(db, "Directory", [(logical, blogical, default)])
-
-    def start_component(self, component = None, feature = None, flags = None, keyfile = None, uuid=None):
-        """Add an entry to the Component table, and make this component the current for this
-        directory. If no component name is given, the directory name is used. If no feature
-        is given, the current feature is used. If no flags are given, the directory's default
-        flags are used. If no keyfile is given, the KeyPath is left null in the Component
-        table."""
-        if flags is None:
-            flags = self.componentflags
-        if uuid is None:
-            uuid = gen_uuid()
-        else:
-            uuid = uuid.upper()
-        if component is None:
-            component = self.logical
-        self.component = component
-        if Win64:
-            flags |= 256
-        if keyfile:
-            keyid = self.cab.gen_id(self.absolute, keyfile)
-            self.keyfiles[keyfile] = keyid
-        else:
-            keyid = None
-        add_data(self.db, "Component",
-                        [(component, uuid, self.logical, flags, None, keyid)])
-        if feature is None:
-            feature = current_feature
-        add_data(self.db, "FeatureComponents",
-                        [(feature.id, component)])
-
-    def make_short(self, file):
-        long = file
-        file = re.sub(r'[\?|><:/*"+,;=\[\]]', '_', file) # restrictions on short names
-        parts = file.split(".", 1)
-        if len(parts)>1:
-            suffix = parts[1].upper()
-        else:
-            suffix = ''
-        prefix = parts[0].upper()
-        if len(prefix) <= 8 and '.' not in suffix and len(suffix) <= 3:
-            if suffix:
-                file = prefix+"."+suffix
-            else:
-                file = prefix
-            assert file not in self.short_names, (file, self.short_names[file])
-        else:
-            prefix = prefix[:6]
-            if suffix:
-                # last three characters of last suffix
-                suffix = suffix.rsplit('.')[-1][:3]
-            pos = 1
-            while 1:
-                if suffix:
-                    file = "%s~%d.%s" % (prefix, pos, suffix)
-                else:
-                    file = "%s~%d" % (prefix, pos)
-                if file not in self.short_names: break
-                pos += 1
-                assert pos < 10000
-                if pos in (10, 100, 1000):
-                    prefix = prefix[:-1]
-        self.short_names[file] = long
-        return file
-
-    def add_file(self, file, src=None, version=None, language=None):
-        """Add a file to the current component of the directory, starting a new one
-        if there is no current component. By default, the file name in the source
-        and the file table will be identical. If the src file is specified, it is
-        interpreted relative to the current directory. Optionally, a version and a
-        language can be specified for the entry in the File table."""
-        if not self.component:
-            self.start_component(self.logical, current_feature)
-        if not src:
-            # Allow relative paths for file if src is not specified
-            src = file
-            file = os.path.basename(file)
-        absolute = os.path.join(self.absolute, src)
-        if absolute.startswith(self.absolute):
-            # mark file as packaged
-            relative = absolute[len(self.absolute)+1:]
-            if relative in self.unpackaged_files:
-                self.unpackaged_files.remove(relative)
-        assert not re.search(r'[\?|><:/*]"', file) # restrictions on long names
-        if self.keyfiles.has_key(file):
-            logical = self.keyfiles[file]
-        else:
-            logical = None
-        sequence, logical = self.cab.append(absolute, file, logical)
-        assert logical not in self.ids
-        self.ids.add(logical)
-        short = self.make_short(file)
-        full = "%s|%s" % (short, file)
-        filesize = os.stat(absolute).st_size
-        # constants.msidbFileAttributesVital
-        # Compressed omitted, since it is the database default
-        # could add r/o, system, hidden
-        attributes = 512
-        add_data(self.db, "File",
-                        [(logical, self.component, full, filesize, version,
-                         language, attributes, sequence)])
-        if not version:
-            # Add hash if the file is not versioned
-            filehash = MakeInstaller().FileHash(absolute, 0)
-            add_data(self.db, "MsiFileHash",
-                     [(logical, 0, filehash.IntegerData(1),
-                       filehash.IntegerData(2), filehash.IntegerData(3),
-                       filehash.IntegerData(4))])
-        # Automatically remove .pyc/.pyo files on uninstall (2)
-        # XXX: adding so many RemoveFile entries makes installer unbelievably
-        # slow. So instead, we have to use wildcard remove entries
-        # if file.endswith(".py"):
-        #     add_data(self.db, "RemoveFile",
-        #              [(logical+"c", self.component, "%sC|%sc" % (short, file),
-        #                self.logical, 2),
-        #               (logical+"o", self.component, "%sO|%so" % (short, file),
-        #                self.logical, 2)])
-
-    def glob(self, pattern, exclude = None):
-        """Add a list of files to the current component as specified in the
-        glob pattern. Individual files can be excluded in the exclude list."""
-        files = glob.glob1(self.absolute, pattern)
-        for f in files:
-            if exclude and f in exclude: continue
-            self.add_file(f)
-        return files
-
-    def remove_pyc(self):
-        "Remove .pyc/.pyo files from __pycache__ on uninstall"
-        directory = self.logical + "_pycache"
-        add_data(self.db, "Directory", [(directory, self.logical, "__PYCA~1|__pycache__")])
-        flags = 256 if Win64 else 0
-        add_data(self.db, "Component",
-                [(directory, gen_uuid(), directory, flags, None, None)])
-        add_data(self.db, "FeatureComponents", [(current_feature.id, directory)])
-        add_data(self.db, "CreateFolder", [(directory, directory)])
-        add_data(self.db, "RemoveFile",
-                 [(self.component, self.component, "*.*", directory, 2),
-                 ])
-
-    def removefile(self, key, pattern):
-        "Add a RemoveFile entry"
-        add_data(self.db, "RemoveFile", [(self.component+key, self.component, pattern, self.logical, 2)])
-
-
-class Feature:
-    def __init__(self, db, id, title, desc, display, level = 1,
-                 parent=None, directory = None, attributes=0):
-        self.id = id
-        if parent:
-            parent = parent.id
-        add_data(db, "Feature",
-                        [(id, parent, title, desc, display,
-                          level, directory, attributes)])
-    def set_current(self):
-        global current_feature
-        current_feature = self
-
-class Control:
-    def __init__(self, dlg, name):
-        self.dlg = dlg
-        self.name = name
-
-    def event(self, ev, arg, cond = "1", order = None):
-        add_data(self.dlg.db, "ControlEvent",
-                 [(self.dlg.name, self.name, ev, arg, cond, order)])
-
-    def mapping(self, ev, attr):
-        add_data(self.dlg.db, "EventMapping",
-                 [(self.dlg.name, self.name, ev, attr)])
-
-    def condition(self, action, condition):
-        add_data(self.dlg.db, "ControlCondition",
-                 [(self.dlg.name, self.name, action, condition)])
-
-class RadioButtonGroup(Control):
-    def __init__(self, dlg, name, property):
-        self.dlg = dlg
-        self.name = name
-        self.property = property
-        self.index = 1
-
-    def add(self, name, x, y, w, h, text, value = None):
-        if value is None:
-            value = name
-        add_data(self.dlg.db, "RadioButton",
-                 [(self.property, self.index, value,
-                   x, y, w, h, text, None)])
-        self.index += 1
-
-class Dialog:
-    def __init__(self, db, name, x, y, w, h, attr, title, first, default, cancel):
-        self.db = db
-        self.name = name
-        self.x, self.y, self.w, self.h = x,y,w,h
-        add_data(db, "Dialog", [(name, x,y,w,h,attr,title,first,default,cancel)])
-
-    def control(self, name, type, x, y, w, h, attr, prop, text, next, help):
-        add_data(self.db, "Control",
-                 [(self.name, name, type, x, y, w, h, attr, prop, text, next, help)])
-        return Control(self, name)
-
-    def text(self, name, x, y, w, h, attr, text):
-        return self.control(name, "Text", x, y, w, h, attr, None,
-                     text, None, None)
-
-    def bitmap(self, name, x, y, w, h, text):
-        return self.control(name, "Bitmap", x, y, w, h, 1, None, text, None, None)
-
-    def line(self, name, x, y, w, h):
-        return self.control(name, "Line", x, y, w, h, 1, None, None, None, None)
-
-    def pushbutton(self, name, x, y, w, h, attr, text, next):
-        return self.control(name, "PushButton", x, y, w, h, attr, None, text, next, None)
-
-    def radiogroup(self, name, x, y, w, h, attr, prop, text, next):
-        add_data(self.db, "Control",
-                 [(self.name, name, "RadioButtonGroup",
-                   x, y, w, h, attr, prop, text, next, None)])
-        return RadioButtonGroup(self, name, prop)
-
-    def checkbox(self, name, x, y, w, h, attr, prop, text, next):
-        return self.control(name, "CheckBox", x, y, w, h, attr, prop, text, next, None)
-
-def pe_type(path):
-    header = open(path, "rb").read(1000)
-    # offset of PE header is at offset 0x3c
-    pe_offset = struct.unpack("<i", header[0x3c:0x40])[0]
-    assert header[pe_offset:pe_offset+4] == "PE\0\0"
-    machine = struct.unpack("<H", header[pe_offset+4:pe_offset+6])[0]
-    return machine
-
-def set_arch_from_file(path):
-    global msi_type, Win64, arch_ext
-    machine = pe_type(path)
-    if machine == 0x14c:
-        # i386
-        msi_type = "Intel"
-        Win64 = 0
-        arch_ext = ''
-    elif machine == 0x200:
-        # Itanium
-        msi_type = "Intel64"
-        Win64 = 1
-        arch_ext = '.ia64'
-    elif machine == 0x8664:
-        # AMD64
-        msi_type = "x64"
-        Win64 = 1
-        arch_ext = '.amd64'
-    else:
-        raise ValueError, "Unsupported architecture"
-    msi_type += ";1033"
diff --git a/Tools/msi/msisupport.c b/Tools/msi/msisupport.c
deleted file mode 100644
--- a/Tools/msi/msisupport.c
+++ /dev/null
@@ -1,93 +0,0 @@
-#include "windows.h"
-#include "msiquery.h"
-
-/* Print a debug message to the installer log file.
- * To see the debug messages, install with
- * msiexec /i pythonxy.msi /l*v python.log
- */
-static UINT debug(MSIHANDLE hInstall, LPCSTR msg)
-{
-    MSIHANDLE hRec = MsiCreateRecord(1);
-    if (!hRec || MsiRecordSetStringA(hRec, 1, msg) != ERROR_SUCCESS) {
-        return ERROR_INSTALL_FAILURE;
-    }
-    MsiProcessMessage(hInstall, INSTALLMESSAGE_INFO, hRec);
-    MsiCloseHandle(hRec);
-    return ERROR_SUCCESS;
-}
-
-/* Check whether the TARGETDIR exists and is a directory.
- * Set TargetExists appropriately.
- */
-UINT __declspec(dllexport) __stdcall CheckDir(MSIHANDLE hInstall)
-{
-#define PSIZE 1024
-    WCHAR wpath[PSIZE];
-    char path[PSIZE];
-    UINT result;
-    DWORD size = PSIZE;
-    DWORD attributes;
-
-
-    result = MsiGetPropertyW(hInstall, L"TARGETDIR", wpath, &size);
-    if (result != ERROR_SUCCESS)
-        return result;
-    wpath[size] = L'\0';
-    path[size] = L'\0';
-
-    attributes = GetFileAttributesW(wpath);
-    if (attributes == INVALID_FILE_ATTRIBUTES ||
-        !(attributes & FILE_ATTRIBUTE_DIRECTORY))
-    {
-        return MsiSetPropertyA(hInstall, "TargetExists", "0");
-    } else {
-        return MsiSetPropertyA(hInstall, "TargetExists", "1");
-    }
-}
-
-/* Update the state of the REGISTRY.tcl component according to the
- * Extension and TclTk features. REGISTRY.tcl must be installed
- * if both features are installed, and must be absent otherwise.
- */
-UINT __declspec(dllexport) __stdcall UpdateEditIDLE(MSIHANDLE hInstall)
-{
-    INSTALLSTATE ext_old, ext_new, tcl_old, tcl_new, reg_new;
-    UINT result;
-
-    result = MsiGetFeatureStateA(hInstall, "Extensions", &ext_old, &ext_new);
-    if (result != ERROR_SUCCESS)
-        return result;
-    result = MsiGetFeatureStateA(hInstall, "TclTk", &tcl_old, &tcl_new);
-    if (result != ERROR_SUCCESS)
-        return result;
-
-    /* If the current state is Absent, and the user did not select
-       the feature in the UI, Installer apparently sets the "selected"
-       state to unknown. Update it to the current value, then. */
-    if (ext_new == INSTALLSTATE_UNKNOWN)
-        ext_new = ext_old;
-    if (tcl_new == INSTALLSTATE_UNKNOWN)
-        tcl_new = tcl_old;
-
-    // XXX consider current state of REGISTRY.tcl?
-    if (((tcl_new == INSTALLSTATE_LOCAL) ||
-             (tcl_new == INSTALLSTATE_SOURCE) ||
-             (tcl_new == INSTALLSTATE_DEFAULT)) &&
-        ((ext_new == INSTALLSTATE_LOCAL) ||
-         (ext_new == INSTALLSTATE_SOURCE) ||
-         (ext_new == INSTALLSTATE_DEFAULT))) {
-        reg_new = INSTALLSTATE_SOURCE;
-    } else {
-        reg_new = INSTALLSTATE_ABSENT;
-    }
-    result = MsiSetComponentStateA(hInstall, "REGISTRY.tcl", reg_new);
-    return result;
-}
-
-BOOL APIENTRY DllMain(HANDLE hModule,
-                      DWORD  ul_reason_for_call,
-                      LPVOID lpReserved)
-{
-    return TRUE;
-}
-
diff --git a/Tools/msi/msisupport.mak b/Tools/msi/msisupport.mak
deleted file mode 100644
--- a/Tools/msi/msisupport.mak
+++ /dev/null
@@ -1,9 +0,0 @@
-# /OPT: REF and ICF are added by VS.NET by default
-msisupport.dll:	msisupport.obj
-	link.exe /OUT:msisupport.dll /INCREMENTAL:NO /NOLOGO /DLL /SUBSYSTEM:WINDOWS /OPT:REF /OPT:ICF msisupport.obj msi.lib kernel32.lib
-
-# We request a static CRT, so that there will be no CRT dependencies
-# for the target system. We cannot do without a CRT, since it provides
-# the DLL entry point.
-msisupport.obj:	msisupport.c
-	cl /O2 /D WIN32 /D NDEBUG /D _WINDOWS /MT /W3 /c msisupport.c
diff --git a/Tools/msi/path/path.wixproj b/Tools/msi/path/path.wixproj
new file mode 100644
--- /dev/null
+++ b/Tools/msi/path/path.wixproj
@@ -0,0 +1,19 @@
+<?xml version="1.0" encoding="utf-8"?>
+<Project ToolsVersion="4.0" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+    <PropertyGroup>
+        <ProjectGuid>{91C99298-8E2E-4422-A5AF-CC4FFF9A58D3}</ProjectGuid>
+        <SchemaVersion>2.0</SchemaVersion>
+        <OutputName>path</OutputName>
+        <OutputType>Package</OutputType>
+        <SuppressIces>ICE71</SuppressIces>
+    </PropertyGroup>
+    <Import Project="..\msi.props" />
+    <ItemGroup>
+        <Compile Include="*.wxs" />
+    </ItemGroup>
+    <ItemGroup>
+        <EmbeddedResource Include="*.wxl" />
+    </ItemGroup>
+    
+    <Import Project="..\msi.targets" />
+</Project>
\ No newline at end of file
diff --git a/Tools/msi/path/path.wxs b/Tools/msi/path/path.wxs
new file mode 100644
--- /dev/null
+++ b/Tools/msi/path/path.wxs
@@ -0,0 +1,35 @@
+?<?xml version="1.0" encoding="UTF-8"?>
+<Wix xmlns="http://schemas.microsoft.com/wix/2006/wi">
+    <Product Id="*" Language="!(loc.LCID)" Name="!(loc.Title)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)">
+        <Package InstallerVersion="300" Compressed="yes" InstallScope="perUser" Platform="$(var.Platform)" />
+        
+        <PropertyRef Id="UpgradeTable" />
+        <PropertyRef Id="REGISTRYKEY" />
+        
+        <Feature Id="DefaultFeature" AllowAdvertise="no" Title="!(loc.Title)" Description="!(loc.Description)">
+            <Component Id="PrependPath_CU" Directory="InstallDirectory" Guid="*">
+                <Condition>NOT ALLUSERS=1</Condition>
+                <RegistryValue KeyPath="yes" Root="HKCU" Key="[REGISTRYKEY]\PrependPath" Value="1" Type="integer" />
+                
+                <CreateFolder Directory="Scripts" />
+                <RemoveFolder Id="Remove_Scripts_CU" Directory="Scripts" On="uninstall" />
+                
+                <Environment Id="PATH_CU" Action="set" Name="PATH" Part="first" Value="[InstallDirectory]" />
+                <Environment Id="SCRIPTS_PATH_CU" Action="set" Name="PATH" Part="first" Value="[Scripts]" />
+            </Component>
+            <Component Id="PrependPath_LM" Directory="InstallDirectory" Guid="*">
+                <Condition>ALLUSERS=1</Condition>
+                <RegistryValue KeyPath="yes" Root="HKLM" Key="[REGISTRYKEY]\PrependPath" Value="1" Type="integer" />
+                
+                <CreateFolder Directory="Scripts" />
+                <RemoveFolder Id="Remove_Scripts_LM" Directory="Scripts" On="uninstall" />
+                
+                <Environment Id="PATH_LM" Action="set" Name="PATH" Part="first" Value="[InstallDirectory]" System="yes" />
+                <Environment Id="SCRIPTS_PATH_LM" Action="set" Name="PATH" Part="first" Value="[Scripts]" System="yes" />
+                <Environment Id="PY_PATHEXT_LM" Action="set" Name="PATHEXT" Part="last" Value=".PY" System="yes" />
+                <Environment Id="PYW_PATHEXT_LM" Action="set" Name="PATHEXT" Part="last" Value=".PYW" System="yes" />
+            </Component>
+        </Feature>
+    </Product>
+</Wix>
+
diff --git a/Tools/msi/path/path_en-US.wxl b/Tools/msi/path/path_en-US.wxl
new file mode 100644
--- /dev/null
+++ b/Tools/msi/path/path_en-US.wxl
@@ -0,0 +1,6 @@
+?<?xml version="1.0" encoding="utf-8"?>
+<WixLocalization Culture="en-us" xmlns="http://schemas.microsoft.com/wix/2006/localization">
+    <String Id="Descriptor">Add to Path</String>
+    <String Id="ShortDescriptor">Path</String>
+    <String Id="NoPython">No !(loc.ProductName) installation was detected.</String>
+</WixLocalization>
diff --git a/Tools/msi/pip/pip.wixproj b/Tools/msi/pip/pip.wixproj
new file mode 100644
--- /dev/null
+++ b/Tools/msi/pip/pip.wixproj
@@ -0,0 +1,19 @@
+<?xml version="1.0" encoding="utf-8"?>
+<Project ToolsVersion="4.0" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
+    <PropertyGroup>
+        <ProjectGuid>{91C99298-8E2E-4422-A5AF-CC4FFF9A58D3}</ProjectGuid>
+        <SchemaVersion>2.0</SchemaVersion>
+        <OutputName>pip</OutputName>
+        <OutputType>Package</OutputType>
+        <SuppressIces>ICE71</SuppressIces>
+    </PropertyGroup>
+    <Import Project="..\msi.props" />
+    <ItemGroup>
+        <Compile Include="*.wxs" />
+    </ItemGroup>
+    <ItemGroup>
+        <EmbeddedResource Include="*.wxl" />
+    </ItemGroup>
+    
+    <Import Project="..\msi.targets" />
+</Project>
\ No newline at end of file
diff --git a/Tools/msi/pip/pip.wxs b/Tools/msi/pip/pip.wxs
new file mode 100644
--- /dev/null
+++ b/Tools/msi/pip/pip.wxs
@@ -0,0 +1,41 @@
+?<?xml version="1.0" encoding="UTF-8"?>
+<Wix xmlns="http://schemas.microsoft.com/wix/2006/wi">
+    <Product Id="*" Language="!(loc.LCID)" Name="!(loc.Title)" Version="$(var.Version)" Manufacturer="!(loc.Manufacturer)" UpgradeCode="$(var.UpgradeCode)">
+        <Package InstallerVersion="300" Compressed="yes" InstallScope="perUser" Platform="$(var.Platform)" />
+        
+        <PropertyRef Id="UpgradeTable" />
+        <PropertyRef Id="REGISTRYKEY" />
+        
+        <Property Id="PYTHON_EXE" Secure="yes">
+            <ComponentSearch Id="PythonExe" Guid="$(var.PythonExeComponentGuid)">
+                <FileSearch Name="python.exe" />
+            </ComponentSearch>
+        </Property>
+
+        <Condition Message="!(loc.NoPython)">PYTHON_EXE</Condition>
+        
+        <Feature Id="DefaultFeature" AllowAdvertise="no" Title="!(loc.Title)" Description="!(loc.Description)">
+            <Component Id="pip" Guid="*" Directory="InstallDirectory">
+                <RegistryValue KeyPath="yes" Root="HKMU" Key="[REGISTRYKEY]\EnsurePipRun" Value="1" Type="integer" />
+            </Component>
+        </Feature>
+        
+        <?if $(var.Platform)~="x64" ?>
+        <CustomAction Id="UpdatePip" BinaryKey="WixCA" DllEntry="CAQuietExec64" Execute="deferred" Return="ignore"/>
+        <?else ?>
+        <CustomAction Id="UpdatePip" BinaryKey="WixCA" DllEntry="CAQuietExec" Execute="deferred" Return="ignore"/>
+        <?endif ?>
+        
+        <!-- Install/uninstall pip -->
+        <CustomAction Id="SetUpdatePipCommandLine" Property="UpdatePip" Value='"[PYTHON_EXE]" -m ensurepip -U --default-pip' Execute="immediate" />
+        <CustomAction Id="SetRemovePipCommandLine" Property="UpdatePip" Value='"[PYTHON_EXE]" -B -m ensurepip._uninstall' Execute="immediate" />
+        
+        <InstallExecuteSequence>
+            <Custom Action="SetUpdatePipCommandLine" Before="UpdatePip">(&amp;DefaultFeature=3) AND NOT (!DefaultFeature=3)</Custom>
+            <Custom Action="SetRemovePipCommandLine" Before="UpdatePip">(&amp;DefaultFeature=2) AND (!DefaultFeature=3)</Custom>
+
+            <Custom Action="UpdatePip" Before="InstallFinalize">UpdatePip</Custom>
+        </InstallExecuteSequence>
+    </Product>
+</Wix>
+
diff --git a/Tools/msi/pip/pip_en-US.wxl b/Tools/msi/pip/pip_en-US.wxl
new file mode 100644
--- /dev/null
+++ b/Tools/msi/pip/pip_en-US.wxl
@@ -0,0 +1,6 @@
+?<?xml version="1.0" encoding="utf-8"?>
+<WixLocalization Culture="en-us" xmlns="http://schemas.microsoft.com/wix/2006/localization">
+    <String Id="Descriptor">pip Bootstrap</String>
+    <String Id="ShortDescriptor">pip</String>
+    <String Id="NoPython">No !(loc.ProductName) installation was detected.</String>
+</WixLocalization>
diff --git a/Tools/msi/schema.py b/Tools/msi/schema.py
deleted file mode 100644
--- a/Tools/msi/schema.py
+++ /dev/null
@@ -1,1007 +0,0 @@
-from msilib import Table
-
-_Validation = Table('_Validation')
-_Validation.add_field(1,'Table',11552)
-_Validation.add_field(2,'Column',11552)
-_Validation.add_field(3,'Nullable',3332)
-_Validation.add_field(4,'MinValue',4356)
-_Validation.add_field(5,'MaxValue',4356)
-_Validation.add_field(6,'KeyTable',7679)
-_Validation.add_field(7,'KeyColumn',5378)
-_Validation.add_field(8,'Category',7456)
-_Validation.add_field(9,'Set',7679)
-_Validation.add_field(10,'Description',7679)
-
-ActionText = Table('ActionText')
-ActionText.add_field(1,'Action',11592)
-ActionText.add_field(2,'Description',7936)
-ActionText.add_field(3,'Template',7936)
-
-AdminExecuteSequence = Table('AdminExecuteSequence')
-AdminExecuteSequence.add_field(1,'Action',0x2DFF)
-AdminExecuteSequence.add_field(2,'Condition',7679)
-AdminExecuteSequence.add_field(3,'Sequence',5378)
-
-Condition = Table('Condition')
-Condition.add_field(1,'Feature_',11558)
-Condition.add_field(2,'Level',9474)
-Condition.add_field(3,'Condition',7679)
-
-AdminUISequence = Table('AdminUISequence')
-AdminUISequence.add_field(1,'Action',0x2DFF)
-AdminUISequence.add_field(2,'Condition',7679)
-AdminUISequence.add_field(3,'Sequence',5378)
-
-AdvtExecuteSequence = Table('AdvtExecuteSequence')
-AdvtExecuteSequence.add_field(1,'Action',0x2DFF)
-AdvtExecuteSequence.add_field(2,'Condition',7679)
-AdvtExecuteSequence.add_field(3,'Sequence',5378)
-
-AdvtUISequence = Table('AdvtUISequence')
-AdvtUISequence.add_field(1,'Action',11592)
-AdvtUISequence.add_field(2,'Condition',7679)
-AdvtUISequence.add_field(3,'Sequence',5378)
-
-AppId = Table('AppId')
-AppId.add_field(1,'AppId',11558)
-AppId.add_field(2,'RemoteServerName',7679)
-AppId.add_field(3,'LocalService',7679)
-AppId.add_field(4,'ServiceParameters',7679)
-AppId.add_field(5,'DllSurrogate',7679)
-AppId.add_field(6,'ActivateAtStorage',5378)
-AppId.add_field(7,'RunAsInteractiveUser',5378)
-
-AppSearch = Table('AppSearch')
-AppSearch.add_field(1,'Property',11592)
-AppSearch.add_field(2,'Signature_',11592)
-
-Property = Table('Property')
-Property.add_field(1,'Property',11592)
-Property.add_field(2,'Value',3840)
-
-BBControl = Table('BBControl')
-BBControl.add_field(1,'Billboard_',11570)
-BBControl.add_field(2,'BBControl',11570)
-BBControl.add_field(3,'Type',3378)
-BBControl.add_field(4,'X',1282)
-BBControl.add_field(5,'Y',1282)
-BBControl.add_field(6,'Width',1282)
-BBControl.add_field(7,'Height',1282)
-BBControl.add_field(8,'Attributes',4356)
-BBControl.add_field(9,'Text',7986)
-
-Billboard = Table('Billboard')
-Billboard.add_field(1,'Billboard',11570)
-Billboard.add_field(2,'Feature_',3366)
-Billboard.add_field(3,'Action',7474)
-Billboard.add_field(4,'Ordering',5378)
-
-Feature = Table('Feature')
-Feature.add_field(1,'Feature',11558)
-Feature.add_field(2,'Feature_Parent',7462)
-Feature.add_field(3,'Title',8000)
-Feature.add_field(4,'Description',8191)
-Feature.add_field(5,'Display',5378)
-Feature.add_field(6,'Level',1282)
-Feature.add_field(7,'Directory_',0x1DFF)
-Feature.add_field(8,'Attributes',1282)
-
-Binary = Table('Binary')
-Binary.add_field(1,'Name',11592)
-Binary.add_field(2,'Data',2304)
-
-BindImage = Table('BindImage')
-BindImage.add_field(1,'File_',0x2DFF)
-BindImage.add_field(2,'Path',7679)
-
-File = Table('File')
-File.add_field(1,'File',0x2DFF)
-File.add_field(2,'Component_',0xDFF)
-File.add_field(3,'FileName',4095)
-File.add_field(4,'FileSize',260)
-File.add_field(5,'Version',0x1DFF)
-File.add_field(6,'Language',7444)
-File.add_field(7,'Attributes',5378)
-File.add_field(8,'Sequence',1282)
-
-CCPSearch = Table('CCPSearch')
-CCPSearch.add_field(1,'Signature_',11592)
-
-CheckBox = Table('CheckBox')
-CheckBox.add_field(1,'Property',11592)
-CheckBox.add_field(2,'Value',7488)
-
-Class = Table('Class')
-Class.add_field(1,'CLSID',11558)
-Class.add_field(2,'Context',11552)
-Class.add_field(3,'Component_',0x2DFF)
-Class.add_field(4,'ProgId_Default',7679)
-Class.add_field(5,'Description',8191)
-Class.add_field(6,'AppId_',7462)
-Class.add_field(7,'FileTypeMask',7679)
-Class.add_field(8,'Icon_',7496)
-Class.add_field(9,'IconIndex',5378)
-Class.add_field(10,'DefInprocHandler',7456)
-Class.add_field(11,'Argument',7679)
-Class.add_field(12,'Feature_',3366)
-Class.add_field(13,'Attributes',5378)
-
-Component = Table('Component')
-Component.add_field(1,'Component',0x2DFF)
-Component.add_field(2,'ComponentId',7462)
-Component.add_field(3,'Directory_',0xDFF)
-Component.add_field(4,'Attributes',1282)
-Component.add_field(5,'Condition',7679)
-Component.add_field(6,'KeyPath',0x1DFF)
-
-Icon = Table('Icon')
-Icon.add_field(1,'Name',11592)
-Icon.add_field(2,'Data',2304)
-
-ProgId = Table('ProgId')
-ProgId.add_field(1,'ProgId',11775)
-ProgId.add_field(2,'ProgId_Parent',7679)
-ProgId.add_field(3,'Class_',7462)
-ProgId.add_field(4,'Description',8191)
-ProgId.add_field(5,'Icon_',7496)
-ProgId.add_field(6,'IconIndex',5378)
-
-ComboBox = Table('ComboBox')
-ComboBox.add_field(1,'Property',11592)
-ComboBox.add_field(2,'Order',9474)
-ComboBox.add_field(3,'Value',3392)
-ComboBox.add_field(4,'Text',8000)
-
-CompLocator = Table('CompLocator')
-CompLocator.add_field(1,'Signature_',11592)
-CompLocator.add_field(2,'ComponentId',3366)
-CompLocator.add_field(3,'Type',5378)
-
-Complus = Table('Complus')
-Complus.add_field(1,'Component_',0x2DFF)
-Complus.add_field(2,'ExpType',13570)
-
-Directory = Table('Directory')
-Directory.add_field(1,'Directory',0x2DFF)
-Directory.add_field(2,'Directory_Parent',0x1DFF)
-Directory.add_field(3,'DefaultDir',4095)
-
-Control = Table('Control')
-Control.add_field(1,'Dialog_',11592)
-Control.add_field(2,'Control',11570)
-Control.add_field(3,'Type',3348)
-Control.add_field(4,'X',1282)
-Control.add_field(5,'Y',1282)
-Control.add_field(6,'Width',1282)
-Control.add_field(7,'Height',1282)
-Control.add_field(8,'Attributes',4356)
-Control.add_field(9,'Property',7474)
-Control.add_field(10,'Text',7936)
-Control.add_field(11,'Control_Next',7474)
-Control.add_field(12,'Help',7986)
-
-Dialog = Table('Dialog')
-Dialog.add_field(1,'Dialog',11592)
-Dialog.add_field(2,'HCentering',1282)
-Dialog.add_field(3,'VCentering',1282)
-Dialog.add_field(4,'Width',1282)
-Dialog.add_field(5,'Height',1282)
-Dialog.add_field(6,'Attributes',4356)
-Dialog.add_field(7,'Title',8064)
-Dialog.add_field(8,'Control_First',3378)
-Dialog.add_field(9,'Control_Default',7474)
-Dialog.add_field(10,'Control_Cancel',7474)
-
-ControlCondition = Table('ControlCondition')
-ControlCondition.add_field(1,'Dialog_',11592)
-ControlCondition.add_field(2,'Control_',11570)
-ControlCondition.add_field(3,'Action',11570)
-ControlCondition.add_field(4,'Condition',11775)
-
-ControlEvent = Table('ControlEvent')
-ControlEvent.add_field(1,'Dialog_',11592)
-ControlEvent.add_field(2,'Control_',11570)
-ControlEvent.add_field(3,'Event',11570)
-ControlEvent.add_field(4,'Argument',11775)
-ControlEvent.add_field(5,'Condition',15871)
-ControlEvent.add_field(6,'Ordering',5378)
-
-CreateFolder = Table('CreateFolder')
-CreateFolder.add_field(1,'Directory_',0x2DFF)
-CreateFolder.add_field(2,'Component_',0x2DFF)
-
-CustomAction = Table('CustomAction')
-CustomAction.add_field(1,'Action',0x2DFF)
-CustomAction.add_field(2,'Type',1282)
-CustomAction.add_field(3,'Source',0x1DFF)
-CustomAction.add_field(4,'Target',7679)
-
-DrLocator = Table('DrLocator')
-DrLocator.add_field(1,'Signature_',11592)
-DrLocator.add_field(2,'Parent',15688)
-DrLocator.add_field(3,'Path',15871)
-DrLocator.add_field(4,'Depth',5378)
-
-DuplicateFile = Table('DuplicateFile')
-DuplicateFile.add_field(1,'FileKey',11592)
-DuplicateFile.add_field(2,'Component_',0xDFF)
-DuplicateFile.add_field(3,'File_',0xDFF)
-DuplicateFile.add_field(4,'DestName',8191)
-DuplicateFile.add_field(5,'DestFolder',7496)
-
-Environment = Table('Environment')
-Environment.add_field(1,'Environment',11592)
-Environment.add_field(2,'Name',4095)
-Environment.add_field(3,'Value',8191)
-Environment.add_field(4,'Component_',0xDFF)
-
-Error = Table('Error')
-Error.add_field(1,'Error',9474)
-Error.add_field(2,'Message',7936)
-
-EventMapping = Table('EventMapping')
-EventMapping.add_field(1,'Dialog_',11592)
-EventMapping.add_field(2,'Control_',11570)
-EventMapping.add_field(3,'Event',11570)
-EventMapping.add_field(4,'Attribute',3378)
-
-Extension = Table('Extension')
-Extension.add_field(1,'Extension',11775)
-Extension.add_field(2,'Component_',0x2DFF)
-Extension.add_field(3,'ProgId_',7679)
-Extension.add_field(4,'MIME_',7488)
-Extension.add_field(5,'Feature_',3366)
-
-MIME = Table('MIME')
-MIME.add_field(1,'ContentType',11584)
-MIME.add_field(2,'Extension_',3583)
-MIME.add_field(3,'CLSID',7462)
-
-FeatureComponents = Table('FeatureComponents')
-FeatureComponents.add_field(1,'Feature_',11558)
-FeatureComponents.add_field(2,'Component_',0x2DFF)
-
-FileSFPCatalog = Table('FileSFPCatalog')
-FileSFPCatalog.add_field(1,'File_',0x2DFF)
-FileSFPCatalog.add_field(2,'SFPCatalog_',11775)
-
-SFPCatalog = Table('SF