Mailman 3 August 2015 - Python-Dev

How To Forward-Merge Your Change After Your Pull Request Is Accepted Into Python 3.5.0rcX
by Larry Hastings Aug. 27, 2015

Aug. 27, 2015

Now that we're in the "release candidate" phase of Python 3.5.0, the workflow has changed a little. We're trying an experiment using Bitbucket and pull requests. You can read about that workflow, here: https://mail.python.org/pipermail/python-dev/2015-August/141167.html But the instructions for that workflow are pretty hazy on what you do after your pull request is accepted. This message is an addendum to those instructions, describing exactly what you should do after your pull request is … [View More]accepted. To save wear and tear on my hands (and your eyes), for the rest of these instructions, I'm going to refer to each place-you-can-check-things-in-to by version number as follows: 3.4 : hg.python.org/cpython (branch "3.4") 3.5.0 : https://bitbucket.org/larry/cpython350 (branch "3.5") 3.5.1 : hg.python.org/cpython (branch "3.5") 3.6 : hg.python.org/cpython (branch "default") With that nomenclature established I can now precisely say: when your pull request is accepted into 3.5.0, you need to merge from 3.5.0 into 3.5.1, and then from 3.5.1 into 3.6. Doing this is much like the existing workflow. You use "hg merge" to merge your changes from previous versions into subsequent versions (what I call "forward merging"). What complicates matters is the fact that the 3.5.0 release candidates don't live in the normal repo--they lives in a repo on Bitbucket which is only writeable by me. In order to keep a tight lid on the changes checked in to the 3.5.0 release candidates, I won't pull revisions from the normal CPython repo. (If I did, I'd also pull in changes intended for 3.5.1, and... it'd be a mess.) So here come the instructions. They look long, but that's just because I go into a lot of detail to try and make them as foolproof as possible. They aren't really much longer or more complicated than the steps you already follow to perform forward-merges. Note that these are easy, guaranteed-clean instructions on how to perform the merge. Are there shortcuts you could take that might speed things up? Yes. But I encourage you to skip those shortcuts and stick to my instructions. Working with multiple branches is complicated enough, and this external repo makes things even more complicated. It's all too easy to make a mistake. HOW TO FORWARD-MERGE FROM 3.5.0 TO 3.5.1 ---------------------------------------- 1: Wait until your pull request has been accepted. 2: Make a *clean* local clone of the CPython tree, updated to the "3.5" branch. In my instructions I'll call the clone "cpython351-merge": % hg clone ssh://hg@hg.python.org/cpython -u 3.5 cpython351-merge % cd cpython351-merge 3: Confirm that you're in the correct branch. You should be in the "3.5" branch. Run this command: % hg id Let's assume that the current head in the "3.5" branch has changeset ID "7890abcdef". If that were true, the output of "hg id" would look like this: 7890abcdef (3.5) It might also say "tip" on the end, like this: 7890abcdef (3.5) tip If it doesn't say "3.5", switch to the 3.5 branch: % hg up -r 3.5 and repeat this step. 4: Pull from the 3.5.0 repo into your "cpython351-merge" directory. % hg pull ssh://hg@bitbucket.org/larry/cpython350 You should now have two "heads" in the 3.5 branch; the existing head you saw in the previous step, and the new head you just pulled in, which should be the changeset from your pull request. 5: As an optional step: confirm you have the correct two heads. This command will print a list of all the heads in the current repo: % hg heads Again, you should have exactly two identified as being on the "3.5" branch; one should have the changeset ID shown by "hg id" in step 3, the other should be your change from the pull request. 6: Merge the two heads together: % hg merge If there are merge conflicts, Mercurial will guide you through the conflict resolution process as normal. 7: Make sure that all your changes merged properly and you didn't merge anything else by accident. I run these two commands: % hg stat % hg diff and read all the output. 8: Make sure Misc/NEWS has your update in the right spot. (See below.) 9: Check in. The checkin comment should be something like Merge from 3.5.0 to 3.5.1. 10: Push your changes back into the main CPython repo. % hg push 11: Now forward-merge your change to 3.6 as normal, following the CPython Dev Guide instructions: https://docs.python.org/devguide/committing.html#merging-between-different-… FREQUENTLY ASKED QUESTIONS -------------------------- Q: I screwed something up! What do I do now? If you haven't pushed your changes out, it's no problem. Just delete your repo and start over. If you *have* pushed your changes out, obviously we'll need to fix the mistake. If you're not sure how to fix the problem, I suggest logging in to the #python-dev IRC channel and asking for help. Q: What do I need to do about Misc/NEWS? I'm glad you asked! First, you *must* put your Misc/NEWS update into the correct section. If you're creating a pull request for Python 3.5.0 rc-something, put it in the 3.5.0 rc-something section. If you're checking in to 3.5.1, put it in the 3.5.1 section. If you're just checking into 3.6, put it in the 3.6.0 alpha 1 section. Second, when you merge forward, make sure the merge tool puts your Misc/NEWS entry in the right section. The merge tool I seem to use isn't particularly smart about this, so I've had to manually edit Misc/NEWS many times to fix it. (When I released 3.5.0rc2, I had to do a lot of cleanup on Misc/NEWS, and again in the 3.5.1 branch, and again in 3.6.) Every time you merge forward, make sure your Misc/NEWS entry is in the right spot. Q: What if a second pull request is accepted before I get around to doing the merge? Well, *someone* needs to merge, and they're going to have to merge *both* changes. I can't come up with a good general policy here. Hopefully this won't happen often; for now let's just handle it on a case-by-case basis. Q: What if I have a bugfix for 3.4 that I want to ship with 3.5.0? You have to check in twice, and merge-forward twice. First, check in to 3.4, then merge forward into 3.5.1 and 3.6. Then, once your pull request is accepted into 3.5.0, do a "null merge" (a merge where no files are changed) from 3.5.0 into 3.5.1 and 3.6. Q: What if my pull request is turned down? If your bug fix isn't critical enough to merit shipping with 3.5.0, just check it into the normal 3.5 branch on hg.python.org and it'll ship with 3.5.1. (And, naturally, forward-merge it into 3.6.) //arry/ [View Less]

1 0

Python Issue-subprocess problem
by Low, Wai HoeX Aug. 27, 2015

Aug. 27, 2015

Dear, I have faced a problem as below when I startup the python IDLE [cid:image001.png@01D0E0CE.454096D0] Then I try to run a simple program like this: [cid:image002.png@01D0E0CE.454096D0] Other issue is pop out [cid:image003.png@01D0E0CE.454096D0] May I know how to fix the issue? My laptop is window 7 64bits. Thanks and having a nice day ! Regards, Arthur Low Wai Hoe

3 2

django_v2 benchmark compatibility fix for Python 3.6
by Papa, Florin Aug. 26, 2015

Aug. 26, 2015

Hi All, My name is Florin Papa and I work in the Server Languages Optimizations Team at Intel Corporation. I would like to submit a patch that solves compatibility issues of the django_v2 benchmark in the Grand Unified Python Benchmark. The django_v2 benchmark uses inspect.getargspec(), which is deprecated and was removed in Python 3.6. Therefore, it crashes with the message "ImportError: cannot import name 'getargspec'" when using the latest version of Python on the default branch. The … [View More]

4 5

[RELEASED] Python 3.5.0rc2 is now available
by Larry Hastings Aug. 25, 2015

Aug. 25, 2015

On behalf of the Python development community and the Python 3.5 release team, I'm relieved to announce the availability of Python 3.5.0rc2, also known as Python 3.5.0 Release Candidate 2. Python 3.5 has now entered "feature freeze". By default new features may no longer be added to Python 3.5. This is a preview release, and its use is not recommended for production settings. You can find Python 3.5.0rc2 here: https://www.python.org/downloads/release/python-350rc2/ Windows and … [View More]

1 0

Profile Guided Optimization active by-default
by Patrascu, Alecsandru Aug. 25, 2015

Aug. 25, 2015

Hi All, This is Alecsandru from Server Scripting Languages Optimization team at Intel Corporation. I would like to submit a request to turn-on Profile Guided Optimization or PGO as the default build option for Python (both 2.7 and 3.6), given its performance benefits on a wide variety of workloads and hardware. For instance, as shown from attached sample performance results from the Grand Unified Python Benchmark, >20% speed up was observed. In addition, we are seeing 2-9% performance … [View More]boost from OpenStack/Swift where more than 60% of the codes are in Python 2.7. Our analysis indicates the performance gain was mainly due to reduction of icache misses and CPU front-end stalls. Attached is the Makefile patches that modify the all build target and adds a new one called "disable-profile-opt". We built and tested this patch for Python 2.7 and 3.6 on our Linux machines (CentOS 7/Ubuntu Server 14.04, Intel Xeon Haswell/Broadwell with 18/8 cores). We use "regrtest" suite for training as it provides the best performance improvement. Some of the test programs in the suite may fail which leads to build fail. One solution is to disable the specific failed test using the "-x " flag (as shown in the patch) Steps to apply the patch: 1. hg clone https://hg.python.org/cpython cpython 2. cd cpython 3. hg update 2.7 (needed for 2.7 only) 4. Copy *.patch to the current directory 5. patch < python2.7-pgo.patch (or patch < python3.6-pgo.patch) 6. ./configure 7. make To disable PGO 7b. make disable-profile-opt In the following, please find our sample performance results from latest XEON machine, XEON Broadwell EP. Hardware (HW): Intel XEON (Broadwell) 8 Cores BIOS settings: Intel Turbo Boost Technology: false Hyper-Threading: false Operating System: Ubuntu 14.04.3 LTS trusty OS configuration: CPU freq set at fixed: 2.6GHz by echo 2600000 > /sys/devices/system/cpu/cpu*/cpufreq/scaling_min_freq echo 2600000 > /sys/devices/system/cpu/cpu*/cpufreq/scaling_max_freq Address Space Layout Randomization (ASLR) disabled (to reduce run to run variation) by echo 0 > /proc/sys/kernel/randomize_va_space GCC version: gcc version 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04) Benchmark: Grand Unified Python Benchmark (GUPB) GUPB Source: https://hg.python.org/benchmarks/ Python2.7 results: Python source: hg clone https://hg.python.org/cpython cpython Python Source: hg update 2.7 hg id: 0511b1165bb6 (2.7) hg id -r 'ancestors(.) and tag()': 15c95b7d81dc (2.7) v2.7.10 hg --debug id -i: 0511b1165bb6cf40ada0768a7efc7ba89316f6a5 Benchmarks Speedup(%) simple_logging 20 raytrace 20 silent_logging 19 richards 19 chaos 16 formatted_logging 16 json_dump 15 hexiom2 13 pidigits 12 slowunpickle 12 django_v2 12 unpack_sequence 11 float 11 mako 11 slowpickle 11 fastpickle 11 django 11 go 10 json_dump_v2 10 pathlib 10 regex_compile 10 pybench 9.9 etree_process 9 regex_v8 8 bzr_startup 8 2to3 8 slowspitfire 8 telco 8 pickle_list 8 fannkuch 8 etree_iterparse 8 nqueens 8 mako_v2 8 etree_generate 8 call_method_slots 7 html5lib_warmup 7 html5lib 7 nbody 7 spectral_norm 7 spambayes 7 fastunpickle 6 meteor_contest 6 chameleon 6 rietveld 6 tornado_http 5 unpickle_list 5 pickle_dict 4 regex_effbot 3 normal_startup 3 startup_nosite 3 etree_parse 2 call_method_unknown 2 call_simple 1 json_load 1 call_method 1 Python3.6 results Python source: hg clone https://hg.python.org/cpython cpython hg id: 96d016f78726 tip hg id -r 'ancestors(.) and tag()': 1a58b1227501 (3.5) v3.5.0rc1 hg --debug id -i: 96d016f78726afbf66d396f084b291ea43792af1 Benchmark Speedup(%) fastunpickle 22.94 fastpickle 21.67 json_load 17.64 simple_logging 17.49 meteor_contest 16.67 formatted_logging 15.33 etree_process 14.61 raytrace 13.57 etree_generate 13.56 chaos 12.09 hexiom2 12 nbody 11.88 json_dump_v2 11.24 richards 11.02 nqueens 10.96 fannkuch 10.79 go 10.77 float 10.26 regex_compile 9.8 silent_logging 9.63 pidigits 9.58 etree_iterparse 9.48 2to3 8.44 regex_v8 8.09 regex_effbot 7.88 call_simple 7.63 tornado_http 7.38 etree_parse 4.92 spectral_norm 4.72 normal_startup 4.39 telco 3.88 startup_nosite 3.7 call_method 3.63 unpack_sequence 3.6 call_method_slots 2.91 call_method_unknown 2.59 iterative_count 0.45 threaded_count -2.79 Thank you, Alecsandru [View Less]

11 27

Re: [Python-Dev] Profile Guided Optimization active by-default
by Stewart, David C Aug. 25, 2015

Aug. 25, 2015

(Sorry about the format here - I honestly just subscribed to Python-dev so be gentle ...) > Date: Sat, 22 Aug 2015 11:25:59 -0600 > From: Eric Snow <ericsnowcurrently(a)gmail.com> >On Aug 22, 2015 9:02 AM, "Patrascu, Alecsandru" <alecsandru.patrascu at >intel.com <https://mail.python.org/mailman/listinfo/python-dev>> >wrote:[snip]> For instance, as shown from attached sample performance >results from theGrand Unified Python Benchmark, >20% speed up … [View More]was >observed. > > Eric I'm the manager of Intel's server scripting language optimization team, so I'll answer from that perspective. >Are you referring to the tests in the benchmarks repo? [1] How does the >real-world performance improvement compare with otherlanguages you are >targeting for optimization? Yes, we're using [1]. We're seeing up to 10% improvement on Swift (a project in OpenStack) on some architectures using the ssbench workload, which is as close to real-world as we can get. Relative to other languages we target, this is quite good actually. For example, Java's Hotspot JIT is driven by profiling at its core so it's hard to distinguish the value profiling alone brings. We have seen a nice boost on PHP running Wordpress using PGO, but not as impressive as Python and Swift. By the way, I think letting the compiler optimize the code is a good strategy. Not the only strategy we want to use, but it seems like one we could do more of. > And thanks for working on this! I have several more questions: What >sorts of future changes in CPython's code might interfere with >youroptimizations? > > We're also looking at other source-level optimizations, like the CGOTO patch Vamsi submitted in June. Some of these may reduce the value of PGO, but in general it's nice to let the compiler do some optimization for you. > What future additions might stand to benefit? > It's a good question. Our intent is to continue to evaluate and measure different training workloads for improvement. In other words, as with any good open source project, this patch should improve things a lot and should be accepted upstream, but we will continue to make it better. > What changes in existing code might improve optimization opportunities? > > We intend to continue to work on source-level optimizations and measuring them against GUPB and Swift. > What is the added maintenance burden of the optimizations on CPython, >ifany? > > I think the answer is none. Our goal was to introduce performance improvements without adding to maintenance effort. >What is the performance impact on non-Intel architectures? What >aboutolder Intel architectures? ...and future ones? > > We should modify the patch to make it for Intel only, since we're not evaluating non-Intel architectures. Unfortunately for us, I suspect that older Intel CPUs might benefit more than current and future ones. Future architectures will benefit from other enabling work we're planning. > What is Intel's commitment to supporting these (or other) optimizations >inthe future? How is the practical EOL of the optimizations managed? > > As with any corporation's budgeting process, it's hard to know exactly what my managers will let me spend money on. :-) But we're definitely convinced of the value of dynamic languages for servers and the need to work on optimization. As far as I have visibility, it appears to be holding true. > Finally, +1 on adding an opt-in Makefile target rather than enabling >theoptimizations by default. > > Frankly since Ubuntu has been running this way for past two years, I think it's fine to make it opt-in, but eventually I hope it can be the default once we're happy with it. > Thanks again! -eric [View Less]

3 2

How are we merging forward from the Bitbucket 3.5 repo?
by Larry Hastings Aug. 24, 2015

Aug. 24, 2015

So far I've accepted two pull requests into bitbucket.com/larry/cpython350 in the 3.5 branch, what will become 3.5.0rc2. As usual, it's the contributor's responsibility to merge forward; if their checkin goes in to 3.5, it's their responsibility to also merge it into the hg.python.org/cpython 3.5 branch (which will be 3.5.1) and default branch (which right now is 3.6). But... what's the plan here? I didn't outline anything specific, I just assumed we'd do the normal thing, pulling from … [View More]

6 9

Summary of Python tracker Issues
by Python tracker Aug. 21, 2015

Aug. 21, 2015

ACTIVITY SUMMARY (2015-08-14 - 2015-08-21) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 5013 (+11) closed 31656 (+29) total 36669 (+40) Open issues with patches: 2241 Issues opened (33) ================== #21167: float('nan') returns 0.0 on Python compiled with icc http://bugs.python.org/issue21167 reopened by r.david.murray #21192: Idle: Print … [View More]

1 0

PEP-498: Literal String Formatting
by Eric V. Smith Aug. 21, 2015

Aug. 21, 2015

Following a long discussion on python-ideas, I've posted my draft of PEP-498. It describes the "f-string" approach that was the subject of the "Briefer string format" thread. I'm open to a better title than "Literal String Formatting". I need to add some text to the discussion section, but I think it's in reasonable shape. I have a fully working implementation that I'll get around to posting somewhere this weekend. >>> def how_awesome(): return 'very' ... >>> f'f-strings are … [View More]

30 96

Burning down the backlog.
by Robert Collins Aug. 18, 2015

Aug. 18, 2015

On 21 July 2015 at 19:40, Nick Coghlan <ncoghlan(a)gmail.com> wrote: > All of this is why the chart that I believe should be worrying people > is the topmost one on this page: > http://bugs.python.org/issue?@template=stats > > Both the number of open issues and the number of open issues with > patches are steadily trending upwards. That means the bottleneck in > the current process *isn't* getting patches written in the first > place, it's getting them up to the … [View More]

10 13