Mailman 3 February 2017 - Python-Dev

Deterministic builds of the interpreter
by Freddy Rietdijk Feb. 19, 2017

Feb. 19, 2017

Hi, I'm attempting to make the builds of the Python interpreters for Nixpkgs [1] deterministic. In the case of Python 2.7 we have a patch [2] that fixes the timestamp used in .pyc files in case the env var `DETERMINISTIC_BUILD` is set. We also remove `wininst*.exe`. This works fine, although there are 4 small issues left [3]. Do you have any idea what is going on in these files that could make them indeterministic? For Python 3.x I disabled ensurepip, removed `wininst*.exe`, and modified `py_compile` to use `0` instead of `source_stats['mtime']`. The builds are not yet deterministic [4]. Any suggestions what could be fixed here? Kind regards, Freddy [1] https://github.com/NixOS/nixpkgs [2] https://github.com/NixOS/nixpkgs/blob/1da6775/pkgs/development/interpreters… [3] https://github.com/NixOS/nixpkgs/issues/22570#issuecomment-278474082 [4] https://gist.github.com/anonymous/7cc147af6511dee2dc5a5b8d110f0e6b

3 9

Summary of Python tracker Issues
by Python tracker Feb. 17, 2017

Feb. 17, 2017

ACTIVITY SUMMARY (2017-02-10 - 2017-02-17) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 5837 (+38) closed 35511 (+28) total 41348 (+66) Open issues with patches: 2488 Issues opened (51) ================== #29442: Use argparse and drop dirty optparse hacks in setup.py http://bugs.python.org/issue29442 reopened by Chi Hsuan Yen #29529: Backport Travis configuration http://bugs.python.org/issue29529 opened by brett.cannon #29530: Windows buildbots broken by the migration to GitHub (meta issu http://bugs.python.org/issue29530 opened by haypo #29532: functools.partial is not compatible between 2.7 and 3.5 http://bugs.python.org/issue29532 opened by naoyuki #29533: urllib2 works slowly with proxy on windows http://bugs.python.org/issue29533 opened by juliadolgova #29535: datetime hash is deterministic in some cases http://bugs.python.org/issue29535 opened by arigo #29537: Alternative fix for BUILD_MAP_UNPACK_WITH_CALL opcode in 3.5 http://bugs.python.org/issue29537 opened by serhiy.storchaka #29539: [smtplib] collect response data for all recipients http://bugs.python.org/issue29539 opened by David Ford (FirefighterBlu3) #29540: Add compact=True flag to json.dump/dumps http://bugs.python.org/issue29540 opened by Alex Gordon #29541: Python3 error while building on Alt-F http://bugs.python.org/issue29541 opened by Honza Skýpala #29545: Python behavioral difference between Linux and AIX http://bugs.python.org/issue29545 opened by gireeshpunathil #29546: A more helpful ImportError message http://bugs.python.org/issue29546 opened by barry #29547: Add c code coverage to codecov http://bugs.python.org/issue29547 opened by belopolsky #29548: Recommend PyObject_Call* APIs over PyEval_Call*() APIs http://bugs.python.org/issue29548 opened by inada.naoki #29549: Improve docstring for str.index http://bugs.python.org/issue29549 opened by rhettinger #29550: Mac build-installer touch step fails after github conversion http://bugs.python.org/issue29550 opened by db3l #29551: unittest: TestSuite.debug() does not like subTest() http://bugs.python.org/issue29551 opened by wrohdewald #29553: Argparser does not display closing parentheses in nested mutex http://bugs.python.org/issue29553 opened by christofsteel #29554: profile/pstat doc clariification http://bugs.python.org/issue29554 opened by mbussonn #29555: Update Python Software Foundation Copyright Year http://bugs.python.org/issue29555 opened by orsenthil #29556: Remove unused #include <langinfo.h> http://bugs.python.org/issue29556 opened by Chi Hsuan Yen #29557: binhex documentation claims unknown bug http://bugs.python.org/issue29557 opened by David Wilemski #29558: Provide run_until_complete inside loop http://bugs.python.org/issue29558 opened by txomon #29559: Detect mouse over lines on canvas while mouse button is down http://bugs.python.org/issue29559 opened by Ted Shaneyfelt2 #29560: Turtle graphics fill behavior differs between versions http://bugs.python.org/issue29560 opened by John Simmons #29561: Interactive mode gives sys.ps2 not sys.ps1 after comment-only http://bugs.python.org/issue29561 opened by JDLH #29562: test_getgroups of test_posix fails (on OS X 10.10) http://bugs.python.org/issue29562 opened by JDLH #29564: ResourceWarning: suggest to enable tracemalloc in the message http://bugs.python.org/issue29564 opened by David Ford (FirefighterBlu3) #29565: Still broken ctypes calling convention on MSVC / 64-bit Window http://bugs.python.org/issue29565 opened by Igor Kudrin #29566: binhex() creates files with mixed line endings http://bugs.python.org/issue29566 opened by serhiy.storchaka #29568: undefined parsing behavior with the old style string formattin http://bugs.python.org/issue29568 opened by Jerry Dimitrov #29569: threading.Timer class: Continue periodical execution till acti http://bugs.python.org/issue29569 opened by slytomcat #29570: Windows Buildbot 2.7 is broken http://bugs.python.org/issue29570 opened by haypo #29571: test_re is failing when local is set for `en_IN` http://bugs.python.org/issue29571 opened by jaysinh.shukla #29572: Upgrade installers to OpenSSL 1.0.2k http://bugs.python.org/issue29572 opened by zach.ware #29573: NamedTemporaryFile with delete=True should not fail if file al http://bugs.python.org/issue29573 opened by richardxia #29574: python-3.6.0.tgz permissions borked http://bugs.python.org/issue29574 opened by Dave_Anderson #29575: doc 17.2.1: basic Pool example is too basic http://bugs.python.org/issue29575 opened by j5w6 #29577: Enum: mixin classes don't mix well with already mixed Enums http://bugs.python.org/issue29577 opened by ethan.furman #29579: Windows Python 3.7 installer broken by README.txt renamed to R http://bugs.python.org/issue29579 opened by haypo #29580: "Built-in Functions" not being functions http://bugs.python.org/issue29580 opened by Stefan Pochmann #29581: __init_subclass__ causes TypeError when used with standard lib http://bugs.python.org/issue29581 opened by Nate Soares #29582: Add Python coverage make target(s) http://bugs.python.org/issue29582 opened by zach.ware #29585: site.py imports relatively large `sysconfig` module. http://bugs.python.org/issue29585 opened by inada.naoki #29586: Cannot run pip in fresh install of py 3.5.3 http://bugs.python.org/issue29586 opened by Adrian Chan #29587: Generator/coroutine 'throw' discards exc_info state, which is http://bugs.python.org/issue29587 opened by njs #29588: importing ssl can fail with NameError: name 'PROTOCOL_TLS' is http://bugs.python.org/issue29588 opened by jdemeyer #29589: test_asyncio & test_multiprocessing_forkserver failed http://bugs.python.org/issue29589 opened by Dima Zhukov #29590: Incorrect stack traces when re-entering a generator/coroutine http://bugs.python.org/issue29590 opened by njs #29591: Various security vulnerabilities in bundled expat http://bugs.python.org/issue29591 opened by Natanael Copa #29592: abs_paths() in site.py is slow http://bugs.python.org/issue29592 opened by inada.naoki Most recent 15 issues with no replies (15) ========================================== #29592: abs_paths() in site.py is slow http://bugs.python.org/issue29592 #29590: Incorrect stack traces when re-entering a generator/coroutine http://bugs.python.org/issue29590 #29589: test_asyncio & test_multiprocessing_forkserver failed http://bugs.python.org/issue29589 #29581: __init_subclass__ causes TypeError when used with standard lib http://bugs.python.org/issue29581 #29575: doc 17.2.1: basic Pool example is too basic http://bugs.python.org/issue29575 #29572: Upgrade installers to OpenSSL 1.0.2k http://bugs.python.org/issue29572 #29570: Windows Buildbot 2.7 is broken http://bugs.python.org/issue29570 #29569: threading.Timer class: Continue periodical execution till acti http://bugs.python.org/issue29569 #29566: binhex() creates files with mixed line endings http://bugs.python.org/issue29566 #29560: Turtle graphics fill behavior differs between versions http://bugs.python.org/issue29560 #29559: Detect mouse over lines on canvas while mouse button is down http://bugs.python.org/issue29559 #29555: Update Python Software Foundation Copyright Year http://bugs.python.org/issue29555 #29554: profile/pstat doc clariification http://bugs.python.org/issue29554 #29551: unittest: TestSuite.debug() does not like subTest() http://bugs.python.org/issue29551 #29549: Improve docstring for str.index http://bugs.python.org/issue29549 Most recent 15 issues waiting for review (15) ============================================= #29571: test_re is failing when local is set for `en_IN` http://bugs.python.org/issue29571 #29568: undefined parsing behavior with the old style string formattin http://bugs.python.org/issue29568 #29557: binhex documentation claims unknown bug http://bugs.python.org/issue29557 #29555: Update Python Software Foundation Copyright Year http://bugs.python.org/issue29555 #29540: Add compact=True flag to json.dump/dumps http://bugs.python.org/issue29540 #29537: Alternative fix for BUILD_MAP_UNPACK_WITH_CALL opcode in 3.5 http://bugs.python.org/issue29537 #29533: urllib2 works slowly with proxy on windows http://bugs.python.org/issue29533 #29532: functools.partial is not compatible between 2.7 and 3.5 http://bugs.python.org/issue29532 #29523: latest setuptools breaks virtualenvs due to unbundling depende http://bugs.python.org/issue29523 #29509: redundant interning in PyObject_GetAttrString http://bugs.python.org/issue29509 #29507: Use FASTCALL in call_method() to avoid temporary tuple http://bugs.python.org/issue29507 #29476: Simplify set_add_entry() http://bugs.python.org/issue29476 #29475: option to not follow symlinks when globbing http://bugs.python.org/issue29475 #29469: AST-level Constant folding http://bugs.python.org/issue29469 #29465: Add _PyObject_FastCall() to reduce stack consumption http://bugs.python.org/issue29465 Top 10 most discussed issues (10) ================================= #29514: Add a test case that prevents magic number changes in minor re http://bugs.python.org/issue29514 30 msgs #29548: Recommend PyObject_Call* APIs over PyEval_Call*() APIs http://bugs.python.org/issue29548 23 msgs #29470: ssl: SNI callbacks should not modify context objects http://bugs.python.org/issue29470 14 msgs #29585: site.py imports relatively large `sysconfig` module. http://bugs.python.org/issue29585 11 msgs #29540: Add compact=True flag to json.dump/dumps http://bugs.python.org/issue29540 10 msgs #29564: ResourceWarning: suggest to enable tracemalloc in the message http://bugs.python.org/issue29564 10 msgs #29521: Minor warning messages when compiling documentation http://bugs.python.org/issue29521 9 msgs #29442: Use argparse and drop dirty optparse hacks in setup.py http://bugs.python.org/issue29442 8 msgs #29586: Cannot run pip in fresh install of py 3.5.3 http://bugs.python.org/issue29586 8 msgs #19675: Pool dies with excessive workers, but does not cleanup http://bugs.python.org/issue19675 7 msgs Issues closed (28) ================== #5667: Interpreter fails to initialize on build dir when IO encoding http://bugs.python.org/issue5667 closed by haypo #18779: Misleading documentations and comments in regular expression H http://bugs.python.org/issue18779 closed by akuchling #23109: French quotes in the documentation are often ungrammatical http://bugs.python.org/issue23109 closed by akuchling #27536: Convert readme to reStructuredText http://bugs.python.org/issue27536 closed by zach.ware #28929: Provide a link from documentation back to its source file http://bugs.python.org/issue28929 closed by Mariatta #28941: Update the link to Source Code in Python Docs from hg to githu http://bugs.python.org/issue28941 closed by Mariatta #29438: use after free in key sharing dict http://bugs.python.org/issue29438 closed by inada.naoki #29474: Grammatical errors in weakref.WeakValueDictionary docs http://bugs.python.org/issue29474 closed by Mariatta #29481: 3.6.0 doc describes 3.6.1 feature - typing.Deque http://bugs.python.org/issue29481 closed by Mariatta #29511: Add 'find' as build-in method for lists http://bugs.python.org/issue29511 closed by terry.reedy #29518: 'for' loop not automatically breaking (index error on line of http://bugs.python.org/issue29518 closed by brett.cannon #29524: Move functions to call objects into a new Objects/call.c file http://bugs.python.org/issue29524 closed by haypo #29527: Travis: doc job is broken http://bugs.python.org/issue29527 closed by haypo #29528: Encrypt IRC channel details for Travis http://bugs.python.org/issue29528 closed by zach.ware #29531: Update Doc/README.txt to README.rst http://bugs.python.org/issue29531 closed by Roger Sachan #29534: _decimal difference with _pydecimal http://bugs.python.org/issue29534 closed by mark.dickinson #29536: test_hashlib failure on Ubuntu 16.04 http://bugs.python.org/issue29536 closed by christian.heimes #29538: bugs.python.org does not redirect to HTTPS http://bugs.python.org/issue29538 closed by berker.peksag #29542: python -m spacy.en.download undefined symbol: _ZSt24__throw_ou http://bugs.python.org/issue29542 closed by berker.peksag #29543: Allow login to bugs.python.org with email address http://bugs.python.org/issue29543 closed by berker.peksag #29544: Allow login to bugs.python.org with Google account http://bugs.python.org/issue29544 closed by berker.peksag #29552: Issue in Dictionary fromkeys http://bugs.python.org/issue29552 closed by prudvibt81 #29563: Update Devguide about building documentation. http://bugs.python.org/issue29563 closed by brett.cannon #29567: Allow argparse to be used in setup.py http://bugs.python.org/issue29567 closed by Chi Hsuan Yen #29576: Improve some deprecations in the importlib http://bugs.python.org/issue29576 closed by brett.cannon #29578: "python.exe t2.py" doesn't work the same on Python-3.6 as Pyth http://bugs.python.org/issue29578 closed by paul.moore #29583: Python 3.6 won't install on Windows 2012 R2 http://bugs.python.org/issue29583 closed by eryksun #29584: 2D list with Bugs when defined by using Multiplication http://bugs.python.org/issue29584 closed by ezio.melotti

1 0

Is this a bug or a feature?
by Patrick Wallinger Feb. 17, 2017

Feb. 17, 2017

I'm brand new to python and was trying to install the newest version (3.6.0) on my ubuntu 16.04.1 but couldn't get it to build error free. I tried searching the bugs but didn't see a match for what I am seeing. Thanks for any help, Patrick Here is the report: patrick@ubuntu-MacBook:~/Programs/Python-3.6.0$ ./python -m test -v test_venv == CPython 3.6.0 (default, Feb 16 2017, 13:20:29) [GCC 5.4.0 20160609] == Linux-4.4.0-62-generic-x86_64-with-debian-stretch-sid little-endian == hash algorithm: siphash24 64bit == cwd: /home/patrick/Programs/Python-3.6.0/build/test_python_20536 == encodings: locale=UTF-8, FS=utf-8 Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=0, verbose=0, bytes_warning=0, quiet=0, hash_randomization=1, isolated=0) Run tests sequentially 0:00:00 [1/1] test_venv test_defaults (test.test_venv.BasicTest) ... ok test_executable (test.test_venv.BasicTest) ... ok test_executable_symlinks (test.test_venv.BasicTest) ... ok test_isolation (test.test_venv.BasicTest) ... ok test_overwrite_existing (test.test_venv.BasicTest) ... ok test_prefixes (test.test_venv.BasicTest) ... ok test_prompt (test.test_venv.BasicTest) ... ok test_symlinking (test.test_venv.BasicTest) ... ok test_unoverwritable_fails (test.test_venv.BasicTest) ... ok test_upgrade (test.test_venv.BasicTest) ... ok test_devnull (test.test_venv.EnsurePipTest) ... ok test_explicit_no_pip (test.test_venv.EnsurePipTest) ... ok test_no_pip_by_default (test.test_venv.EnsurePipTest) ... ok test_with_pip (test.test_venv.EnsurePipTest) ... FAIL ====================================================================== FAIL: test_with_pip (test.test_venv.EnsurePipTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/patrick/Programs/Python-3.6.0/Lib/test/test_venv.py", line 372, in test_with_pip with_pip=True) subprocess.CalledProcessError: Command '['/tmp/tmp51me85og/bin/python', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/patrick/Programs/Python-3.6.0/Lib/test/test_venv.py", line 378, in test_with_pip self.fail(msg.format(exc, details)) AssertionError: Command '['/tmp/tmp51me85og/bin/python', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1. **Subprocess Output** Traceback (most recent call last): File "/home/patrick/Programs/Python-3.6.0/Lib/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/home/patrick/Programs/Python-3.6.0/Lib/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/patrick/Programs/Python-3.6.0/Lib/ensurepip/__main__.py", line 4, in <module> ensurepip._main() File "/home/patrick/Programs/Python-3.6.0/Lib/ensurepip/__init__.py", line 189, in _main default_pip=args.default_pip, File "/home/patrick/Programs/Python-3.6.0/Lib/ensurepip/__init__.py", line 102, in bootstrap _run_pip(args + [p[0] for p in _PROJECTS], additional_paths) File "/home/patrick/Programs/Python-3.6.0/Lib/ensurepip/__init__.py", line 27, in _run_pip import pip zipimport.ZipImportError: can't decompress data; zlib not available ---------------------------------------------------------------------- Ran 14 tests in 1.278s FAILED (failures=1) test test_venv failed test_venv failed 1 test failed: test_venv Total duration: 1 sec Tests result: FAILURE

4 4

cgi.FieldStorage with multipart/form-data tries to decode binary file as UTF-8 if "filename=" not specified
by Ben Hoyt Feb. 15, 2017

Feb. 15, 2017

I posted this on StackOverflow [1], but I'm posting it here as well, as I believe this is a bug (or at least quirk) in cgi.FieldStorage where you can't access a file upload properly if "filename=" is not present in the MIME part's Content-Disposition header. There are a couple of related bugs open (and closed) on bugs.python.ord, but not quite this issue. Is it legitimate for cgi.FieldStorage to use the presence of "filename=" to determine "this is a binary file" (in which case this is not a bug and my client is just buggy), or is this a bug? I lean towards the latter as the spec indicates that the filename is optional [2]. Copying from my StackOverflow question, including a test/repro case: When I use `cgi.FieldStorage` to parse a `multipart/form-data` request (or any web framework like Pyramid which uses `cgi.FieldStorage`) I have trouble processing file uploads from certain clients which don't provide a `filename=file.ext` in the part's `Content-Disposition` header. If the `filename=` option is missing, `FieldStorage()` tries to decode the contents of the file as UTF-8 and return a string. And obviously many files are binary and not UTF-8 and as such give bogus results. For example: >>> import cgi >>> import io >>> body = (b'--KQNTvuH-itP09uVKjjZiegh7\r\n' + ... b'Content-Disposition: form-data; name=payload\r\n\r\n' + ... b'\xff\xd8\xff\xe0\x00\x10JFIF') >>> env = { ... 'REQUEST_METHOD': 'POST', ... 'CONTENT_TYPE': 'multipart/form-data; boundary=KQNTvuH-itP09uVKjjZiegh7', ... 'CONTENT_LENGTH': len(body), ... } >>> fs = cgi.FieldStorage(fp=io.BytesIO(body), environ=env) >>> (fs['payload'].filename, fs['payload'].file.read()) (None, '��\x00\x10JFIF') Browsers, and *most* HTTP libraries do include the `filename=` option for file uploads, but I'm currently dealing with a client that doesn't (and omitting the `filename` does seem to be valid according to the spec). Currently I'm using a pretty hacky workaround by subclassing `FieldStorage` and replacing the relevant `Content-Disposition` header with one that does have the filename: import cgi import os class FileFieldStorage(cgi.FieldStorage): """To use, subclass FileFieldStorage and override _file_fields with a tuple of the names of the file field(s). You can also override _file_name with the filename to add. """ _file_fields = () _file_name = 'file_name' def __init__(self, fp=None, headers=None, outerboundary=b'', environ=os.environ, keep_blank_values=0, strict_parsing=0, limit=None, encoding='utf-8', errors='replace'): if self._file_fields and headers and headers.get('content-disposition'): content_disposition = headers['content-disposition'] key, pdict = cgi.parse_header(content_disposition) if (key == 'form-data' and pdict.get('name') in self._file_fields and 'filename' not in pdict): del headers['content-disposition'] quoted_file_name = self._file_name.replace('"', '\\"') headers['content-disposition'] = '{}; filename="{}"'.format( content_disposition, quoted_file_name) super().__init__(fp=fp, headers=headers, outerboundary=outerboundary, environ=environ, keep_blank_values=keep_blank_values, strict_parsing=strict_parsing, limit=limit, encoding=encoding, errors=errors) Using the `body` and `env` in my first test, this works now: >>> class TestFieldStorage(FileFieldStorage): ... _file_fields = ('payload',) >>> fs = TestFieldStorage(fp=io.BytesIO(body), environ=env) >>> (fs['payload'].filename, fs['payload'].file.read()) ('file_name', b'\xff\xd8\xff\xe0\x00\x10JFIF') Is there some way to avoid this hack and tell `FieldStorage` not to decode as UTF-8? It would be nice if you could provide `encoding=None` or something, but it doesn't look like it supports that. Thanks, Ben. [1] https://stackoverflow.com/questions/42213318/cgi-fieldstorage-with-multipar… [2] https://www.w3.org/Protocols/rfc2616/rfc2616-sec19.html#sec19.5.1

2 2

Re: [Python-Dev] Python-Dev Digest
by 陈氏 Feb. 15, 2017

Feb. 15, 2017

help ________________________________ From: Python-Dev <python-dev-bounces+www.2849007069=outlook.com(a)python.org> on behalf of python-dev-request(a)python.org <python-dev-request(a)python.org> Sent: Wednesday, February 15, 2017 1:00:05 AM To: python-dev(a)python.org Subject: Python-Dev Digest, Vol 163, Issue 20 Send Python-Dev mailing list submissions to python-dev(a)python.org To subscribe or unsubscribe via the World Wide Web, visit https://mail.python.org/mailman/listinfo/python-dev or, via email, send a message with subject or body 'help' to python-dev-request(a)python.org You can reach the person managing the list at python-dev-owner(a)python.org When replying, please edit your Subject line so it is more specific than "Re: Contents of Python-Dev digest..." Today's Topics: 1. Re: [Python-checkins] [python/cpython] 4538dd: Fix bpo-29528 Use a secure variable to stop spam (... (Brett Cannon) 2. Re: [Python-checkins] [python/cpython] 4538dd: Fix bpo-29528 Use a secure variable to stop spam (... (Berker Peksa?) 3. Heads-up, I've rejoined (Eric S. Raymond) 4. Update on labels to be used on GitHub (Brett Cannon) 5. Re: Update on labels to be used on GitHub (Victor Stinner) ---------------------------------------------------------------------- Message: 1 Date: Mon, 13 Feb 2017 17:54:51 +0000 From: Brett Cannon <brett(a)snarky.ca> To: python-dev(a)python.org, python-checkins(a)python.org Cc: amk(a)amk.ca Subject: Re: [Python-Dev] [Python-checkins] [python/cpython] 4538dd: Fix bpo-29528 Use a secure variable to stop spam (... Message-ID: <CAP1=2W5ijfhoYPk7JdVKdyDBfEzQWK09rMw-FRPd+pm0yr0LiQ(a)mail.gmail.com> Content-Type: text/plain; charset="utf-8" On Mon, 13 Feb 2017 at 07:10 Barry Warsaw <barry(a)python.org> wrote: > On Feb 11, 2017, at 08:11 PM, Andrew M. Kuchling wrote: > > >Are we planning to include the diffs in python-checkins e-mail, or are > >they gone for good? > > > >(While they made the list's messages and digests larger, I liked > >having the diffs because you could page through them to see what > >changes were made without having to go through to another site.) > > Yep, this was recently discussed and I think the plan is to bring them back > for python-checkins, but that it's not possible to enable them on PRs due > to > GH limitations. > Serhiy asked about it and I said that the email integration we are using from GitHub doesn't allow for customizing the email it sends out beyond how is listed in the From line. If people find a different integration that they want to use for sending email then that can be set up. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20170213/cc0a0a6f/a…> ------------------------------ Message: 2 Date: Mon, 13 Feb 2017 21:25:34 +0300 From: Berker Peksa? <berker.peksag(a)gmail.com> To: Python Dev <python-dev(a)python.org>, python-checkins(a)python.org Subject: Re: [Python-Dev] [Python-checkins] [python/cpython] 4538dd: Fix bpo-29528 Use a secure variable to stop spam (... Message-ID: <CAF4280K=N__3jNAjeBDNO5F8pYkkcg6DeOFSkGy0iKZg01JOsg(a)mail.gmail.com> Content-Type: text/plain; charset=UTF-8 --Berker On Mon, Feb 13, 2017 at 8:54 PM, Brett Cannon <brett(a)snarky.ca> wrote: > > > On Mon, 13 Feb 2017 at 07:10 Barry Warsaw <barry(a)python.org> wrote: >> >> On Feb 11, 2017, at 08:11 PM, Andrew M. Kuchling wrote: >> >> >Are we planning to include the diffs in python-checkins e-mail, or are >> >they gone for good? >> > >> >(While they made the list's messages and digests larger, I liked >> >having the diffs because you could page through them to see what >> >changes were made without having to go through to another site.) >> >> Yep, this was recently discussed and I think the plan is to bring them >> back >> for python-checkins, but that it's not possible to enable them on PRs due >> to >> GH limitations. > > > Serhiy asked about it and I said that the email integration we are using > from GitHub doesn't allow for customizing the email it sends out beyond how > is listed in the From line. If people find a different integration that they > want to use for sending email then that can be set up. I wrote a webhook that includes diffs: An example output can be seen at https://github.com/python/core-workflow/issues/24#issuecomment-279162079 Since it listens all 'push' events, it's possible to catch all commits and send them to python-checkins. --Berker ------------------------------ Message: 3 Date: Mon, 13 Feb 2017 14:05:55 -0500 (EST) From: esr(a)thyrsus.com (Eric S. Raymond) To: python-dev(a)python.org Subject: [Python-Dev] Heads-up, I've rejoined Message-ID: <20170213190555.7073113A019A(a)snark.thyrsus.com> Some of the older Pythonistas will remember my previous time on this list, and possibly that I faded away quietly under time pressure from other projects and the whole being-famous nonsense. I'm back, for now primarily to listen. I have some questions about the future direction of Python which I'll bring up when I have time for the vigorous discussion that will certainly ensue. -- >>esr>> ------------------------------ Message: 4 Date: Mon, 13 Feb 2017 20:08:55 +0000 From: Brett Cannon <brett(a)python.org> To: python-dev <python-dev(a)python.org> Subject: [Python-Dev] Update on labels to be used on GitHub Message-ID: <CAP1=2W7WO5LXrE+V9tkxQPFo99fryV6+2U8SK=5by529kaLa6Q(a)mail.gmail.com> Content-Type: text/plain; charset="utf-8" We now have two sets of labels for representing cherry-picking statuses: "backport to N.M" and "cherry-pick for N.M". The former are to help keep track of what branches a PR should be cherry-picked *to* and can be removed once the PR for a cherry-pick has been created (you can add a comment like "cherry-picked from GH-NN" to make the link between the PRs)*.* The latter labels are to signify that the PR is a cherry-pick *itself*. As with all aspects of the new GitHub workflow, we will give this a solid try for about a month and then we can re-evaluate whether we like this approach or want a more traditional prefix format in the title to represent the same thing. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.python.org/pipermail/python-dev/attachments/20170213/225558f5/a…> ------------------------------ Message: 5 Date: Mon, 13 Feb 2017 23:35:58 +0100 From: Victor Stinner <victor.stinner(a)gmail.com> To: Brett Cannon <brett(a)python.org> Cc: python-dev <python-dev(a)python.org> Subject: Re: [Python-Dev] Update on labels to be used on GitHub Message-ID: <CAMpsgwY6d8MfauH7-SgwmGFQXEfuS5kVa0aE4BuSxRLsdcQEBQ(a)mail.gmail.com> Content-Type: text/plain; charset=UTF-8 2017-02-13 21:08 GMT+01:00 Brett Cannon <brett(a)python.org>: > We now have two sets of labels for representing cherry-picking statuses: > "backport to N.M" and "cherry-pick for N.M". The former are to help keep > track of what branches a PR should be cherry-picked to and can be removed > once the PR for a cherry-pick has been created (you can add a comment like > "cherry-picked from GH-NN" to make the link between the PRs). The first time I looked at the "backport to 3.6" flag, I understood that the change was written on the 3.6 branch and asked to rebase the change on 3.6. If I was confused, I'm sure that someone else will be confused as well :-) What do you think of renaming the label to "need 3.6 backport" (or "need backport to 3.6")? Victor ------------------------------ Subject: Digest Footer _______________________________________________ Python-Dev mailing list Python-Dev(a)python.org https://mail.python.org/mailman/listinfo/python-dev ------------------------------ End of Python-Dev Digest, Vol 163, Issue 20 *******************************************

1 0

Update on labels to be used on GitHub
by Brett Cannon Feb. 14, 2017

Feb. 14, 2017

We now have two sets of labels for representing cherry-picking statuses: "backport to N.M" and "cherry-pick for N.M". The former are to help keep track of what branches a PR should be cherry-picked *to* and can be removed once the PR for a cherry-pick has been created (you can add a comment like "cherry-picked from GH-NN" to make the link between the PRs)*.* The latter labels are to signify that the PR is a cherry-pick *itself*. As with all aspects of the new GitHub workflow, we will give this a solid try for about a month and then we can re-evaluate whether we like this approach or want a more traditional prefix format in the title to represent the same thing.

4 6

Heads-up, I've rejoined
by esr＠thyrsus.com Feb. 13, 2017

Feb. 13, 2017

Some of the older Pythonistas will remember my previous time on this list, and possibly that I faded away quietly under time pressure from other projects and the whole being-famous nonsense. I'm back, for now primarily to listen. I have some questions about the future direction of Python which I'll bring up when I have time for the vigorous discussion that will certainly ensue. -- >>esr>>

1 0

Re: [Python-Dev] [Python-checkins] [python/cpython] 4538dd: Fix bpo-29528 Use a secure variable to stop spam (...
by Barry Warsaw Feb. 13, 2017

Feb. 13, 2017

On Feb 11, 2017, at 08:11 PM, Andrew M. Kuchling wrote: >Are we planning to include the diffs in python-checkins e-mail, or are >they gone for good? > >(While they made the list's messages and digests larger, I liked >having the diffs because you could page through them to see what >changes were made without having to go through to another site.) Yep, this was recently discussed and I think the plan is to bring them back for python-checkins, but that it's not possible to enable them on PRs due to GH limitations. Cheers, -Barry

3 2

bugs.python.org access problems
by Terry Reedy Feb. 12, 2017

Feb. 12, 2017

Today I have been getting the following for about 1/3 of access attempts (with Firefox). ''' Secure Connection Failed An error occurred during a connection to bugs.python.org. A PKCS #11 module returned CKR_DEVICE_ERROR, indicating that a problem has occurred with the token or slot. Error code: SEC_ERROR_PKCS11_DEVICE_ERROR The page you are trying to view cannot be shown because the authenticity of the received data could not be verified. Please contact the website owners to inform them of this problem. [Try Again] ''' A couple of retries usually work, but I don't remember such repeated errors before. Anyone else having a similar problem? -- Terry Jan Reedy

4 5

CPython is now on GitHub
by Brett Cannon Feb. 10, 2017

Feb. 10, 2017

[this is a duplicate of what I just sent python-committers; rendered version at https://paper.dropbox.com/doc/CPython-workflow-changes-mx1k8G6M0rg5JLy80F1r6 ] # CPython workflow changes After more than two years, our new GitHub workflow is ready to accept changes (you can look back to my first “[vision statement]( https://mail.python.org/pipermail/python-dev/2014-December/137472.html)” on changing our workflow to see how things have changed since I started working on this)! I hope you are all excited to see this finished; I know my wife is very excited as she’s tired of listening to me talk about it for a third of our marriage. ;) # Thanks First and foremost, I want to thank everyone who helped with this. Thanks to Donald and Barry for writing the initial PEPs proposing [GitHub]( https://www.python.org/dev/peps/pep-0481/) and [GitLab]( https://www.python.org/dev/peps/pep-0507/) and Nick [pre-proposing RhodeCode](https://www.python.org/dev/peps/pep-0474/). Thanks to everyone on [core-workflow](https://mail.python.org/mailman/listinfo/core-workflow) for helping out with various discussion (and which will continue to host discussions on future improvements on our workflow). Thanks to Ezio, Maciej, and Anish Shah for helping with the changes required to bugs.python.org in order to keep the issue tracker around. Thanks to the infrastructure team for helping deal with the migration of the [peps]( https://github.com/python/peps) and [devguide]( https://github.com/python/devguide) repos (especially Donald and Ernest). Thanks to Senthil for doing the conversion of the repo itself. Thanks to Benjamin for helping with hg.python.org stuff. Thanks to Zach for helping with the buildbots (and the devguide). Thanks to Mariatta, Carol Willing, Berker, Oleg, and Stéphane Wirtel for helping with the devguide. There are also plenty of other people who have provided feedback over the past 2 years on mailing lists and in-person. # What has changed The documentation in the [devguide](https://cpython-devguide.readthedocs.io) should be up-to-date, so don’t worry about keeping this around as a reference. Consider this more of an announcement letter to get people quickly up-to-speed and excited about the new workflow. ## The location of the code repository CPython’s code now lives at https://github.com/python/cpython . hg.python.org/cpython is and will stay read-only (no determination has been made as to how long the Mercurial repository will be kept running). It should also be mentioned that we are doing squash commits and not rebased commits or merge commits as some projects on GitHub do. This basically means we will continue to have a single contribution lead to a single commit, keeping our history linear and compact. To up the bus factor on the new repository, I have set up a team for [Python release managers]( https://github.com/orgs/python/teams/python-release-managers) and made them administrators on the repository. I don’t necessarily expect RMs to use this power, but it’s there in case any of them need to change a setting in order to get a release out (to be upfront about it, I’m also in the team as its creator, but I have administrative privileges for the [Python team]( https://github.com/python) on GitHub so it doesn’t change what I’m able to do). ## Specifying issue numbers Traditionally we have specified issues as “Issue #NNNN”. The problem with this format is that [GitHub automatically links]( https://help.github.com/articles/autolinked-references-and-urls/#issues-and…) text in this format to GitHub issues and pull requests. While our repository will initially have no issues or PRs to automatically link to, this might not be true long-term (GitHub does the automatic linking eagerly at push time, so there’s no worry for older commit messages; we actually almost mutated the history to match the new format but in the end I made the decision not to as I didn’t consult with python-committers prior to the migration to make sure such a change was acceptable to everyone). To avoid this issue we are going to start specifying issues as “bpo-NNNN”. This clearly delineates that issue numbers directly relate to bugs.python.org since “[namespaces are one honking great idea]( https://www.python.org/dev/peps/pep-0020/)”. This is also a format that GitHub supports — “GH-NNNN” — as well as JIRA who [lets you specify the prefix]( https://confluence.atlassian.com/adminjiraserver072/changing-the-project-ke…), so there’s precedent for choosing it. This change applies both to `[Misc/NEWS]( https://cpython-devguide.readthedocs.io/committing.html#what-s-new-and-news… and in [commit messages]( https://cpython-devguide.readthedocs.io/committing.html#commit-messages). Mentioning an issue this way in a pull request title or comment will connect the PR to the corresponding issue on bugs.python.org. Mentioning an issue this way in a commit message will cause a comment to show up in the issue relating to the commit. ## Cherry-picking instead of merging When a patch has spanned more than one version/branch we have always done a forward merge. The common issue with this, though, is it leads to racing with other committers from when you make to your initial commit in the oldest version/branch to pushing to hg.python.org on the newest version/branch. There was also the problem of having to remember that Python 2.7 is a special branch which was never merged forward. To deal with these issues we will use [cherry-picking]( https://cpython-devguide.readthedocs.io/committing.html#backporting-changes…) going forward. This allows changes to be pulled into other branches as independent commits. This prevents any commit races with other core developers as we have traditionally needed to deal with when doing forward merges that span e.g. three branches. It also allows using CI to easily verify a change works if each cherry-pick is done as a separate pull request. The Python 2.7 branch also stops being a special case when backporting. It also prevents potential issues stemming from contributors submitting pull requests against the master branch by default and not the oldest branch a change should be applied to. Finally, this also removes the discussion of whether a change should be backported or not from blocking the commit into master to begin with. Labels will be provided for people to use to help track any cherry-picking that needs to occur for a pull request (e.g. “backport to 3.6”). I also left the “bug” and “enhancement” labels to help classify PRs (adding more labels is easy so we can do that as our experience and workflow organically converge towards common practices). ## Protected branches All feature branches have been marked as [protected]( https://help.github.com/articles/about-protected-branches/). This means that feature branches cannot be deleted nor can they be pushed to directly. The latter will be the biggest change as it means all changes must go through a pull request. This helps make sure that there are no accidental breakage of code (I know I have done this multiple times when in a rush and I didn’t take my time when preparing a commit). This also means that all core developers follow the same development workflow as any other contributor. This not only allows all core developers to be able to help any other contributors with our workflow, but it also helps make sure we are aware of any sticking points in the contributor process so that we can all work towards resolving them for everyone’s benefit. (If experience shows that this is too much overhead we can turn this off.) All feature branches that are in security-only mode are locked down so that only [Python release managers]( https://github.com/orgs/python/teams/python-release-managers) can approve pull requests to them. For all branches that have reached EOL, no one is able to push to them. I expect that RMs will also use this feature when they are ready to gate all commits to a branch on their approval (e.g. when a release reaches RC, maybe even beta if they choose to go that far). # What has improved ## Accepting PRs through GitHub’s web UI While using hg.python.org, all commits had to be done through Mercurial’s CLI. With the move to GitHub we gain the ability to accept pull requests through a web UI. While this will only accept the change into the branch it was submitted against (which can be changed in the web UI), for situations where a change does not need to be backported it will allow for easier acceptance of a change. (When a change does need to be backported this is when you need to cherry-pick and that requires using the git CLI). If a change does need to be cherry-picked into an older branch you can either wait to accept the PR when you have a clone to work with or accept the change into master now and then cherry-pick later when you have a clone available. ## Continuous integration Previously changes required running the test suite manually along with verifying various other things like the documentation building. Moving to GitHub allows us to leverage the Travis continuous integration service to test several things in parallel automatically for each pull request: 1. Debug build under gcc 2. Debug build under clang 3. Documentation is valid and has no stale links 4. Python.h C++ compatibility While this doesn’t solve all testing scenarios (e.g. this doesn’t test a macOS or Windows-related change due to the added hours it take for a PR to be “green” when run on Travis for macOS or AppVeyor for Windows), it does help with the common case of a cross-platform change. (There is an [open issue](https://github.com/python/core-workflow/issues/14) to add some code so that these tests only run when appropriate files have changed so that e.g. fixing a spelling mistake doesn’t run the test suite.) It should be mentioned that status checks on issues are **not** required prior to committing a pull request. While this may be a good idea long-term, until we know that our test suite is stable enough to not have regular flaky tests this would be more trouble than it’s worth (GitHub does visibly show, though, when not all status checks have passed so you won’t easily ignore this situation either). ## Code coverage Traditionally the code coverage of our tests was only known when someone ran the test suite manually under something like [coverage.py]( https://coverage.readthedocs.io). Even when someone did generate a coverage report it was generally not shared with other developers, and so it wasn’t widely known if a pull request increased or lowered test coverage. With the move to GitHub we are able to use [Codecov](https://codecov.io/) to calculate code coverage for each pull request. This also implicitly tests a non-debug build as that’s used to make the coverage results run faster. It should be noted, though, that some tests are skipped due to them holding up the coverage run from completing. (There is an [open issue]( https://github.com/python/core-workflow/issues/18) to use coverage.py’s fullcoverage hack so that the coverage report can even be accurate for modules imported during interpreter startup.) ## CLA enforcement To tell if someone has signed the PSF contributor license agreement you have to look to see if they have an asterisk by their name on the issue tracker. Unfortunately this is a passive thing to need to check for and is easily forgotten. Thanks to GitHub’s webhook events and developer API we now have a bot which checks if the contributor(s) to a pull request have signed the CLA, adding an appropriate label to the PR to signal the CLA signing status (the bot is named [The Knights Who Say Ni]( https://github.com/python/the-knights-who-say-ni)). If the contributor(s) have not signed the CLA then a message is left on the PR explaining how to rectify the issue (it’s either they need to connect their GitHub account to their bugs.python.org account or they need to sign the CLA; there’s also is an easter egg that occasionally appears in the message). If a contributor does end up fixing the issue that leads to the bot thinking the contributor had not signed the CLA, you can remove the “CLA not signed” label and the bot will recheck the PR and add the appropriate label (this also happens automatically if any code changes are made to the PR). If for some the reason the bot has a hiccup then no label will be applied (this is to act as a safeguard against false-negatives and to make it easy to spot when something has gone wrong due to the absence of either a “CLA signed” or “CLA not signed” label). To trigger the bot again you can simply apply the “CLA not signed” label and then remove it. ## Contribution guidelines There is now a `[.github/CONTRIBUTING.rst]( https://github.com/python/cpython/blob/master/.github/CONTRIBUTING.rst)` file which gives general info on contributing. Primarily it has all the various build status badges and links, link to the devguide and an overview of how we differ from most GitHub projects, and a mention of the CoC. For core devs I suspect the badge list for all active branches will be useful to know when something is broken (I am only listing the test coverage for the master branch to prevent encouraging people spending time trying to increase test coverage for bugfix-only branches) . # The future This isn’t here for discussion per-se, but to let people know what I am thinking should change next. If you want to help or discuss anything in this section, please subscribe to core-workflow and participate there. Ideas are also tracked on the [core-workflow issue tracker]( https://github.com/python/core-workflow/issues) (where there are other ideas for improvements beyond the two listed below). First, we need to solve the Misc/NEWS problem. The [plan]( https://github.com/python/core-workflow/issues/6) is to move to individual files per NEWS entry and then have a script to roll them all up into the NEWS file at release time. This will do away with merge conflicts which has always been the biggest hassle when a change spanned versions. Second, we will probably develop some solution to [automatically generate cherry-picking PRs](https://github.com/python/core-workflow/issues/8). This plus the Misc/NEWS solution should be enough to allow most patches to be accepted through the web UI entirely. All the other changes are nice-to-have, but I think these two will lead to the greatest improvement to our workflow and get us far beyond the workflow we had on hg.python.org.

1 0