Mailman 3 October 2017 - Python-Dev

Investigating time for `import requests`
by INADA Naoki Oct. 20, 2017

Oct. 20, 2017

See also https://github.com/requests/requests/issues/4315 I tried new `-X importtime` option to `import requests`. Full output is here: https://gist.github.com/methane/96d58a29e57e5be97769897462ee1c7e Currently, it took about 110ms. And major parts are from Python stdlib. Followings are root of slow stdlib subtrees. import time: self [us] | cumulative | imported package import time: 1374 | 14038 | logging import time: 2636 | 4255 | socket import time: 2902 | 11004 | ssl import time: 1162 | 16694 | http.client import time: 656 | 5331 | cgi import time: 7338 | 7867 | http.cookiejar import time: 2930 | 2930 | http.cookies *1. logging* logging is slow because it is imported in early stage. It imports many common, relatively slow packages. (collections, functools, enum, re). Especially, traceback module is slow because linecache. import time: 1419 | 5016 | tokenize import time: 200 | 5910 | linecache import time: 347 | 8869 | traceback I think it's worth enough to import linecache lazily. *2. socket* import time: 807 | 1221 | selectors import time: 2636 | 4255 | socket socket imports selectors for socket.send_file(). And selectors module use ABC. That's why selectors is bit slow. And socket module creates four enums. That's why import socket took more than 2.5ms excluding subimports. *3. ssl* import time: 2007 | 2007 | ipaddress import time: 2386 | 2386 | textwrap import time: 2723 | 2723 | _ssl ... import time: 306 | 988 | base64 import time: 2902 | 11004 | ssl I already created pull request about removing textwrap dependency from ssl. https://github.com/python/cpython/pull/3849 ipaddress and _ssl module are bit slow too. But I don't know we can improve them or not. ssl itself took 2.9 ms. It's because ssl has six enums. *4. http.client* import time: 1376 | 2448 | email.header ... import time: 1469 | 7791 | email.utils import time: 408 | 10646 | email._policybase import time: 939 | 12210 | email.feedparser import time: 322 | 12720 | email.parser ... import time: 599 | 1361 | email.message import time: 1162 | 16694 | http.client email.parser has very large import tree. But I don't know how to break the tree. *5. cgi* import time: 1083 | 1083 | html.entities import time: 560 | 1643 | html ... import time: 656 | 2609 | shutil import time: 424 | 3033 | tempfile import time: 656 | 5331 | cgi cgi module uses tempfile to save uploaded file. But requests imports cgi just for `cgi.parse_header()`. tempfile is not used. Maybe, it's worth enough to import it lazily. FYI, cgi depends on very slow email.parser too. But this tree doesn't contain it because http.client is imported before cgi. Even though it's not problem for requests, it may affects to real CGI application. Of course, startup time is very important for CGI applications too. *6. http.cookiejar and http.cookies* It's slow because it has many `re.compile()` *Ideas* There are some places to break large import tree by "import in function" hack. ABC is slow, and it's used widely without almost no real need. (Who need selectors is ABC?) We can't remove ABC dependency because of backward compatibility. But I hope ABC is implemented in C by Python 3.7. Enum is slow, maybe slower than most people think. I don't know why exactly, but I suspect that it's because namespace dict implemented in Python. Anyway, I think we can have C implementation of IntEnum and IntFlag, like namedtpule vs PyStructSequence. It doesn't need to 100% compatible with current enum. Especially, no need for using metaclass. Another major slowness comes from compiling regular expression. I think we can increase cache size of `re.compile` and use ondemand cached compiling (e.g. `re.match()`), instead of "compile at import time" in many modules. PEP 562 -- Module __getattr__ helps a lot too. It make possible to split collection module and strings module. (strings module is used often for constants like strings.ascii_letters, but strings.Template cause import time re.compile()) Regards, -- Inada Naoki <songofacandy(a)gmail.com>

17 26

Summary of Python tracker Issues
by Python tracker Oct. 20, 2017

Oct. 20, 2017

ACTIVITY SUMMARY (2017-10-13 - 2017-10-20) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 6260 ( +9) closed 37318 (+38) total 43578 (+47) Open issues with patches: 2415 Issues opened (36) ================== #31761: Failures and crashes when running tests by import in IDLE https://bugs.python.org/issue31761 reopened by terry.reedy #31772: SourceLoader uses stale bytecode in case of equal mtime second https://bugs.python.org/issue31772 reopened by brett.cannon #31781: crashes when calling methods of an uninitialized zipimport.zip https://bugs.python.org/issue31781 opened by Oren Milman #31782: Add a timeout to multiprocessing's Pool.join https://bugs.python.org/issue31782 opened by Will Starms #31783: Race condition in ThreadPoolExecutor when scheduling new jobs https://bugs.python.org/issue31783 opened by Steven.Barker #31784: Implementation of the PEP 564: Add time.time_ns() https://bugs.python.org/issue31784 opened by haypo #31787: various refleaks when calling the __init__() method of an obje https://bugs.python.org/issue31787 opened by Oren Milman #31789: Better error message when failing to overload metaclass.mro https://bugs.python.org/issue31789 opened by bup #31790: double free or corruption (while using smem) https://bugs.python.org/issue31790 opened by zulthan #31791: Ensure that all PyTypeObject fields are set to non-NULL defaul https://bugs.python.org/issue31791 opened by pdox #31793: Allow to specialize smart quotes in documentation translations https://bugs.python.org/issue31793 opened by mdk #31794: Issues with test.autotest https://bugs.python.org/issue31794 opened by serhiy.storchaka #31798: `site.abs__file__` fails for modules where `__file__` cannot b https://bugs.python.org/issue31798 opened by Alexander McFarlane #31800: datetime.strptime: Support for parsing offsets with a colon https://bugs.python.org/issue31800 opened by mariocj89 #31801: vars() manipulation encounters problems with Enum https://bugs.python.org/issue31801 opened by ethan.furman #31802: 'import posixpath' fails if 'os.path' has not be imported alre https://bugs.python.org/issue31802 opened by Mark.Shannon #31803: time.clock() should emit a DeprecationWarning https://bugs.python.org/issue31803 opened by haypo #31804: multiprocessing calls flush on sys.stdout at exit even if it i https://bugs.python.org/issue31804 opened by Pox TheGreat #31805: support._is_gui_available() hangs on x86-64 Sierra 3.6/3.x bui https://bugs.python.org/issue31805 opened by haypo #31807: unitest.mock: Using autospec=True conflicts with 'wraps' https://bugs.python.org/issue31807 opened by John Villalovos #31808: tarfile.extractall fails to overwrite symlinks https://bugs.python.org/issue31808 opened by Frederic Beister #31809: ssl module unnecessarily pins the client curve when using ECDH https://bugs.python.org/issue31809 opened by grrrrrrrrr #31810: Travis CI, buildbots: run "make smelly" to check if CPython le https://bugs.python.org/issue31810 opened by haypo #31811: async and await missing from keyword list in lexical analysis https://bugs.python.org/issue31811 opened by Colin Dunklau #31812: Document PEP 545 (documentation translation) in What's New in https://bugs.python.org/issue31812 opened by haypo #31813: python -m enshure pip stucks https://bugs.python.org/issue31813 opened by Serhy Pyton #31814: subprocess_fork_exec more stable with vfork https://bugs.python.org/issue31814 opened by Albert.Zeyer #31815: Make itertools iterators interruptible https://bugs.python.org/issue31815 opened by serhiy.storchaka #31817: Compilation Error with Python 3.6.1/3.6.3 with Tkinter https://bugs.python.org/issue31817 opened by jpc2350 #31818: [macOS] _scproxy.get_proxies() crash -- get_proxies() is not f https://bugs.python.org/issue31818 opened by Mirko Friedenhagen #31821: pause_reading() doesn't work from connection_made() https://bugs.python.org/issue31821 opened by pitrou #31822: Document that urllib.parse.{Defrag,Split,Parse}Result are name https://bugs.python.org/issue31822 opened by Allen Li #31823: Opaque default value for close_fds argument in Popen.__init__ https://bugs.python.org/issue31823 opened by Алексей Аверченко #31824: Missing default argument detail in documentation of StreamRead https://bugs.python.org/issue31824 opened by PeterLovett #31826: Misleading __version__ attribute of modules in standard librar https://bugs.python.org/issue31826 opened by abukaj #31827: Remove os.stat_float_times() https://bugs.python.org/issue31827 opened by haypo Most recent 15 issues with no replies (15) ========================================== #31826: Misleading __version__ attribute of modules in standard librar https://bugs.python.org/issue31826 #31824: Missing default argument detail in documentation of StreamRead https://bugs.python.org/issue31824 #31823: Opaque default value for close_fds argument in Popen.__init__ https://bugs.python.org/issue31823 #31821: pause_reading() doesn't work from connection_made() https://bugs.python.org/issue31821 #31817: Compilation Error with Python 3.6.1/3.6.3 with Tkinter https://bugs.python.org/issue31817 #31812: Document PEP 545 (documentation translation) in What's New in https://bugs.python.org/issue31812 #31808: tarfile.extractall fails to overwrite symlinks https://bugs.python.org/issue31808 #31807: unitest.mock: Using autospec=True conflicts with 'wraps' https://bugs.python.org/issue31807 #31804: multiprocessing calls flush on sys.stdout at exit even if it i https://bugs.python.org/issue31804 #31802: 'import posixpath' fails if 'os.path' has not be imported alre https://bugs.python.org/issue31802 #31801: vars() manipulation encounters problems with Enum https://bugs.python.org/issue31801 #31794: Issues with test.autotest https://bugs.python.org/issue31794 #31793: Allow to specialize smart quotes in documentation translations https://bugs.python.org/issue31793 #31790: double free or corruption (while using smem) https://bugs.python.org/issue31790 #31789: Better error message when failing to overload metaclass.mro https://bugs.python.org/issue31789 Most recent 15 issues waiting for review (15) ============================================= #31827: Remove os.stat_float_times() https://bugs.python.org/issue31827 #31821: pause_reading() doesn't work from connection_made() https://bugs.python.org/issue31821 #31815: Make itertools iterators interruptible https://bugs.python.org/issue31815 #31810: Travis CI, buildbots: run "make smelly" to check if CPython le https://bugs.python.org/issue31810 #31809: ssl module unnecessarily pins the client curve when using ECDH https://bugs.python.org/issue31809 #31804: multiprocessing calls flush on sys.stdout at exit even if it i https://bugs.python.org/issue31804 #31803: time.clock() should emit a DeprecationWarning https://bugs.python.org/issue31803 #31802: 'import posixpath' fails if 'os.path' has not be imported alre https://bugs.python.org/issue31802 #31800: datetime.strptime: Support for parsing offsets with a colon https://bugs.python.org/issue31800 #31793: Allow to specialize smart quotes in documentation translations https://bugs.python.org/issue31793 #31787: various refleaks when calling the __init__() method of an obje https://bugs.python.org/issue31787 #31784: Implementation of the PEP 564: Add time.time_ns() https://bugs.python.org/issue31784 #31782: Add a timeout to multiprocessing's Pool.join https://bugs.python.org/issue31782 #31781: crashes when calling methods of an uninitialized zipimport.zip https://bugs.python.org/issue31781 #31779: assertion failures and a crash when using an uninitialized str https://bugs.python.org/issue31779 Top 10 most discussed issues (10) ================================= #31803: time.clock() should emit a DeprecationWarning https://bugs.python.org/issue31803 17 msgs #30744: Local variable assignment is broken when combined with threads https://bugs.python.org/issue30744 11 msgs #31815: Make itertools iterators interruptible https://bugs.python.org/issue31815 10 msgs #31778: ast.literal_eval supports non-literals in Python 3 https://bugs.python.org/issue31778 7 msgs #31818: [macOS] _scproxy.get_proxies() crash -- get_proxies() is not f https://bugs.python.org/issue31818 7 msgs #31800: datetime.strptime: Support for parsing offsets with a colon https://bugs.python.org/issue31800 6 msgs #29696: Use namedtuple in string.Formatter.parse iterator response https://bugs.python.org/issue29696 4 msgs #31742: Default to emitting FutureWarning for provisional APIs https://bugs.python.org/issue31742 4 msgs #31753: Unnecessary closure in ast.literal_eval https://bugs.python.org/issue31753 4 msgs #31791: Ensure that all PyTypeObject fields are set to non-NULL defaul https://bugs.python.org/issue31791 4 msgs Issues closed (40) ================== #13802: IDLE font settings: use multiple character sets in examples https://bugs.python.org/issue13802 closed by terry.reedy #25588: Run test suite from IDLE idlelib.run subprocess https://bugs.python.org/issue25588 closed by terry.reedy #26098: [WIP] PEP 510: Specialize functions with guards https://bugs.python.org/issue26098 closed by haypo #26145: [WIP] PEP 511: Add sys.set_code_transformers() https://bugs.python.org/issue26145 closed by haypo #28603: traceback module can't format/print unhashable exceptions https://bugs.python.org/issue28603 closed by serhiy.storchaka #30013: Compiler warning in Modules/posixmodule.c https://bugs.python.org/issue30013 closed by serhiy.storchaka #30457: Allow retrieve the number of waiters pending for most of the a https://bugs.python.org/issue30457 closed by yselivanov #30541: Add restricted mocks to the python unittest mocking framework https://bugs.python.org/issue30541 closed by haypo #31234: Make support.threading_cleanup() stricter https://bugs.python.org/issue31234 closed by haypo #31334: select.poll.poll fails on BSDs with arbitrary negative timeout https://bugs.python.org/issue31334 closed by serhiy.storchaka #31452: asyncio.gather does not cancel tasks if one fails https://bugs.python.org/issue31452 closed by yselivanov #31558: gc.freeze() - an API to mark objects as uncollectable https://bugs.python.org/issue31558 closed by lukasz.langa #31618: Change sys.settrace opcode tracing to occur after frame line n https://bugs.python.org/issue31618 closed by ncoghlan #31622: Make threading.get_ident() return an opaque type https://bugs.python.org/issue31622 closed by pdox #31632: Asyncio: SSL transport does not support set_protocol() https://bugs.python.org/issue31632 closed by yselivanov #31676: test.test_imp.ImportTests.test_load_source has side effects https://bugs.python.org/issue31676 closed by haypo #31692: [2.7] Test `test_huntrleaks()` of test_regrtest fails in debug https://bugs.python.org/issue31692 closed by haypo #31714: Improve re documentation https://bugs.python.org/issue31714 closed by serhiy.storchaka #31733: [2.7] Add PYTHONSHOWREFCOUNT environment variable to Python 2. https://bugs.python.org/issue31733 closed by haypo #31738: Lib/site.py: method `abs_paths` is not documented https://bugs.python.org/issue31738 closed by merwok #31754: Documented type of parameter 'itemsize' to PyBuffer_FillContig https://bugs.python.org/issue31754 closed by berker.peksag #31757: Tutorial: Fibonacci numbers start with 1, 1 https://bugs.python.org/issue31757 closed by rhettinger #31760: Re-definition of _POSIX_C_SOURCE with Fedora 26. https://bugs.python.org/issue31760 closed by martin.panter #31763: Add NOTICE level to the logging module https://bugs.python.org/issue31763 closed by rhettinger #31765: BUG: System deadlocks performing big loop operations in python https://bugs.python.org/issue31765 closed by terry.reedy #31776: Missing "raise from None" in /Lib/xml/etree/ElementPath.py https://bugs.python.org/issue31776 closed by serhiy.storchaka #31780: Using format spec ',x' displays incorrect error message https://bugs.python.org/issue31780 closed by terry.reedy #31785: Move instruction code from ceval.c to a separate file https://bugs.python.org/issue31785 closed by rhettinger #31786: In select.poll.poll() ms can be 0 if timeout < 0 https://bugs.python.org/issue31786 closed by serhiy.storchaka #31788: Typo in comments Modules/_ssl.c https://bugs.python.org/issue31788 closed by benjamin.peterson #31792: test_buffer altered the execution environment https://bugs.python.org/issue31792 closed by serhiy.storchaka #31795: Slicings documentation doesn't mention Ellipsis https://bugs.python.org/issue31795 closed by rhettinger #31796: The mccabe complexity output in module flake8. https://bugs.python.org/issue31796 closed by ned.deily #31797: Python 3.6.3: JSON loop fails using elif https://bugs.python.org/issue31797 closed by eric.smith #31799: Improve __spec__ discoverability https://bugs.python.org/issue31799 closed by barry #31806: Use _PyTime_ROUND_TIMEOUT in _threadmodule.c, timemodule.c and https://bugs.python.org/issue31806 closed by serhiy.storchaka #31816: Unexpected behaviour of `dir()` after implementation of __dir_ https://bugs.python.org/issue31816 closed by christian.heimes #31819: Add sock_recv_into to AbstractEventLoop https://bugs.python.org/issue31819 closed by yselivanov #31820: Calling email.message.set_payload twice produces an invalid em https://bugs.python.org/issue31820 closed by Zirak Ertan #31825: bytes decode raises OverflowError desipte errors='ignore' https://bugs.python.org/issue31825 closed by serhiy.storchaka

1 0

What is the design purpose of metaclasses vs code generating decorators? (was Re: PEP 557: Data Classes)
by Nick Coghlan Oct. 20, 2017

Oct. 20, 2017

On 13 October 2017 at 04:21, Martin Teichmann <lkb.teichmann(a)gmail.com> wrote: > For me, the dataclasses were a typical example for inheritance, to be > more precise, for metaclasses. I was astonished to see them > implemented using decorators, and I was not the only one, citing > Guido: > > > I think it would be useful to write 1-2 sentences about the problem with > > inheritance -- in that case you pretty much have to use a metaclass, and > the > > use of a metaclass makes life harder for people who want to use their own > > metaclass (since metaclasses don't combine without some manual > > intervention). > > Python is at a weird point here. At about every new release of Python, > a new idea shows up that could be easily solved using metaclasses, yet > every time we hesitate to use them, because of said necessary manual > intervention for metaclass combination. > Metaclasses currently tend to serve two distinct purposes: 1. Actually altering the runtime behaviour of a class and its children in non-standard ways (e.g. enums, ABCs, ORMs) 2. Boilerplate reduction in class definitions, reducing the amount of code you need to write as the author of that class Nobody has a problem with using metaclasses for the first purpose - that's what they're for. It's the second use case where they're problematic, as the fact that they're preserved on the class becomes a leaky implementation detail, and the lack of a JIT in CPython means they can also end up being expensive from a runtime performance perspective. Mixin classes have the same problem: something that the author may want to handle as an internal implementation detail leaks through to the runtime state of the class object. Code generating decorators like functools.total_ordering and dataclasses.dataclass (aka attr.s) instead aim at the boilerplate reduction problem directly: they let you declare in the class body the parts that you need to specify as the class designer, and then fill in at class definition time the parts that can be inferred from that base. If all you have access to is the runtime class, it behaves almost exactly as if you had written out all the autogenerated methods by hand (there may be subtle differences in the method metadata, such as the values of `__qualname__` and `__globals__`). Such decorators also do more work at class definition time in order to reduce the amount of runtime overhead introduced by reliance on chained method calls in a non-JITted Python runtime. As such, the code generating decorators have a clear domain of applicability: boilerplate reduction for class definitions without impacting the way instances behave (other than attribute and method injection), and without implicitly impacting subclass definitions (other than through regular inheritance behaviour). As far as the dataclass interaction with `__slots__` goes, that's a problem largely specific to slots (and `__metaclass__` before it), in that they're the only characteristics of a class definition that affect how CPython allocates memory for the class object itself (the descriptors for the slots are stored as a pointer array after the class struct, rather than only in the class dict). Given PEP 526 variable annotations, __slots__ could potentially benefit from a __metaclass__ style makeover, allowing an "infer_slots=True" keyword argument to type.__new__ to request that the list of slots be inferred from __annotations__ (Slot inference would conflict with setting class level default values, but that's a real conflict, as you'd be trying to use the same name on the class object for both the slot descriptor and the default value) Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

8 14

Timeout for PEP 550 / Execution Context discussion
by Yury Selivanov Oct. 18, 2017

Oct. 18, 2017

Hi, It looks like the discussion about the execution context became extremely hard to follow. There are many opinions on how the spec for generators should look like. What seems to be "natural" behaviour/example to one, seems to be completely unreasonable to other people. Recent emails from Guido indicate that he doesn't want to implement execution contexts for generators (at least in 3.7). In another thread Guido said this: "... Because coroutines and generators are similar under the covers, Yury demonstrated the issue with generators instead of coroutines (which are unfamiliar to many people). And then somehow we got hung up about fixing the problem in the example." And Guido is right. My initial motivation to write PEP 550 was to solve my own pain point, have a solution for async code. 'threading.local' is completely unusable there, but complex code bases demand a working solution. I thought that because coroutines and generators are so similar under the hood, I can design a simple solution that will cover all edge cases. Turns out it is not possible to do it in one pass. Therefore, in order to make some progress, I propose to split the problem in half: Stage 1. A new execution context PEP to solve the problem *just for async code*. The PEP will target Python 3.7 and completely ignore synchronous generators and asynchronous generators. It will be based on PEP 550 v1 (no chained lookups, immutable mapping or CoW as an optimization) and borrow some good API decisions from PEP 550 v3+ (contextvars module, ContextVar class). The API (and C-API) will be designed to be future proof and ultimately allow transition to the stage 2. Stage 2. When Python 3.7 is out, we'll see how people use execution contexts for async code and collect feedback. If we recognize that Python users want execution contexts for generators/asynchronous generators, we'll make a new PEP to add support for them in Python 3.8. That future discussion will be focused on generators specifically, and therefore I expect it to be somewhat more focused. I will start working on the new PEP for stage 1 tomorrow. I expect to have a first version by the end of the week. I will also publish PEP 550 v1 as a separate PEP (as v1 is a totally different PEP anyways). Thanks, Yury

6 40

Tracking fixes of security vulnerabilies: we are good!
by Victor Stinner Oct. 18, 2017

Oct. 18, 2017

Hi, Since the beginning of the year, I'm working on a tool to track if all security vulnerabilities are fixed in all Python maintained versions (versions still accepting security fixes): http://python-security.readthedocs.io/vulnerabilities.html Currently, five branches are maintained: 2.7, 3.4, 3.5, 3.6 and master. https://devguide.python.org/#status-of-python-branches Thanks to Ned Deily and Georg Brandl, Python 3.3 reached its end-of-life (EOL) last month, after 5 years of good service (as expected). It reduced the number of maintained branches from six to five :-) Python 3.3.7 released last months contains the last security fixes. The good news is that we got releases last months with fixes for almost all security vulnerabilities. Only Python 3.4 and Python 3.5 have two known vulnerabilities, but I consider that their severity is not high hopefully. "Expat 2.2.3" is not fixed yet in Python 3.4 and 3.5, but I'm not sure that Python is really affected by fixed Expat vulnerabilities, since Python uses its own code to generate a secret key for the Expat "hash secret". Our embedded expat copy is used on Windows and macOS, but not on Linux. "update zlib to 1.2.11" was fixed in the Python 3.4 branch, but no release was made yet. This issue only impacts Windows. Linux and macOS use the system zlib. Victor

1 0

PEP 510 (function specialization) rejected
by Victor Stinner Oct. 18, 2017

Oct. 18, 2017

Hi, I rejected my own PEP 510 "Specialize functions with guards" that I wrote in January 2016: https://github.com/python/peps/commit/c99fb8bf5b5c16c170e1603a1c66a74e93a4a… "This PEP was rejected by its author since the design didn't show any significant speedup, but also because of the lack of time to implement the most advanced and complex optimizations." I stopped working on my FAT Python project almost one year ago: https://faster-cpython.readthedocs.io/fat_python.html Victor

3 3

Python startup optimization: script vs. service
by Christian Heimes Oct. 17, 2017

Oct. 17, 2017

Hello python-dev, it's great to see that so many developers are working on speeding up Python's startup. The improvements are going to make Python more suitable for command line scripts. However I'm worried that some approaches are going to make other use cases slower and less efficient. I'm talking about downsides of lazy initialization and deferred imports. For short running command line scripts, lazy initialization of regular expressions and deferred import of rarely used modules can greatly reduce startup time and reduce memory usage. For long running processes, deferring imports and initialization can be a huge performance problem. A typical server application should initialize as much as possible at startup and then signal its partners that it is ready to serve requests. A deferred import of a module is going to slow down the first request that happens to require the module. This is unacceptable for some applications, e.g. Raymond's example of speed trading. It's even worse for forking servers. A forking HTTP server handles each request in a forked child. Each child process has to compile a lazy regular expression or important a deferred module over and over. uWSGI's emperor / vassal mode us a pre-fork model with multiple server processes to efficiently share memory with copy-on-write semantics. Lazy imports will make the approach less efficient and slow down forking of new vassals. TL;DR please refrain from moving imports into functions or implementing lazy modes, until we have figured out how to satisfy requirements of both scripts and long running services. We probably need a PEP... Christian

12 18

[RELEASE] Python 3.7.0a2 is now available for testing
by Ned Deily Oct. 17, 2017

Oct. 17, 2017

Python 3.7.0a2 is the second of four planned alpha previews of Python 3.7, the next feature release of Python. During the alpha phase, Python 3.7 remains under heavy development: additional features will be added and existing features may be modified or deleted. Please keep in mind that this is a preview release and its use is not recommended for production environments. The next preview, 3.7.0a3, is planned for 2017-11-27. You can find Python 3.7.0a2 and more information here: https://www.python.org/downloads/release/python-370a2/ -- Ned Deily nad(a)python.org -- []

1 0

Reorganize Python categories (Core, Library, ...)?
by Victor Stinner Oct. 17, 2017

Oct. 17, 2017

Hi, Python uses a few categories to group bugs (on bugs.python.org) and NEWS entries (in the Python changelog). List used by the blurb tool: #.. section: Security #.. section: Core and Builtins #.. section: Library #.. section: Documentation #.. section: Tests #.. section: Build #.. section: Windows #.. section: macOS #.. section: IDLE #.. section: Tools/Demos #.. section: C API My problem is that almost all changes go into "Library" category. When I read long changelogs, it's sometimes hard to identify quickly the context (ex: impacted modules) of a change. It's also hard to find open bugs of a specific module on bugs.python.org, since almost all bugs are in the very generic "Library" category. Using full text returns "false positives". I would prefer to see more specific categories like: * Buildbots: only issues specific to buildbots * Networking: socket, asyncio, asyncore, asynchat modules * Security: ssl module but also vulnerabilities in any other part of CPython -- we already added a Security category in NEWS/blurb * Parallelim: multiprocessing and concurrent.futures modules It's hard to find categories generic enough to not only contain a single item, but not contain too many items neither. Other ideas: * XML: xml.doc, xml.etree, xml.parsers, xml.sax modules * Import machinery: imp and importlib modules * Typing: abc and typing modules The best would be to have a mapping of a module name into a category, and make sure that all modules have a category. We might try to count the number of commits and NEWS entries of the last 12 months to decide if a category has the correct size. I don't think that we need a distinct categoy for each module. We can put many uncommon modules in a generic category. By the way, we need maybe also a new "module name" field in the bug tracker. But then comes the question of normalizing module names. For example, should "email.message" be normalized to "email"? Maybe store "email.message" but use "email" for search, display the module in the issue title, etc. Victor

7 9

Two new environment variables to debug Python 2.7
by Victor Stinner Oct. 17, 2017

Oct. 17, 2017

Hi, FYI I just merged two pull requests into Python 2.7, each add a new environment variable changing the behaviour in debug mode: bpo-31733: don't dump "[xxx refs]" into stderr by default anymore Set PYTHONSHOWREFCOUNT=1 commit 3c082a7fdb472f02bcac7a7f8fe1e3a34a11b70b bpo-31692, bpo-19527: don't dump allocations counts by default anymore Set PYTHONSHOWALLOCCOUNT=1 commit 7b4ba62e388474e811268322b47f80d464933541 I never ever used "[xxx refs]" to detect a reference leak. I only use "./python -m test -R 3:3 test_xxx" to detect reference leaks. To be honest, usually I only run such test explicitly when one of our "Refleaks" buildbot starts to comlain :-) Victor

1 0