Mailman 3 November 2013 - Python-Dev

Track ResourceWarning warnings with tracemalloc
by Victor Stinner 30 Nov '13

30 Nov '13

Hi, I'm trying to write an example of usage of the new tracemalloc.get_object_traceback() function. Last month I proposed to use it to give the traceback where a file/socket was allocated on ResourceWarning: https://mail.python.org/pipermail/python-dev/2013-October/129923.html I found a working solution using monkey patching of the socket, _pyio and builtins modules: https://bitbucket.org/haypo/misc/src/tip/python/res_warn.py This hack uses tracemalloc to retrieve the traceback where the file/socket was allocated, but you can do the same using traceback.extract_stack() for older Python version. So it's just an example to show how tracemalloc can be used. Example with test_poplib (without res_warn.py): [1/1] test_poplib /home/haypo/prog/python/default/Lib/test/support/__init__.py:1331: ResourceWarning: unclosed <ssl.SSLSocket fd=5, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('127.0.0.1', 45747), raddr=('127.0.0.1', 48933)> gc.collect() Ok, you now that the socket was destroyed be the garbage collector... But which test created this socket??? Example of res_warn.py with test_poplib: [1/1] test_poplib /home/haypo/prog/python/default/Lib/test/support/__init__.py:1331: ResourceWarning: unclosed <ssl.SSLSocket fd=5, family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('127.0.0.1', 49715), raddr=('127.0.0.1', 43904)> gc.collect() Allocation traceback (most recent first): File '/home/haypo/prog/python/default/Lib/ssl.py', line 340 _context=self) File '/home/haypo/prog/python/default/Lib/poplib.py', line 390 self.sock = context.wrap_socket(self.sock) File '/home/haypo/prog/python/default/Lib/test/test_poplib.py', line 407 self.client.stls() File '/home/haypo/prog/python/default/Lib/unittest/case.py', line 567 self.setUp() File '/home/haypo/prog/python/default/Lib/unittest/case.py', line 610 return self.run(*args, **kwds) (...) You can see that the socket was created by test_poplib at line 407 TestPOP3_TLSClass.setUp). It's more useful than the previous warning :-) I found two issues in _pyio and tracemalloc modules. These issues should be fixed to use res_warn.py on text or buffered files and use it during Python shutdown: http://bugs.python.org/issue19831 http://bugs.python.org/issue19829 Victor

3 2

Summary of Python tracker Issues
by Python tracker 30 Nov '13

30 Nov '13

ACTIVITY SUMMARY (2013-11-22 - 2013-11-29) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 4317 (+37) closed 27289 (+83) total 31606 (+120) Open issues with patches: 1981 Issues opened (78) ================== #19352: unittest loader barfs on symlinks http://bugs.python.org/issue19352 reopened by doko #19579: test_asyncio: test__run_once timings should be relaxed http://bugs.python.org/issue19579 reopened by haypo #19638: dtoa: conversion from '__int64' to 'int', possible loss of dat http://bugs.python.org/issue19638 reopened by mark.dickinson #19715: test_touch_common failure under Windows http://bugs.python.org/issue19715 opened by pitrou #19717: resolve() fails when the path doesn't exist http://bugs.python.org/issue19717 opened by pitrou #19719: add importlib.abc.SpecLoader and SpecFinder http://bugs.python.org/issue19719 opened by brett.cannon #19720: suppress context for some exceptions in importlib? http://bugs.python.org/issue19720 opened by eric.snow #19721: Move all test_importlib utility code into test_importlib.util http://bugs.python.org/issue19721 opened by brett.cannon #19723: Argument Clinic should add markers for humans http://bugs.python.org/issue19723 opened by pitrou #19725: Richer stat object http://bugs.python.org/issue19725 opened by pjenvey #19726: BaseProtocol is not an ABC http://bugs.python.org/issue19726 opened by pitrou #19728: PEP 453: enable pip by default in the Windows binary installer http://bugs.python.org/issue19728 opened by ncoghlan #19731: Fix copyright footer http://bugs.python.org/issue19731 opened by pitrou #19732: python fails to build when configured with --with-system-libmp http://bugs.python.org/issue19732 opened by doko #19733: Setting image parameter of a button crashes with Cocoa Tk http://bugs.python.org/issue19733 opened by serhiy.storchaka #19734: venv and ensurepip are affected by pip environment variable se http://bugs.python.org/issue19734 opened by haypo #19736: posixmodule.c: Add flags for statvfs.f_flag to constant list http://bugs.python.org/issue19736 opened by doko #19737: Documentation of globals() and locals() should be improved http://bugs.python.org/issue19737 opened by Zahari.Dim #19738: pytime.c loses precision under Windows http://bugs.python.org/issue19738 opened by pitrou #19741: tracemalloc: tracemalloc_log_alloc() doesn't check _Py_HASHTAB http://bugs.python.org/issue19741 opened by haypo #19743: test_gdb failures http://bugs.python.org/issue19743 opened by larry #19744: ensurepip should refuse to install pip if SSL/TLS is not avail http://bugs.python.org/issue19744 opened by tim.peters #19745: TEST_DATA_DIR for out-of-tree builds http://bugs.python.org/issue19745 opened by christian.heimes #19746: No introspective way to detect ModuleImportFailure in unittest http://bugs.python.org/issue19746 opened by rbcollins #19748: test_time failures on AIX http://bugs.python.org/issue19748 opened by haypo #19749: test_venv failure on AIX: 'module' object has no attribute 'O_ http://bugs.python.org/issue19749 opened by haypo #19750: test_asyncio.test_unix_events constructor failures on AIX http://bugs.python.org/issue19750 opened by haypo #19751: test_sys: sys.hash_info.algorithm failure on SPARC Solaris bui http://bugs.python.org/issue19751 opened by haypo #19753: test_gdb failure on SystemZ buildbot http://bugs.python.org/issue19753 opened by haypo #19754: pickletools.optimize doesn't reframe correctly http://bugs.python.org/issue19754 opened by pitrou #19756: test_nntplib: sporadic failures, network isses? server down? http://bugs.python.org/issue19756 opened by haypo #19757: _tracemalloc.c: compiler warning with gil_state http://bugs.python.org/issue19757 opened by haypo #19758: Warnings in tests http://bugs.python.org/issue19758 opened by serhiy.storchaka #19761: test_tk fails on OS X with multiple test case failures with bo http://bugs.python.org/issue19761 opened by ned.deily #19763: Make it easier to backport statistics to 2.7 http://bugs.python.org/issue19763 opened by christian.heimes #19764: subprocess: use PROC_THREAD_ATTRIBUTE_HANDLE_LIST with STARTUP http://bugs.python.org/issue19764 opened by haypo #19766: test_venv: test_with_pip() failed on "AMD64 Fedora without thr http://bugs.python.org/issue19766 opened by haypo #19767: pathlib: iterfiles() and iterdirs() http://bugs.python.org/issue19767 opened by jonash #19768: Not so correct error message when giving incorrect type to max http://bugs.python.org/issue19768 opened by vajrasky #19769: test_venv: test_with_pip() failure on "AMD64 Windows Server 20 http://bugs.python.org/issue19769 opened by haypo #19770: NNTP.post broken http://bugs.python.org/issue19770 opened by sobczyk #19771: runpy should check ImportError.name before wrapping it http://bugs.python.org/issue19771 opened by ncoghlan #19772: str serialization of Message object may mutate the payload and http://bugs.python.org/issue19772 opened by r.david.murray #19775: Provide samefile() on Path objects http://bugs.python.org/issue19775 opened by pitrou #19776: Provide expanduser() on Path objects http://bugs.python.org/issue19776 opened by pitrou #19777: Provide a home() classmethod on Path objects http://bugs.python.org/issue19777 opened by pitrou #19780: Pickle 4 frame headers optimization http://bugs.python.org/issue19780 opened by serhiy.storchaka #19781: No SSL match_hostname() in ftplib http://bugs.python.org/issue19781 opened by christian.heimes #19782: No SSL match_hostname() in imaplib http://bugs.python.org/issue19782 opened by christian.heimes #19783: No SSL match_hostname() in nntplib http://bugs.python.org/issue19783 opened by christian.heimes #19784: No SSL match_hostname() in poplib http://bugs.python.org/issue19784 opened by christian.heimes #19785: No SSL match_hostname() in smtplib http://bugs.python.org/issue19785 opened by christian.heimes #19787: tracemalloc: set_reentrant() should not have to call PyThread_ http://bugs.python.org/issue19787 opened by haypo #19789: Improve wording of how to "undo" a call to logging.disable(lvl http://bugs.python.org/issue19789 opened by simonmweber #19791: test_pathlib should use can_symlink or skip_unless_symlink fro http://bugs.python.org/issue19791 opened by vajrasky #19792: pathlib does not support symlink in Windows XP http://bugs.python.org/issue19792 opened by vajrasky #19795: Formatting of True/False in docs http://bugs.python.org/issue19795 opened by serhiy.storchaka #19796: urllib2.HTTPError.reason is not documented as "Added in 2.7" http://bugs.python.org/issue19796 opened by moreati #19800: Write more detailed framing tests http://bugs.python.org/issue19800 opened by pitrou #19806: smtpd crashes when a multi-byte UTF-8 sequence is split betwee http://bugs.python.org/issue19806 opened by petri.lehtinen #19808: IDLE applies syntax highlighting to user input in its shell http://bugs.python.org/issue19808 opened by peter.otten #19809: Doc: subprocess should warn uses on race conditions when multi http://bugs.python.org/issue19809 opened by owenlin #19814: Document prefix matching behavior of argparse, particularly fo http://bugs.python.org/issue19814 opened by eli.bendersky #19816: test.regrtest: use tracemalloc to detect memory leaks? http://bugs.python.org/issue19816 opened by haypo #19817: tracemalloc add a memory limit feature http://bugs.python.org/issue19817 opened by haypo #19818: tracemalloc: comments on the doc http://bugs.python.org/issue19818 opened by haypo #19820: docs are missing info about module attributes http://bugs.python.org/issue19820 opened by eric.snow #19821: pydoc.ispackage() could be more accurate http://bugs.python.org/issue19821 opened by eric.snow #19824: string.Template: Add PHP-style variable expansion example http://bugs.python.org/issue19824 opened by techtonik #19826: Document that bug reporting by email is possible http://bugs.python.org/issue19826 opened by techtonik #19827: Optimize socket.settimeout() and socket.setblocking(): avoid s http://bugs.python.org/issue19827 opened by haypo #19828: test_site fails with -S flag http://bugs.python.org/issue19828 opened by vajrasky #19829: _pyio.BufferedReader and _pyio.TextIOWrapper destructor don't http://bugs.python.org/issue19829 opened by haypo #19830: test_poplib emits resource warning http://bugs.python.org/issue19830 opened by vajrasky #19831: tracemalloc: stop the module later at Python shutdown http://bugs.python.org/issue19831 opened by haypo #19832: XML version is ignored http://bugs.python.org/issue19832 opened by tkuhn #19833: asyncio: patches to document EventLoop and add more examples http://bugs.python.org/issue19833 opened by haypo #19834: Unpickling exceptions pickled by Python 2 http://bugs.python.org/issue19834 opened by doerwalter Most recent 15 issues with no replies (15) ========================================== #19834: Unpickling exceptions pickled by Python 2 http://bugs.python.org/issue19834 #19830: test_poplib emits resource warning http://bugs.python.org/issue19830 #19828: test_site fails with -S flag http://bugs.python.org/issue19828 #19827: Optimize socket.settimeout() and socket.setblocking(): avoid s http://bugs.python.org/issue19827 #19820: docs are missing info about module attributes http://bugs.python.org/issue19820 #19817: tracemalloc add a memory limit feature http://bugs.python.org/issue19817 #19816: test.regrtest: use tracemalloc to detect memory leaks? http://bugs.python.org/issue19816 #19808: IDLE applies syntax highlighting to user input in its shell http://bugs.python.org/issue19808 #19800: Write more detailed framing tests http://bugs.python.org/issue19800 #19796: urllib2.HTTPError.reason is not documented as "Added in 2.7" http://bugs.python.org/issue19796 #19789: Improve wording of how to "undo" a call to logging.disable(lvl http://bugs.python.org/issue19789 #19785: No SSL match_hostname() in smtplib http://bugs.python.org/issue19785 #19784: No SSL match_hostname() in poplib http://bugs.python.org/issue19784 #19783: No SSL match_hostname() in nntplib http://bugs.python.org/issue19783 #19782: No SSL match_hostname() in imaplib http://bugs.python.org/issue19782 Most recent 15 issues waiting for review (15) ============================================= #19833: asyncio: patches to document EventLoop and add more examples http://bugs.python.org/issue19833 #19831: tracemalloc: stop the module later at Python shutdown http://bugs.python.org/issue19831 #19830: test_poplib emits resource warning http://bugs.python.org/issue19830 #19828: test_site fails with -S flag http://bugs.python.org/issue19828 #19827: Optimize socket.settimeout() and socket.setblocking(): avoid s http://bugs.python.org/issue19827 #19817: tracemalloc add a memory limit feature http://bugs.python.org/issue19817 #19814: Document prefix matching behavior of argparse, particularly fo http://bugs.python.org/issue19814 #19806: smtpd crashes when a multi-byte UTF-8 sequence is split betwee http://bugs.python.org/issue19806 #19796: urllib2.HTTPError.reason is not documented as "Added in 2.7" http://bugs.python.org/issue19796 #19795: Formatting of True/False in docs http://bugs.python.org/issue19795 #19792: pathlib does not support symlink in Windows XP http://bugs.python.org/issue19792 #19791: test_pathlib should use can_symlink or skip_unless_symlink fro http://bugs.python.org/issue19791 #19787: tracemalloc: set_reentrant() should not have to call PyThread_ http://bugs.python.org/issue19787 #19785: No SSL match_hostname() in smtplib http://bugs.python.org/issue19785 #19784: No SSL match_hostname() in poplib http://bugs.python.org/issue19784 Top 10 most discussed issues (10) ================================= #19715: test_touch_common failure under Windows http://bugs.python.org/issue19715 40 msgs #19638: dtoa: conversion from '__int64' to 'int', possible loss of dat http://bugs.python.org/issue19638 23 msgs #19734: venv and ensurepip are affected by pip environment variable se http://bugs.python.org/issue19734 18 msgs #19743: test_gdb failures http://bugs.python.org/issue19743 15 msgs #19787: tracemalloc: set_reentrant() should not have to call PyThread_ http://bugs.python.org/issue19787 15 msgs #19509: No SSL match_hostname() in ftp, imap, nntp, pop, smtp modules http://bugs.python.org/issue19509 14 msgs #19780: Pickle 4 frame headers optimization http://bugs.python.org/issue19780 14 msgs #19744: ensurepip should refuse to install pip if SSL/TLS is not avail http://bugs.python.org/issue19744 12 msgs #19624: Switch constants in the errno module to IntEnum http://bugs.python.org/issue19624 10 msgs #19655: Replace the ASDL parser carried with CPython http://bugs.python.org/issue19655 10 msgs Issues closed (81) ================== #3158: Doctest fails to find doctests in extension modules http://bugs.python.org/issue3158 closed by zach.ware #11357: Add support for PEP 381 Mirror Authenticity http://bugs.python.org/issue11357 closed by eric.araujo #11489: json.dumps not parsable by json.loads (on Linux only) http://bugs.python.org/issue11489 closed by serhiy.storchaka #11508: Virtual Interfaces cause uuid._find_mac to raise a ValueError http://bugs.python.org/issue11508 closed by serhiy.storchaka #13633: Automatically convert character references in HTMLParser http://bugs.python.org/issue13633 closed by ezio.melotti #15204: Deprecate the 'U' open mode http://bugs.python.org/issue15204 closed by serhiy.storchaka #15397: Unbinding of methods http://bugs.python.org/issue15397 closed by alexandre.vassalotti #17201: Use allowZip64=True by default http://bugs.python.org/issue17201 closed by serhiy.storchaka #17523: Additional tests for the os module. http://bugs.python.org/issue17523 closed by willweaver #17747: Deprecate pickle fast mode http://bugs.python.org/issue17747 closed by alexandre.vassalotti #17787: Optimize pickling function dispatch in hot loops. http://bugs.python.org/issue17787 closed by alexandre.vassalotti #17810: Implement PEP 3154 (pickle protocol 4) http://bugs.python.org/issue17810 closed by alexandre.vassalotti #18326: Mention 'keyword only' for list.sort, improve glossary. http://bugs.python.org/issue18326 closed by zach.ware #18874: Add a new tracemalloc module to trace memory allocations http://bugs.python.org/issue18874 closed by haypo #19146: Improvements to traceback module http://bugs.python.org/issue19146 closed by gvanrossum #19308: Tools/gdb/libpython.py does not support GDB linked against Pyt http://bugs.python.org/issue19308 closed by pitrou #19358: Integrate "Argument Clinic" into CPython build http://bugs.python.org/issue19358 closed by larry #19467: asyncore documentation: redirect users to the new asyncio modu http://bugs.python.org/issue19467 closed by gvanrossum #19486: Bump up version of Tcl/Tk in building Python in Windows platfo http://bugs.python.org/issue19486 closed by zach.ware #19545: time.strptime exception context http://bugs.python.org/issue19545 closed by serhiy.storchaka #19551: PEP 453: Mac OS X installer integration http://bugs.python.org/issue19551 closed by ned.deily #19588: Silently skipped test in test_random http://bugs.python.org/issue19588 closed by zach.ware #19616: Fix test.test_importlib.util.test_both() to set __module__ http://bugs.python.org/issue19616 closed by brett.cannon #19620: tokenize documentation contains typos (argment instead of argu http://bugs.python.org/issue19620 closed by ezio.melotti #19622: Default buffering for input and output pipes in subprocess mod http://bugs.python.org/issue19622 closed by python-dev #19631: "exec" BNF in simple statements http://bugs.python.org/issue19631 closed by georg.brandl #19632: doc build warning http://bugs.python.org/issue19632 closed by georg.brandl #19639: Improve re match objects display http://bugs.python.org/issue19639 closed by ezio.melotti #19641: Add audioop.byteswap() http://bugs.python.org/issue19641 closed by serhiy.storchaka #19668: Add support of the cp1125 encoding http://bugs.python.org/issue19668 closed by serhiy.storchaka #19671: Option to select the optimization level on compileall http://bugs.python.org/issue19671 closed by Sworddragon #19674: Add introspection information for builtins http://bugs.python.org/issue19674 closed by larry #19691: Weird wording in RuntimeError doc http://bugs.python.org/issue19691 closed by pitrou #19694: test_venv failing on one of the Ubuntu buildbots http://bugs.python.org/issue19694 closed by python-dev #19706: Check if inspect needs updating for PEP 451 http://bugs.python.org/issue19706 closed by brett.cannon #19709: Check pythonrun.c is fully using PEP 451 http://bugs.python.org/issue19709 closed by brett.cannon #19716: test that touch doesn't change file contents http://bugs.python.org/issue19716 closed by pitrou #19718: Path.glob() on case-insensitive Posix filesystems http://bugs.python.org/issue19718 closed by pitrou #19722: Expose stack effect calculator to Python http://bugs.python.org/issue19722 closed by larry #19724: test_pkgutil buildbot failure (related to PEP 451) http://bugs.python.org/issue19724 closed by eric.snow #19727: os.utime(..., None) has poor resolution on Windows http://bugs.python.org/issue19727 closed by pitrou #19729: [regression] str.format sublevel format parsing broken in Pyth http://bugs.python.org/issue19729 closed by python-dev #19730: Clinic fixes: add converters with length, version directive, " http://bugs.python.org/issue19730 closed by larry #19735: ssl._create_stdlib_context http://bugs.python.org/issue19735 closed by christian.heimes #19739: Legit compiler warnings in new pickle code on 32-bit Windows http://bugs.python.org/issue19739 closed by pitrou #19740: test_asyncio problems on 32-bit Windows http://bugs.python.org/issue19740 closed by gvanrossum #19742: pathlib group unittests can fail http://bugs.python.org/issue19742 closed by pitrou #19747: New failures in test_pickletools on 32-bit Windows Vista http://bugs.python.org/issue19747 closed by tim.peters #19752: os.openpty() failure on Solaris 10: PermissionError: [Errno 13 http://bugs.python.org/issue19752 closed by haypo #19755: PEP 397 listed as accepted instead of finished. http://bugs.python.org/issue19755 closed by python-dev #19759: Deprecation warnings in test_hmac http://bugs.python.org/issue19759 closed by serhiy.storchaka #19760: Deprecation warnings in ttest_sysconfig and test_distutils http://bugs.python.org/issue19760 closed by serhiy.storchaka #19762: Incorrect function documentation of _get_object_traceback and http://bugs.python.org/issue19762 closed by python-dev #19765: test_asyncio: test_create_server() failed on "x86 Windows Serv http://bugs.python.org/issue19765 closed by haypo #19773: Failed to build python 2.6.9 Solaris 10 http://bugs.python.org/issue19773 closed by christian.heimes #19774: strptime incorrect for weekday '0' when using week number form http://bugs.python.org/issue19774 closed by r.david.murray #19778: Change re.compile display http://bugs.python.org/issue19778 closed by ezio.melotti #19779: test_concurrent_futures crashes on 32-bit Windows Vista http://bugs.python.org/issue19779 closed by tim.peters #19786: tracemalloc: remove arbitrary limit of 100 frames http://bugs.python.org/issue19786 closed by python-dev #19788: Always run kill_python(_d).exe when (re-)building on Windows http://bugs.python.org/issue19788 closed by zach.ware #19790: ctypes can't load a c++ dll if the dll calls COM on Windows 8 http://bugs.python.org/issue19790 closed by yinkaisheng #19793: Formatting of True/False in pathlib docs http://bugs.python.org/issue19793 closed by serhiy.storchaka #19794: Formatting of True/False in decimal docs http://bugs.python.org/issue19794 closed by serhiy.storchaka #19797: Solaris 10. mod_python build failed. http://bugs.python.org/issue19797 closed by christian.heimes #19798: tracemalloc: rename "max_size" to "peak_size" in get_traced_me http://bugs.python.org/issue19798 closed by python-dev #19799: Clarifications for the documentation of pathlib http://bugs.python.org/issue19799 closed by eli.bendersky #19801: Concatenating bytes is much slower than concatenating strings http://bugs.python.org/issue19801 closed by benjamin.peterson #19802: socket.SO_PRIORITY is missing http://bugs.python.org/issue19802 closed by python-dev #19803: memoryview complain ctypes byte array are not native single ch http://bugs.python.org/issue19803 closed by skrah #19804: test_uuid failing on 32-bit Windows Vista http://bugs.python.org/issue19804 closed by serhiy.storchaka #19805: Revise FAQ Dictionary: consistent key order item http://bugs.python.org/issue19805 closed by python-dev #19807: calculation of standard math returns incorrect value http://bugs.python.org/issue19807 closed by mark.dickinson #19810: Display function signature in TypeErrors resulting from argume http://bugs.python.org/issue19810 closed by benjamin.peterson #19811: test_pathlib: "The directory is not empty" error on os.rmdir() http://bugs.python.org/issue19811 closed by pitrou #19812: reload() by symbol name http://bugs.python.org/issue19812 closed by brett.cannon #19813: Add a new optional timeout parameter to socket.socket() constr http://bugs.python.org/issue19813 closed by gvanrossum #19815: ElementTree segmentation fault in expat_start_ns_handler http://bugs.python.org/issue19815 closed by eli.bendersky #19819: reversing a Unicode ligature doesn't work http://bugs.python.org/issue19819 closed by haypo #19822: PEP process entrypoint http://bugs.python.org/issue19822 closed by christian.heimes #19823: for-each on list aborts earlier than expected http://bugs.python.org/issue19823 closed by cost6 #19825: test b.p.o email interface http://bugs.python.org/issue19825 closed by techtonik

1 0

py.ini documentation improvement
by Tim Delaney 29 Nov '13

29 Nov '13

I was just getting Jython working with py.exe, and I think the documentation can be made a bit more friendly. In particular think we can make it easier for people to determine the correct folder by changing the line in 3.4.4.1 Customization via INI files: Two .ini files will be searched by the launcher - py.ini in the current user’s "application data" directory (i.e. the directory returned by calling the Windows function SHGetFolderPath with CSIDL_LOCAL_APPDATA) and py.ini in the same directory as the launcher. to Two .ini files will be searched by the launcher - py.ini in the current user’s "application data" directory (i.e. the directory returned by executing `echo %LOCALAPPDATA%` in a command window) and py.ini in the same directory as the launcher. %LOCALAPPDATA% should always contain the same value as would be returned from SHGetFolderPath with CSIDL_LOCAL_APPDATA. Tim Delaney

6 10

buildbot's are needlessly compiling -O0
by Gregory P. Smith 29 Nov '13

29 Nov '13

our buildbots are setup to configure --with-pydebug which also unfortunately causes them to compile with -O0... this results in a python binary that is excruciatingly slow and makes the entire test suite run take a long time. given that nobody is ever going to run a gdb or another debugger on the buildbot generated transient binaries themselves how about we speed all of the buildbot's up by adding CFLAGS=-O2 to the configure command line? Sure, the compile step will take a bit longer but that is dwarfed by the test time as it is: http://buildbot.python.org/all/builders/AMD64%20Ubuntu%20LTS%203.x/builds/3… http://buildbot.python.org/all/builders/ARMv7%203.x/builds/7 http://buildbot.python.org/all/builders/AMD64%20Snow%20Leop%203.x/builds/639 It should dramatically decrease the turnaround latency for buildbot results. -gps

6 5

test_uuid.py on HP NonStop Python-2.7.5 fails (test case: testIssue8621)
by V S, Nagendra (Nonstop Filesystems Team) 28 Nov '13

28 Nov '13

Hi, We are porting python to HP NonStop & to a large extent we have been successful. While running the unit tests we happen to hit upon the problem reported in issue8621(http://bugs.python.org/issue8621), i.e uuid4 sequences on both parent & child were same . On NonStop we lack support for both hardware randomness & uuid_generate family of calls, so on NonStop "uuid.py" falls throw to random.range() call to generate the random number which in turn is used by the UUID class. Below the code snip that can used to repo the problem... #!/usr/bin/python import os import sys import uuid import random #Code taken from Lib/uuid.py def myuuid(): bytes = [chr(random.randrange(256)) for i in range(16)] val = uuid.UUID(bytes=bytes, version=4) return val # Code taken from Lib/test/test_uuid.py # # On at least some versions of OSX myuuid generates # the same sequence of UUIDs in the parent and any # children started using fork. for i in range(10): fds = os.pipe() pid = os.fork() if pid == 0: os.close(fds[0]) value = myuuid() os.write(fds[1], value.hex) os._exit(0) else: os.close(fds[1]) parent_value = myuuid().hex os.waitpid(pid, 0) child_value = os.read(fds[0], 100) os.close(fds[0]) print parent_value, child_value, (parent_value == child_value) We also ran the above test on Linux & were seeing the same results as on NonStop, i.e uuid4 sequences on both parent & child were same. Output snippet..... [vsnag@<hostname> Python-2.7.5]$ uname -a Linux <hostname> 2.6.32-71.el6.x86_64 #1 SMP Wed Sep 1 01:33:01 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux [vsnag@<hostname> Python-2.7.5]$ ./python test.py 2b512ef47c9f471a829148ca4e779dff 2b512ef47c9f471a829148ca4e779dff True e39b605476e04ca2b992b5d055278c3b e39b605476e04ca2b992b5d055278c3b True 8249fb9f6a2045c084da549c9e393a51 8249fb9f6a2045c084da549c9e393a51 True e1d9e84b7f134930abf5fb2760f07585 e1d9e84b7f134930abf5fb2760f07585 True 068f9612f0744eeb92dbca8950028a3c 068f9612f0744eeb92dbca8950028a3c True e85df457614f47d5b6300186bdd1c798 e85df457614f47d5b6300186bdd1c798 True a8628a0817b843778e9565b835b8cfac a8628a0817b843778e9565b835b8cfac True 478d8e9e5090498e9fad5356c5ae9568 478d8e9e5090498e9fad5356c5ae9568 True 151d7f1c8f8b42099a89a0f442e98dc7 151d7f1c8f8b42099a89a0f442e98dc7 True 838571b03a5b46f8a06398e020647d89 838571b03a5b46f8a06398e020647d89 True [vsnag@<hostname> Python-2.7.5]$ Any thoughts what could be going wrong....? Thanks & Regards Nagendra.V.S

3 3

Re: [Python-Dev] [Python-checkins] cpython: asyncio: Change write buffer use to avoid O(N**2). Make write()/sendto() accept
by Victor Stinner 28 Nov '13

28 Nov '13

2013/11/27 guido.van.rossum <python-checkins(a)python.org>: > http://hg.python.org/cpython/rev/80e0040d910c > changeset: 87617:80e0040d910c > user: Guido van Rossum <guido(a)python.org> > date: Wed Nov 27 14:12:48 2013 -0800 > summary: > asyncio: Change write buffer use to avoid O(N**2). Make write()/sendto() accept bytearray/memoryview too. Change some asserts with proper exceptions. Was this change discussed somewhere? Antoine told me on IRC that it was discussed on the tulip mailing list. I would be nice to mention it in the commit message (for the next change ;-). > diff --git a/Lib/asyncio/selector_events.py b/Lib/asyncio/selector_events.py > --- a/Lib/asyncio/selector_events.py > +++ b/Lib/asyncio/selector_events.py > @@ -340,6 +340,8 @@ > > max_size = 256 * 1024 # Buffer size passed to recv(). > > + _buffer_factory = bytearray # Constructs initial value for self._buffer. > + Because the buffer type is now configurable, it would be nice to explain in a short comment why bytearray fits better this use case. (I like the bytearray object, so I like your whole changeset!) > @@ -762,6 +776,8 @@ > > class _SelectorDatagramTransport(_SelectorTransport): > > + _buffer_factory = collections.deque > + > def __init__(self, loop, sock, protocol, address=None, extra=None): > super().__init__(loop, sock, protocol, extra) > self._address = address Why collections.deque is preferred here? Could you also please add a comment? Victor

1 0

potential argparse problem: bad mix of parse_known_args and prefix matching
by Eli Bendersky 27 Nov '13

27 Nov '13

Hello, argparse does prefix matching as long as there are no conflicts. For example: argparser = argparse.ArgumentParser() argparser.add_argument('--sync-foo', action='store_true') args = argparser.parse_args() If I pass "--sync" to this script, it recognizes it as "--sync-foo". This behavior is quite surprising although I can see the motivation for it. At the very least it should be much more explicitly documented (AFAICS it's barely mentioned in the docs). If there's another argument registered, say "--sync-bar" the above will fail due to a conflict. Now comes the nasty part. When using "parse_known_args" instead of "parse_args", the above happens too - --sync is recognized for --sync-foo and captured by the parser. But this is wrong! The whole idea of parse_known_args is to parse the known args, leaving unknowns alone. This prefix matching harms more than it helps here because maybe the program we're actually acting as a front-end for (and hence using parse_known_args) knows about --sync and wants to get it. Unless I'm missing something, this is a bug. But I'm also not sure whether we can do anything about it at this point, as existing code *may* be relying on it. The right thing to do would be to disable this prefix matching when parse_known_args is called. Again, at the very least this should be documented (for parse_known_args not less than a warning box, IMHO). Eli

6 9

Accepting PEP 3154 for 3.4?
by Antoine Pitrou 27 Nov '13

27 Nov '13

Hello, Alexandre Vassalotti (thanks a lot!) has recently finalized his work on the PEP 3154 implementation - pickle protocol 4. I think it would be good to get the PEP and the implementation accepted for 3.4. As far as I can say, this has been a low-controvery proposal, and it brings fairly obvious improvements to the table (which table?). I still need some kind of BDFL or BDFL delegate to do that, though -- unless I am allowed to mark my own PEP accepted :-) (I've asked Tim, specifically, for comments, since he contributed a lot to previous versions of the pickle protocol.) The PEP is at http://www.python.org/dev/peps/pep-3154/ (should be rebuilt soon by the server, I'd say) Alexandre's implementation is tracked at http://bugs.python.org/issue17810 Regards Antoine.

12 69

Re: [Python-Dev] [Python-checkins] cpython (2.7): Fix test_fcntl to run properly on systems that do not support the flags
by Nick Coghlan 26 Nov '13

26 Nov '13

On 25 Nov 2013 14:46, "gregory.p.smith" <python-checkins(a)python.org> wrote: > > http://hg.python.org/cpython/rev/cac7319c5972 > changeset: 87537:cac7319c5972 > branch: 2.7 > parent: 87534:3981e57a7bdc > user: Gregory P. Smith <greg(a)krypto.org> > date: Mon Nov 25 04:45:27 2013 +0000 > summary: > Fix test_fcntl to run properly on systems that do not support the flags > used in the "does the value get passed in properly" test. > > files: > Lib/test/test_fcntl.py | 3 +++ > 1 files changed, 3 insertions(+), 0 deletions(-) > > > diff --git a/Lib/test/test_fcntl.py b/Lib/test/test_fcntl.py > --- a/Lib/test/test_fcntl.py > +++ b/Lib/test/test_fcntl.py > @@ -113,7 +113,10 @@ > self.skipTest("F_NOTIFY or DN_MULTISHOT unavailable") > fd = os.open(os.path.dirname(os.path.abspath(TESTFN)), os.O_RDONLY) > try: > + # This will raise OverflowError if issue1309352 is present. > fcntl.fcntl(fd, cmd, flags) > + except IOError: > + pass # Running on a system that doesn't support these flags. > finally: > os.close(fd) Raising a skip is generally preferred to marking a test that can't be executed as passing. Cheers, Nick. > > > -- > Repository URL: http://hg.python.org/cpython > > _______________________________________________ > Python-checkins mailing list > Python-checkins(a)python.org > https://mail.python.org/mailman/listinfo/python-checkins >

2 1

pathlib and issue 11406 (a directory iterator returning stat-like info)
by Ben Hoyt 26 Nov '13

26 Nov '13

Hi folks, I decided to start another thread for my thoughts on the interaction between pathlib (Antoine's new PEP 428), issue 11406 (proposal for a directory iterator returning stat-like info), and my own scandir library, which implements something along the lines of issue 11406. My scandir library (https://github.com/benhoyt/scandir) is something I've been working on for a while -- it provides a scandir() function which uses the OS's directory iterator functions to expose as much stat-like information as possible (readdir and FindFirstFile etc). This way functions like os.walk() can use the info (particularly "is_dir()") and not require tons of extra calls to os.stat(). This provides a huge speed boost for os.walk() in many cases: I've seen 3-4x on Linux, and up to 20x on Windows. (It depends on various things, not least of which is Windows' weird stat caching -- if I run my scandir benchmark "fresh", I get os.walk() running 8-9 times as fast as the built-in one. But if I run it after an un-hibernate, suddenly it runs 18-20 times as fast as the built-in one. Either way, huge gains, especially on Windows.) scandir.scandir() returns a DirEntry object, which has .isdir(), .isfile(), .islink(), and .lstat() attributes. Look familiar? When I was reading PEP 428 and saw .is_file(), .is_dir(), and .stat(), I thought -- surely I can merge this with pathlib and Path objects. The first thing I can do to scandir is rename my isdir() type attributes to match PEP 428's, so that DirEntry quacks like a Path object where it can. However, I'm wondering if I can change scandir to return actual Path objects. Or better, because Path already helpfully provides iterdir() which yields Path objects, and Path objects have .is_dir() etc, can scandir()-like behaviour simply work out-of-the-box? This mainly depends on how Path is going to cache stat information. If it caches it, then this will just work. Sounds like Guido's opinion was that both cached and uncached use cases are important, but that it should be very clear which one you're getting. I personally like the .stat() and .restat() idea. The other related thing is that DirEntry only provides .lstat(), because it's providing stat-like info without following links. Note in this context that it's not just "network filesystems" on which stat() is slow (https://mail.python.org/pipermail/python-dev/2013-May/125805.html). It's quite slow in Windows under various conditions too. See also Nick Coghlan's post about a DirEntry-style object on the issue 11406 thread: https://mail.python.org/pipermail/python-dev/2013-May/126148.html Thoughts and suggestions for how to merge scandir with pathlib's approach? It's important to me that pathlib's API doesn't cut itself off from a more efficient implement of the ideas from issue 11406 and scandir... Thanks, Ben.

5 12