Mailman 3 December 2010 - Python-Dev

Issue #10348: concurrent.futures doesn't work on BSD
by Victor Stinner Jan. 2, 2011

Jan. 2, 2011

Hi, FreeBSD 7.2 3.x buildbot is red since some weeks (or months?) because of a concurrent.futures failure. The problem is that test_concurrent_futures uses many (multiprocessing) POSIX semaphores, whereas POSIX semaphores support in FreeBSD is recent and limited. We have to use SysV semaphores (ftok, semget, semop, semctl, ...) instead of POSIX semaphores (sem_open, sem_getvalue, sem_unlink, ...). See: * http://bugs.python.org/issue10348 * "Too many open files" errors on "x86 FreeBSD 7.2 3.x" buildbot ^-- thread in python-dev opened last month I would like to know if it should be considered as a release blocker. Georg Brandl said yes on IRC. Does anyone know SysV API? I tried to write a patch, but I never used semaphores (POSIX or SysV). There is a third party module which looks complete and stable: http://semanchuk.com/philip/sysv_ipc/ It is released under the BSD license. It supports semaphores, but also shared memory and message queues. We don't need all of those, semaphores would be enough. I added its author (Philip Semanchuk) to this thread. I don't know how we should decide to use POSIX or SysV semaphores. It looks like SysV is preferred on FreeBSD and Darwin (and maybe all BSD based OSes), and POSIX is preferred on Linux. Victor

13 44

Summary of Python tracker Issues
by Python tracker Dec. 31, 2010

Dec. 31, 2010

ACTIVITY SUMMARY (2010-12-24 - 2010-12-31) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 2525 (-17) closed 20058 (+49) total 22583 (+32) Open issues with patches: 1063 Issues opened (19) ================== #10764: sysconfig and alternative implementations http://bugs.python.org/issue10764 reopened by michael.foord #10771: descriptor protocol documentation has two different definition http://bugs.python.org/issue10771 opened by Devin Jeanpierre #10772: Several actions for argparse arguments missing from docs http://bugs.python.org/issue10772 opened by ipatrol #10775: assertRaises as a context manager should accept a 'msg' keywor http://bugs.python.org/issue10775 opened by r.david.murray #10782: Not possible to cross-compile due to poor detection of %lld su http://bugs.python.org/issue10782 opened by bgamari #10784: os.getpriority() and os.setpriority() http://bugs.python.org/issue10784 opened by giampaolo.rodola #10785: parser: store the filename as an unicode object http://bugs.python.org/issue10785 opened by haypo #10786: unittest.TextTextRunner does not respect redirected stderr http://bugs.python.org/issue10786 opened by cooyeah #10787: [random.gammavariate] Add the expression of the distribution i http://bugs.python.org/issue10787 opened by David.Kremer #10788: test_logging failure http://bugs.python.org/issue10788 opened by pitrou #10789: Lock.acquire documentation is misleading http://bugs.python.org/issue10789 opened by Jyrki.Pulliainen #10790: Header.append's charset logic is bogus, 'shift_jis' and "euc_j http://bugs.python.org/issue10790 opened by r.david.murray #10791: Wrapping TextIOWrapper around gzip files http://bugs.python.org/issue10791 opened by dabeaz #10794: Infinite recursion while garbage collecting loops indefinitely http://bugs.python.org/issue10794 opened by Mihai.Rusu #10796: readline completion flaw http://bugs.python.org/issue10796 opened by rheise #10798: test_concurrent_futures fails on FreeBSD http://bugs.python.org/issue10798 opened by loewis #10799: Improve webbrowser.open doc (and, someday, behavior?) http://bugs.python.org/issue10799 opened by terry.reedy #10800: libffi build failure on HP-UX 11/PA http://bugs.python.org/issue10800 opened by bugs-python(a)vendor.thewrittenword.com #10801: zipfile.ZipFile().extractall() header mismatch for non-ASCII c http://bugs.python.org/issue10801 opened by M..Z. Most recent 15 issues with no replies (15) ========================================== #10800: libffi build failure on HP-UX 11/PA http://bugs.python.org/issue10800 #10799: Improve webbrowser.open doc (and, someday, behavior?) http://bugs.python.org/issue10799 #10798: test_concurrent_futures fails on FreeBSD http://bugs.python.org/issue10798 #10796: readline completion flaw http://bugs.python.org/issue10796 #10789: Lock.acquire documentation is misleading http://bugs.python.org/issue10789 #10787: [random.gammavariate] Add the expression of the distribution i http://bugs.python.org/issue10787 #10775: assertRaises as a context manager should accept a 'msg' keywor http://bugs.python.org/issue10775 #10772: Several actions for argparse arguments missing from docs http://bugs.python.org/issue10772 #10761: tarfile.extractall fails to overwrite symlinks http://bugs.python.org/issue10761 #10760: tarfile doesn't handle sysfs well http://bugs.python.org/issue10760 #10752: build_ssl.py is relying on unreliable behaviour of os.popen http://bugs.python.org/issue10752 #10751: WSGIREF - REMOTE_USER and REMOTE-USER collision http://bugs.python.org/issue10751 #10747: Include version info in Windows shortcuts http://bugs.python.org/issue10747 #10746: ctypes c_long & c_bool have incorrect PEP-3118 type codes http://bugs.python.org/issue10746 #10745: setup.py install --user option undocumented http://bugs.python.org/issue10745 Most recent 15 issues waiting for review (15) ============================================= #10801: zipfile.ZipFile().extractall() header mismatch for non-ASCII c http://bugs.python.org/issue10801 #10798: test_concurrent_futures fails on FreeBSD http://bugs.python.org/issue10798 #10790: Header.append's charset logic is bogus, 'shift_jis' and "euc_j http://bugs.python.org/issue10790 #10787: [random.gammavariate] Add the expression of the distribution i http://bugs.python.org/issue10787 #10786: unittest.TextTextRunner does not respect redirected stderr http://bugs.python.org/issue10786 #10785: parser: store the filename as an unicode object http://bugs.python.org/issue10785 #10784: os.getpriority() and os.setpriority() http://bugs.python.org/issue10784 #10766: optparse uses %s in gettext calls http://bugs.python.org/issue10766 #10765: Build regression from automation changes on windows http://bugs.python.org/issue10765 #10756: Error in atexit._run_exitfuncs [...] Exception expected for v http://bugs.python.org/issue10756 #10755: Add posix.fdlistdir http://bugs.python.org/issue10755 #10746: ctypes c_long & c_bool have incorrect PEP-3118 type codes http://bugs.python.org/issue10746 #10744: ctypes arrays have incorrect buffer information (PEP-3118) http://bugs.python.org/issue10744 #10740: sqlite3 module should allow DDL statements in transactions http://bugs.python.org/issue10740 #10735: platform.architecture() gives misleading results for OS X mult http://bugs.python.org/issue10735 Top 10 most discussed issues (10) ================================= #6210: Exception Chaining missing method for suppressing context http://bugs.python.org/issue6210 23 msgs #2636: Regexp 2.7 (modifications to current re 2.2.2) http://bugs.python.org/issue2636 22 msgs #10542: Py_UNICODE_NEXT and other macros for surrogates http://bugs.python.org/issue10542 19 msgs #10769: ast: provide more useful range information http://bugs.python.org/issue10769 17 msgs #10791: Wrapping TextIOWrapper around gzip files http://bugs.python.org/issue10791 13 msgs #6011: python doesn't build if prefix contains non-ascii characters http://bugs.python.org/issue6011 5 msgs #10348: multiprocessing: use SysV semaphores on FreeBSD http://bugs.python.org/issue10348 5 msgs #10764: sysconfig and alternative implementations http://bugs.python.org/issue10764 5 msgs #7995: On Mac / BSD sockets returned by accept inherit the parent's F http://bugs.python.org/issue7995 4 msgs #9893: Usefulness of the Misc/Vim/ files? http://bugs.python.org/issue9893 4 msgs Issues closed (45) ================== #2708: IDLE subprocess error http://bugs.python.org/issue2708 closed by terry.reedy #3194: Demo/loop.c passing "char *" instead of "wchar_t *" http://bugs.python.org/issue3194 closed by georg.brandl #4212: email.LazyImporter does not use absolute imports http://bugs.python.org/issue4212 closed by r.david.murray #4685: IDLE will not open (2.6.1 on WinXP pro) http://bugs.python.org/issue4685 closed by terry.reedy #5258: addpackage in site.py fails hard on badly formed .pth files http://bugs.python.org/issue5258 closed by r.david.murray #5423: Exception raised when attempting to call set_charset on an ema http://bugs.python.org/issue5423 closed by r.david.murray #6533: Make test_xmlrpc_net functional in the absence of time.xmlrpc. http://bugs.python.org/issue6533 closed by r.david.murray #7056: regrtest runtest_inner calls findtestdir unnecessarily http://bugs.python.org/issue7056 closed by r.david.murray #7962: Demo and Tools need to be tested and pruned http://bugs.python.org/issue7962 closed by georg.brandl #8009: email.parser.Parser is inefficient with large strings http://bugs.python.org/issue8009 closed by r.david.murray #8889: test_support.transient_internet fails on Freebsd because socke http://bugs.python.org/issue8889 closed by r.david.murray #9153: Run tools and demos as part of the test suite http://bugs.python.org/issue9153 closed by georg.brandl #9824: SimpleCookie should escape commas and semi-colons http://bugs.python.org/issue9824 closed by r.david.murray #9891: Minor doc typo at datamodel.rst: "list" -> "alist" http://bugs.python.org/issue9891 closed by eric.araujo #10254: unicodedata.normalize('NFC', s) regression http://bugs.python.org/issue10254 closed by belopolsky #10494: Demo/comparisons/regextest.py needs some usage information. http://bugs.python.org/issue10494 closed by georg.brandl #10495: Demo/comparisons/sortingtest.py needs some usage information. http://bugs.python.org/issue10495 closed by georg.brandl #10609: dbm documentation example doesn't work (iteritems()) http://bugs.python.org/issue10609 closed by georg.brandl #10626: Bad interaction between test_logging and test_concurrent_futur http://bugs.python.org/issue10626 closed by bquinlan #10679: "make altinstall" may clobber OS provided scripts http://bugs.python.org/issue10679 closed by georg.brandl #10738: webbrowser.py bug with Opera on Linux http://bugs.python.org/issue10738 closed by terry.reedy #10739: Subprocess behavior on Windows http://bugs.python.org/issue10739 closed by georg.brandl #10742: memoryview.readonly attribute is not documented http://bugs.python.org/issue10742 closed by georg.brandl #10753: request_uri method of wsgiref module does not support RFC1808 http://bugs.python.org/issue10753 closed by orsenthil #10759: HTMLParser.unescape() fails on HTML entities with incorrect sy http://bugs.python.org/issue10759 closed by orsenthil #10763: subprocess.communicate() doesn't close pipes on Windows http://bugs.python.org/issue10763 closed by haypo #10767: Lib/test/crashers/README is out of date http://bugs.python.org/issue10767 closed by georg.brandl #10768: Bug in scrolledtext http://bugs.python.org/issue10768 closed by georg.brandl #10770: zipinfo - fix of a typo in the doc http://bugs.python.org/issue10770 closed by eric.araujo #10773: "Building C and C++ Extensions on Windows" documentation shows http://bugs.python.org/issue10773 closed by georg.brandl #10774: test_logging leaves temp files http://bugs.python.org/issue10774 closed by vinay.sajip #10776: os.utime returns an error on NTFS-3G partition http://bugs.python.org/issue10776 closed by Aaron.Masover #10777: xml.etree.register_namespace dictionary changed size during it http://bugs.python.org/issue10777 closed by georg.brandl #10778: decoding_fgets() (tokenizer.c) decodes the filename from the w http://bugs.python.org/issue10778 closed by haypo #10779: Change filename encoding to FS encoding in PyErr_WarnExplicit( http://bugs.python.org/issue10779 closed by haypo #10780: Fix filename encoding in PyErr_SetFromWindowsErrWithFilename() http://bugs.python.org/issue10780 closed by haypo #10781: minidom Node.writexml method doesn't manage encoding parameter http://bugs.python.org/issue10781 closed by georg.brandl #10783: struct.pack() and Unicode strings http://bugs.python.org/issue10783 closed by r.david.murray #10792: Compile() and 'Windows/Mac newlines' http://bugs.python.org/issue10792 closed by terry.reedy #10793: hashlib.hash.digest() documentation incorrect re return type http://bugs.python.org/issue10793 closed by orsenthil #10795: standard library do not use ssl as recommended http://bugs.python.org/issue10795 closed by pitrou #10797: Wrong detection of lines in readlines() function http://bugs.python.org/issue10797 closed by eric.smith #1379416: email.Header encode() unicode P2.6 http://bugs.python.org/issue1379416 closed by r.david.murray #1515839: socket timeout inheritance on accept http://bugs.python.org/issue1515839 closed by ned.deily #1685453: email package should work better with unicode http://bugs.python.org/issue1685453 closed by r.david.murray

1 0

Demo is gone
by Georg Brandl Dec. 30, 2010

Dec. 30, 2010

According to the consensus (and loosely following the Google spreadsheet I created for that purpose), I have removed the Demo directory from py3k. Most demos have been deleted; some have been moved into Tools or into the docs as an example. If I removed something you think should have stayed, please speak up. I kept a few less-useful ones in a new directory called Tools/demo that aims to give a broad overview of what you can do with a few lines of Python. Please feel free to remove and/or add demos there, if you find better ones. Georg

1 0

Compile() and Windows/Mac newlines
by Terry Reedy Dec. 29, 2010

Dec. 29, 2010

In python-list thread "Does Python 3.1 accept \r\n in compile()?" jmfauth notes that compile('print(999)\r\n', '<in>', 'exec') works in 2.7 but not 3.1 (and 3.2 not checked) because 3.1 sees '\r' as SyntaxError. I started to respond that this is part of Py3 cleanup with newlines converted on input and back-compatibility with ancient Python not needed. Then I saw in 3.2 manual "Changed in version 3.2: Allowed use of Windows and Mac newlines. Also input in 'exec' mode does not have to end in a newline anymore. Added the optimize parameter." I verified second statement ("print(999)" works) (and remember commit for third), but original above gives same error. Should "Allowed use of Windows and Mac newlines." be deleted? What else could it mean other than use of '\r' or '\r\n'? -- Terry Jan Reedy

2 3

Re: [Python-Dev] [Python-checkins] r87537 - in python/branches/py3k: Doc/library/struct.rst Doc/whatsnew/3.2.rst Lib/test/test_struct.py Lib/wave.py Misc/NEWS Modules/_struct.c
by Terry Reedy Dec. 28, 2010

Dec. 28, 2010

On 12/28/2010 8:26 AM, victor.stinner wrote: > Author: victor.stinner > Date: Tue Dec 28 14:26:42 2010 > New Revision: 87537 > > Log: > Issue #10783: struct.pack() doesn't encode implicitly unicode to UTF-8 > Modified: python/branches/py3k/Doc/whatsnew/3.2.rst > ============================================================================== > --- python/branches/py3k/Doc/whatsnew/3.2.rst (original) > +++ python/branches/py3k/Doc/whatsnew/3.2.rst Tue Dec 28 14:26:42 2010 > @@ -1705,3 +1705,7 @@ > > (Contributed by Georg Brandl and Mattias Brändström; > `appspot issue 53094<http://codereview.appspot.com/53094>`_.) > + > +* :func:`struct.pack` doesn't encode implicitly unicode to UTF-8 anymore: That phrasing is awkward at best. I believe you mean "struct.pack no longer implicitly encodes unicode to UTF-8", which will be clearer to most people. Terry

4 3

Re: [Python-Dev] [Python-checkins] r87523 - python/branches/py3k/Doc/tutorial/interpreter.rst
by Jim Jewett Dec. 28, 2010

Dec. 28, 2010

It might still be worth saying something like: Note that this "python <file" does something subtly different; the details are not included in this tutorial. On Tue, Dec 28, 2010 at 4:18 AM, georg.brandl <python-checkins(a)python.org> wrote: > Author: georg.brandl > Date: Tue Dec 28 10:18:24 2010 > New Revision: 87523 > > Log: > Remove confusing paragraph -- this is relevant only to advanced users anyway and does not belong into the tutorial. > > Modified: > python/branches/py3k/Doc/tutorial/interpreter.rst > > Modified: python/branches/py3k/Doc/tutorial/interpreter.rst > ============================================================================== > --- python/branches/py3k/Doc/tutorial/interpreter.rst (original) > +++ python/branches/py3k/Doc/tutorial/interpreter.rst Tue Dec 28 10:18:24 2010 > @@ -58,14 +58,6 @@ > ``python -m module [arg] ...``, which executes the source file for *module* as > if you had spelled out its full name on the command line. > > -Note that there is a difference between ``python file`` and ``python > -<file``. In the latter case, input requests from the program, such as calling > -``sys.stdin.read()``, are satisfied from *file*. Since this file has already > -been read until the end by the parser before the program starts executing, the > -program will encounter end-of-file immediately. In the former case (which is > -usually what you want) they are satisfied from whatever file or device is > -connected to standard input of the Python interpreter. > - > When a script file is used, it is sometimes useful to be able to run the script > and enter interactive mode afterwards. This can be done by passing :option:`-i` > before the script. (This does not work if the script is read from standard > _______________________________________________ > Python-checkins mailing list > Python-checkins(a)python.org > http://mail.python.org/mailman/listinfo/python-checkins >

2 1

Re: [Python-Dev] [Python-checkins] r87505 - in python/branches/py3k: Doc/c-api/unicode.rst Include/unicodeobject.h
by R. David Murray Dec. 28, 2010

Dec. 28, 2010

On Mon, 27 Dec 2010 02:49:29 +0100, victor.stinner <python-checkins(a)python.org> wrote: > Author: victor.stinner > Date: Mon Dec 27 02:49:29 2010 > New Revision: 87505 > > Log: > Issue #9738: document encodings of unicode functions > > Modified: > python/branches/py3k/Doc/c-api/unicode.rst > python/branches/py3k/Include/unicodeobject.h > > Modified: python/branches/py3k/Doc/c-api/unicode.rst > ============================================================================== > --- python/branches/py3k/Doc/c-api/unicode.rst (original) > +++ python/branches/py3k/Doc/c-api/unicode.rst Mon Dec 27 02:49:29 2010 > @@ -1063,7 +1063,8 @@ > .. c:function:: int PyUnicode_CompareWithASCIIString(PyObject *uni, char *string) > > Compare a unicode object, *uni*, with *string* and return -1, 0, 1 for less > - than, equal, and greater than, respectively. > + than, equal, and greater than, respectively. *string* is an ASCII-encoded > + string (it is interpreted as ISO-8859-1). Does it mean anything to say that an ASCII string is interpreted as ISO-8859-1? If it is ASCII-encoded it shouldn't have any bytes with the 8th bit set, leaving no room for interpretation. So presumably you mean it is (treated as) an ISO-8859-1 encoded string, despite the function name? -- R. David Murray www.bitdance.com

2 3

os.getpriority() and os.setpriority()
by Giampaolo Rodolà Dec. 28, 2010

Dec. 28, 2010

Hi all, after having implemented something very similar in psutil: http://code.google.com/p/psutil/issues/detail?id=142 ...I decided to contribute a patch for exposing getpriority() and setpriority() system calls in the os module: http://bugs.python.org/issue10784 This was also raised some time ago: http://mail.python.org/pipermail/python-ideas/2009-April/004028.html ...and it seems feedbacks were positive. I think it would be nice to have Windows equivalents (GetPriorityClass / SetPriorityClass) as well but I'm not sure how since their notation is different. Regards, --- Giampaolo http://code.google.com/p/pyftpdlib http://code.google.com/p/psutil

2 1

Loggers in the stdlib and logging configuration APIs
by Vinay Sajip Dec. 28, 2010

Dec. 28, 2010

The logging configuration calls fileConfig and dictConfig disable all existing loggers, and enable only loggers explicitly named in the configuration (and their children). Although there is a disable_existing_loggers option for each configuration API, which can be used to prevent disabling of existing loggers, the default value of this option is True (for backward compatibility) and so existing ("old") loggers are disabled by default. This can lead to unexpected behaviour when loggers are used by the standard library. A recent issue, http://bugs.python.org/issue10626 relates to test_concurrent_futures failing, but only when exercised twice with a test_logging call in between, as in regrtest.py test_concurrent_futures test_logging test_concurrent_futures I've fixed test_logging so that it remembers the disabled state of existing loggers and restores it, and now the above test works fine. However, since recent changes in logging, we are expecting to use it to e.g. display messages on stderr when exceptions can't be raised in stdlib code. Thus, I think I may need to change the way disabling of loggers works, so I wanted to get some feedback from python-dev both about possible approaches and when to apply them (With 3.2 so close to release I'm not proposing to do anything precipitate, just raising the issue for discussion). Essentially, some or all of the loggers in the stdlib itself should perhaps be immune to the "disable existing loggers" logic. Otherwise, a fileConfig() or dictConfig() call with default semantics will prevent a stdlib message (logged when exceptions can't be raised) from being displayed, unless a user explicitly names those stdlib loggers in their configuration. This is not practical because users would need to update their configurations whenever a new logger appeared in the stdlib. Some possible approaches are: 1. Change the default configuration behaviour so that by default, existing loggers are NOT disabled. This is of course strongly backwards-incompatible, but would it make sense to consider it for 3.3? Perhaps we should add a warning? 2. Change the disabling logic so that it never disables stdlib loggers. This raises the question of how to identify them, and there are several loggers in the stdlib in 3.2: "py.warnings" (used by the logging package when warnings are redirected to logging), "concurrent.futures", "http.cookiejar", "multiprocessing" However, it would be good to have some consistency of naming stdlib loggers - perhaps using __name__ as is recommended practice for library and application developers, but with a prefix such as "py." to indicate that it's a part of the stdlib. This would make the disabling logic change easier as it could just check the "py." prefix rather than have an internal list of logger names, which would need more maintenance and be more error-prone. There are quite possibly other approaches I haven't thought of. I'd be grateful for your suggestions, as I'd like to minimise the impact on users, while making it easy to add more logging to the stdlib. Regards, Vinay Sajip

3 10

Re: [Python-Dev] [Python-checkins] r87504 - in python/branches/py3k: Doc/c-api/exceptions.rst Include/pyerrors.h Include/warnings.h
by R. David Murray Dec. 28, 2010

Dec. 28, 2010

On Mon, 27 Dec 2010 02:49:27 +0100, victor.stinner <python-checkins(a)python.org> wrote: > Author: victor.stinner > Date: Mon Dec 27 02:49:26 2010 > New Revision: 87504 > > Log: > Issue #9738: Document encodings of error and warning functions > > Modified: > python/branches/py3k/Doc/c-api/exceptions.rst > python/branches/py3k/Include/pyerrors.h > python/branches/py3k/Include/warnings.h > > Modified: python/branches/py3k/Doc/c-api/exceptions.rst > ============================================================================== > --- python/branches/py3k/Doc/c-api/exceptions.rst (original) > +++ python/branches/py3k/Doc/c-api/exceptions.rst Mon Dec 27 02:49:26 2010 > @@ -148,7 +148,8 @@ > This function sets the error indicator and returns *NULL*. *exception* > should be a Python exception class. The *format* and subsequent > parameters help format the error message; they have the same meaning and > - values as in :c:func:`PyUnicode_FromFormat`. > + values as in :c:func:`PyUnicode_FromFormat`. *format* is an ASCII-encoding > + string. ASCII-encoded string, or ASCII encoding-string? > .. c:function:: int PyErr_WarnFormat(PyObject *category, Py_ssize_t stack_level, const char *format, ...) > > Function similar to :c:func:`PyErr_WarnEx`, but use > - :c:func:`PyUnicode_FromFormat` to format the warning message. > + :c:func:`PyUnicode_FromFormat` to format the warning message. *format* is > + an ASCII-encoded string. So, the former, I'd guess :) -- R. David Murray www.bitdance.com

4 5