Mailman 3 December 2007 - Python-Dev

Extend reST spec to allow automatic recognition of identifiers in comments?
by Jameson "Chema" Quinn 06 Jan '08

06 Jan '08

This is a VERY VERY rough draft of a PEP. The idea is that there should be some formal way that reST parsers can differentiate (in docstrings) between variable/function names and identical English words, within comments. PEP: XXX Title: Catching unmarked identifiers in docstrings Version: 0.0.0.0.1 Last-Modified: 23-Aug-2007 Author: Jameson Quinn <firstname dot lastname at gmail> Status: Draft Type: Informational Content-Type: text/x-rst Created: 23-Aug-2007 Post-History: 30-Aug-2002 Abstract ======== This PEP makes explicit some additional ways to parse docstrings and comments for python identifiers. These are intended to be implementable on their own or as extensions to reST, and to make as many existing docstrings as possible usable by tools that change the visible representation of identifiers, such as translating (non-english) code editors or visual programming environments. Docstrings in widely-used modules are encouraged to use \`explicit backquotes\` to mark identifiers which are not caught by these cases. THIS IS AN EARLY DRAFT OF THIS PEP FOR DISCUSSION PURPOSES ONLY. ALL LOGIC IS INTENTIONALLY DEFINED ONLY BY EXAMPLES AND THERE IS NO REFERENCE IMPLEMENTATION UNTIL A THERE ARE AT LEAST GLIMMERINGS OF CONSENSUS ON THE RULE SET. Rationale ========= Python, like most computer languages, is based on English. This can represent a hurdle to those who do not speak English. Work is underway on Bityi_, a code viewer/editor which translates code to another language on load and save. Among the many design issues in Bityi is that of identifiers in docstrings. A view which translates the identifiers in code, but leaves the untranslated identifier in the docstrings, makes the docstrings worse than useless, even if the programmer has a rudimentary grasp of English. Yet if all identifiers in docstrings are translated, there is the problem of overtranslation in either direction. It is necessary to distinguish between the variable named "variable", which should be translated, and the comment that something is "highly variable", which should not. .. _Bityi: http://wiki.laptop.org/go/Bityi Note that this is just one use-case; syntax coloring and docstring hyperlinks are another one. This PEP is not the place for a discussion of all the pros and cons of a translating viewer. PEP 287 standardizes reST as an optional way to markup docstrings. This includes the possibility of using \`backquotes\` to flag Python identifiers. However, as this PEP is purely optional, there are many cases of identifiers in docstrings which are not flagged as such. Moreover, many of these unflagged cases could be caught programatically. This would reduce the task of making a module internationally-viewable, or hyperlinkable, considerably. This syntax is kept relatively open to allow for reuse with other programming languages. Common cases of identifiers in docstrings ========================================= The most common case is that of lists of argument or method names. We call these "identifier lists":: def register(func, *targs, **kargs): """register a function to be executed someday func - function to be called targs - optional arguments to pass kargs - optional keyword arguments to pass """ #func, targs, and kargs would be recognized as identifiers in the above. class MyClass(object): """Just a silly demonstration, with some methods: thisword : is a class method and you can call it - it may even return a value. As with reST, the associated text can have several paragraphs. BUT - you can't nest this construct, so BUT isn't counted. anothermethod: is another method. eventhis -- is counted as a method. anynumber --- of dashes are allowed in this syntax But consider: two words are NOT counted as an identifier. things(that,look,like,functions): are functions (see below) Also, the docstring may have explanatory text, below or by itself: so we have to deal with that. Thus, any paragraph which is NOT preceded by an empty line or another identifier list - like "itself" above - does not count as an identifier. """ #thisword, anothermethod, eventhis, anynumber, and things would be #recognized as identifiers in the above. Another case is things which look like functions, lists, indexes, or dicts:: """ afunction(is,a,word,with,parentheses) [a,list,is,a,bunch,of,words,in,brackets] anindex[is, like, a, cross, between, the, above] {adict:is,just:words,in:curly, brackets: likethis} """ #all of the above would be recogniszed as identifiers. The "syntax" of what goes inside these is very loose. identifier_list ::= [<initial_word>]<opening_symbol> <content_word> {<separator_symbol> <content_word>} <closing symbol> , with no whitespace after initial_word, and where separator_symbol is the set of symbols ".,<>{}[]+-*^%=|/()[]{}" MINUS closing_symbol. content_word could maybe be a quoted string, too. In the "function name", no whitespace is allowed, but the symbols ".,*^=><-" are. Thus:: """ this.long=>function.*name(counts, and: so |do| these {so-called] arguments) {but,you - cant|use[nested]brackets{so,these,are.identifiers }but,these,arent} {heres.an.example.of."a string, no identifiers in here",but.out.here.yes } { even.one.pair.of.words.with.no symbols.means.nothing.here.is.an.identifier} Any of these structures that open on one line {but.close.on. the.next} are NOT counted as identifiers. """ #in the above: lines 1,2,and the parts of 3 outside the quotes #would be recognized as identifiers The above flexibility is intended to cover the various possibilities for argument lists in a fair subset of other languages. Languages which use only whitespace for argument separation are not covered by these rules. The final case is words that are in some_kind of mixedCase. These are only optionally counted as identifiers if they are also present as an identifier OUTSIDE the comments somewhere in the same file. Doctest and preformatted reST sections should be considered as 100% python code and treated as identifiers (or keywords). Recommended use =============== The rules above are designed to catch the large majority of identifiers already present in docstrings, while applying only extremely rarely to words that should properly be considered as natural language. However, they are inevitably imperfect. All docstrings of modules intended for wide use should manually fix all cases in which these rules fail. If the rules underapply, you can use either \`back quotes\` or parentheses() to mark words as identifiers; if they overapply and reformatting tricks don't fix the problem, <SOME DIRECTIVE TO TURN OFF ALL THIS LOGIC FOR A STRING> Optional use inside comments or non-docstring strings ===================================================== Comments -------- Comments or blocks of comments alone on consecutive lines should be able, optionally, to use these same tricks to spotlight identifiers. Other strings ------------- I'm not sure yet what the rules should be here. One option I'm considering is to be able to turn on all the above logic with some evil hack such as '' 'a string like this, concatenated with an empty string'. Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

2 2

Tkinter problems with Tcl/Tk 8.5
by George Peter Staplin 02 Jan '08

02 Jan '08

Hello, I am a Tcl/Tk core developer. I'm trying to resolve some bugs that have surfaced in Tkinter with Tcl/Tk 8.5. 8.5.0 is very near release, and I'm hoping we can determine where the problem is, and resolve it soon. http://sourceforge.net/tracker/index.php?func=detail&aid=1851526&group_id=1… It seems very peculiar how the text widget's bbox is returning a Python-like list and therefore breaking the Tcl callback. I haven't thus far been able to determine which python method is causing that, or if it's something related to the hooks you have added. The same problem doesn't occur with 8.4. It also has been suggested by some on the Tcl core team (during discussions about this bug) that you probably shouldn't be using Tcl_GetObjType and relying on the registered Tcl_ObjTypes, however I'm not sure of a way to get what you need otherwise. Tcl 8.5 now supports big integers, so I think some changes may be needed in _tkinter.c for that. The list internal rep has changed as well, and I'm not sure if that will affect you. Another developer also pointed out that the text widget is returning a Tcl_Obj with a list type, rather than a string for the bbox subcommand, so that could be related to this bug. I hope that we can work together to resolve the issues you may have with Tk. George -- http://www.xmission.com/~georgeps/ http://whim.linuxsys.net http://code.google.com/p/megapkg/ http://code.google.com/p/opennexx/

2 2

Fate of Windows build directories
by Christian Heimes 01 Jan '08

01 Jan '08

Good afternoon fellow developers! We have finished the support for VS 2008 more or less successfully. Python 3.0a2 was build with VS 2008 and shipped out without major issues. The Tcl/Tk 8.5 problem was solved by Amaury and Martin is working on the integration of the C runtime library. What should happen to the other build directories? PC/VS6/ build dir for Visual Studio 6. It's no longer maintained and probably doesn't work at all. I like to remove it. PC/bdist_wininst/ The README contains only XXX comments and the project files must be ported to VS 2005 and VS 2008. I guess the directory contains the executable for distutils' bdist_wininst. PC/example_nt/ Example project and C files for VS 2003. Needs an update or remove it. PC/os2*/ OS2 build dirs. The files are no longer maintained. PCbuild/ Build dir for VS 2003. Keep it or remove it? I'm +1 for removal from 3.0 and +0 for remove from 2.6. PCBuild8/ Build dir for VS 2005. This directory is a nested directory structure and it doesn't integrate all pre-build steps. The user weck has developed a small Python script to back port the PCbuild9 directory to PCbuild8 (http://bugs.python.org/issue1455). The script would make maintenance of the PCbuild8 directory easier. PCbuild9/ The new build dir for VS 2008. Christian

3 2

PyPy Leysin Winter Sprint (12-19th January 2008)
by Armin Rigo 01 Jan '08

01 Jan '08

Hi all, The next PyPy sprint will be held in Leysin, Switzerland, for the fifth time. The overall idea of the sprint is to continue working on making PyPy ready for general use. The proposed topics are: ctypes, JIT, testing, LLVM. This is a fully public sprint, so newcomers and other topics are welcome. And like previous winters, the main side goal is to have fun in winter sports :-) See the sprint announcement for details: http://codespeak.net/pypy/extradoc/sprintinfo/leysin-winter-2008/announceme… Armin

1 0

My pending patches
by Christian Heimes 31 Dec '07

31 Dec '07

Good evening! I've three patches in the bug tracker which are ready but I like to get approval before I submit them. http://bugs.python.org/issue1657 Wrapper for epoll and kqueue sys calls for Python 2.6 and 3.0. I've worked together with therve from the Twisted team. http://bugs.python.org/issue1640 Enhancements for the math module. The latest patch just adds isnan(), isinf() and sign() to the math module. http://bugs.python.org/issue1567 The patch adds a new API function _PyImport_ImportModuleNoLock(char *name). It first tries to get the module from sys.modules and falls back to PyImport_ImportModule unless the import lock is hold. In the latter case it raises an exception to prevent a dead lock. It solves a problem with imports from C code and threads. Christian

1 0

Summary of Tracker Issues
by Tracker 29 Dec '07

29 Dec '07

ACTIVITY SUMMARY (12/21/07 - 12/28/07) Tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 1381 open (+20) / 11808 closed ( +4) / 13189 total (+24) Open issues with patches: 424 Average duration of open issues: 688 days. Median duration of open issues: 948 days. Open Issues Breakdown open 1373 (+20) pending 8 ( +0) Issues Created Or Reopened (24) _______________________________ Thread local storage and PyGILState_* mucked up by os.fork() 12/21/07 http://bugs.python.org/issue1683 created roudkerk CGIHTTPServer does not chdir prior to executing the CGI script 12/21/07 http://bugs.python.org/issue1684 created majid linecache .updatecache fails on utf8 encoded files 12/22/07 http://bugs.python.org/issue1685 created pyscripter string.Template.safe_substitute fail when overriding pattern at 12/22/07 http://bugs.python.org/issue1686 created hagaigold plistlib.py restricts <integer> to Python int when writing 12/22/07 http://bugs.python.org/issue1687 created lafcadio Incorrectly displayed non ascii characters in prompt using "inpu 12/22/07 http://bugs.python.org/issue1688 created vbr Backport PEP 3141 to 2.6 12/23/07 http://bugs.python.org/issue1689 created jyasskin Crash on cancellation of windows install 12/23/07 http://bugs.python.org/issue1690 created mtvernon feature request: methods that modify instances should return ins 12/23/07 CLOSED http://bugs.python.org/issue1691 created eukaryot Syntax Error exception dosen't print string; not informative 12/23/07 http://bugs.python.org/issue1692 created Dude-X Please check merge from trunk 12/24/07 http://bugs.python.org/issue1693 created tiran py3k floating point number round failures under Linux 12/24/07 http://bugs.python.org/issue1694 created falk_steinhauer time.localtime() docstring has a wrong fieldname 12/24/07 CLOSED http://bugs.python.org/issue1695 created tapybugs Distutils ignore dry-run flag at clean-up of distutils.command.b 12/25/07 http://bugs.python.org/issue1696 created ludvig.ericson Problems with float and "6" type 12/25/07 CLOSED http://bugs.python.org/issue1697 created pedrocpneto urlparse and usernames containing @ 12/25/07 http://bugs.python.org/issue1698 created ocroquette unconditional definiton of _BSD_SOURCE breaks builds with g++-4. 12/25/07 http://bugs.python.org/issue1699 created doko Regular Expression inline flags not handled correctly for some u 12/25/07 http://bugs.python.org/issue1700 created sonnq svn checkout does not work 12/26/07 CLOSED http://bugs.python.org/issue1701 created hjmjohnson Word "alias" used in confusing way to compare open() and file() 12/27/07 http://bugs.python.org/issue1702 created Devin Jeanpierre getpass broken in Py3k: must flush() 12/28/07 http://bugs.python.org/issue1703 created pjenvey possible bug in randint 12/28/07 http://bugs.python.org/issue1704 created cephalo trace module does not annotate global statement 12/28/07 http://bugs.python.org/issue1705 created calvin Force WINVER 0x0500 (Windows 2000) 12/28/07 http://bugs.python.org/issue1706 created tiran Issues Now Closed (9) _____________________ IDLE - configDialog - new layout for key config 40 days http://bugs.python.org/issue1457 kbk patch Patch for TCL 8.5 support 15 days http://bugs.python.org/issue1607 tiran py3k, patch IDLE: help() displays output on the wrong shell 10 days http://bugs.python.org/issue1650 kbk pythonw.exe crashes when run in one particular account on Window 4 days http://bugs.python.org/issue1674 kbk feature request: methods that modify instances should return ins 0 days http://bugs.python.org/issue1691 tiran time.localtime() docstring has a wrong fieldname 0 days http://bugs.python.org/issue1695 brett.cannon Problems with float and "6" type 0 days http://bugs.python.org/issue1697 tiran svn checkout does not work 0 days http://bugs.python.org/issue1701 loewis IDLE invokes completion even when running code 461 days http://bugs.python.org/issue1563981 kbk Top Issues Most Discussed (4) _____________________________ 6 urlparse and usernames containing @ 3 days open http://bugs.python.org/issue1698 5 [patch] epoll and kqueue wrappers for the select module 9 days open http://bugs.python.org/issue1657 4 CGIHTTPServer does not chdir prior to executing the CGI script 7 days open http://bugs.python.org/issue1684 3 Patch for TCL 8.5 support 15 days closed http://bugs.python.org/issue1607

1 0

PATCH: Fast globals/builtins lookups for 2.6
by Neil Toronto 28 Dec '07

28 Dec '07

I've posted a patch here: http://bugs.python.org/issue1518 for 2.6a0 that speeds up LOAD_GLOBAL to where it's just barely slower than LOAD_FAST for both globals and builtins. It started life here, in Python-ideas: http://mail.python.org/pipermail/python-ideas/2007-November/001212.html The idea is to cache pointers to dictionary entries rather than dictionary values as is usually suggested for speeding up globals. In my original approach, every dictionary would keep a list of "observers" that it would notify when an entry pointer became invalid. However, the surgery on PyDictObject became too invasive, to the point where the new code was affecting performance of unrelated code paths. (It was probably caching issues.) It also made some (very rare) operations on builtins and globals very, very slow. The new approach tags each PyDictObject with a "version": a 64-bit value that is incremented for every operation that invalidates at least one entry pointer. (Those are inserting set, delete, pop, clear and resize. Non-inserting set and get are unaffected.) In this approach, changes to PyDictObject are uninvasive and do not affect performance in any measurable way (as far as I can tell). Every function has a PyFastGlobalsObject, which keeps entry pointers for everything in co_names and tracks its dicts' versions so it can update the pointers when one might have become invalid. LOAD_GLOBALS uses the PyFastGlobalsObject to get globals and builtins by name index rather than by string key. With the patch, Python behaves differently in these two ways (as far as I can tell): 1. As before, a __builtins__ ({'None': None}) is created for frames whose globals do not have one. Unlike before, it's added to the globals dict rather than just kept internally. This made implementation easier - I don't think it's a big deal (but I might be wrong). 2. A change of __builtins__ (its value, not its contents) always appears at the beginning of a stack frame. (Before, changing __builtins__ in a module would be undetectable if function calls were kept within that module.) This is likely of little importance. I'd love to see it included. I have donned my asbestos underwear and my best chain mail shirt and I'm begging for comments. Neil

5 8

Spurious Buildbot Reports
by Raymond Hettinger 27 Dec '07

27 Dec '07

The bots are kicking-off so many false alarms that it is becoming difficult to tell whether a check-in genuinely broke a build. At the root of the problem is a number of tests in the test suite that randomly blow-up. I now tend to automatically dismiss failures in test_logging and test_threading for example. Raymond

5 5

viewvc at svn.python.org down?
by O.R.Senthil Kumaran 27 Dec '07

27 Dec '07

The viewvc interface of svn.python.org seems down. svn is working properly though. Anyone aware of the problem? -- O.R.Senthil Kumaran http://uthcode.sarovar.org

2 1

Type hierarchy for numbers for 2.6
by Facundo Batista 24 Dec '07

24 Dec '07

Hi! The issue 1689 [1] proposes a patch to backport the PEP 3141 [2], A Type Hierarchy for Numbers, to the trunk. The patch applies cleanly, and all tests pass ok (of course, even the new ones that are specific to this). The patch misses documentation and lines for the NEWS file, I'll ask for them there. But I wanted to tell you that this is going on, just in case you want to take a look to it. Regards, [1] http://bugs.python.org/issue1689 [2] http://www.python.org/dev/peps/pep-3141/ -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/

1 0