Mailman 3 September 2005 - Python-Dev

Skiping searching throw dictionaries of mro() members.
by Sokolov Yura Sept. 14, 2005

Sept. 14, 2005

Excuse my english. I have a complex Idea. It can be not worth, but just look at. It touches the classes and inheritance. Base point: We change classes not too often. But every look at a member leads to search throw a __dict__ of all bases in ierarhy till member found. We could cache this search in class.__dict__ itself and check only bases versions. Lets see: we have some classes: __metaclass__=object class a: ....def meth_a(self): ........pass class b: ....def meth_b(self): ........pass class c(a,b): ....pass Every class has mro. >>>c.mro() [<class '__main__.c'>, <class '__main__.a'>, <class '__main__.b'>, <type 'object'>] if we call methon from a base class, we look at c.__dict__ and at dict of every inherited class till found the method itself. so c.meth_a() leads to search throw c.__dict__ and a.__dict__ c.meth_b() leads to search throw c.__dict__ , a.__dict__ and b.__dict__ We could cash looks by this way: Store with a class its version. Every time after creation when we change a class (add,remove or chage class member, including __base__, __bases__ and mro) , we increase the version number. Lets call it VERSION Also store a tuple MRO_VERSION=tuple(base.VERSION for base in self.mro()). It changes, when we touches __base__, __bases__ or self.mro) Store with every class member a tuple : MEMBER_VERSION=(<self.mro version>,<number in mro>) When we add member to (or change member of) class __dict__ directly, we store in this tuple (MRO_VERSION,class.mro().index(class)). When we search a class member: a) and have not found it in class.__dict__, we search it throw mro() by a classic way. If we've found it (in a class _BASE_), we store it in class.__dict__ with a tuple MEMBER_VERSION=(MRO_VERSION, class.mro().index(_BASE_)). Also we check all of seen base.VERSION to match their cached in MRO_VERSION values. If some is not match, we should update MRO_VERSION (berfore storing founded member in a class.__dict__) and adjust VERSION. If it was not found, we could store a (MRO_VERSION, class.mro().index(class)) with an internal value NotFound or do not store anything, and report that we have not found anything (so we search in an object itself). b) When we've found a class member in class.__dict__ we check versions if MEMBER_VERSION[0]==class.MRO_VERSION (the same object) then we just compare MRO_VERSION[i]==class.mro()[i].VERSION for i in range(MEMBER_VERSION[1]+1) and not search throw mro()[i].__dict__ what we are doing now. If all versions concur, we return found member (or, if we've found previously stored NotFound, we report that we have not found anything) if version of MRO_VERSION[j]!=class.mro()[j].VERSION and class!=class.mro()[j] (here can be a bug, one must think about it more) then we revert to a) point, but search throw [base.__dict__ for base in class.mro()[j:]], (and, of cause, update MRO_VERSION) if MEMBER_VERSION[0]!=class.MRO_VERSION, we reverts to a) point. That's all. PS. We can subclass module from a __builtin__, so we leave one dict lookup.

2 2

Python code.interact() and UTF-8 locale
by Victor STINNER Sept. 14, 2005

Sept. 14, 2005

Hi, I found a bug in Python interactive command line (program python alone: looks to be code.interact() function in code.py). With UTF-8 locale, the command << u"é" >> returns << u'\xc3\xa9' >> and not << u'\xE9' >>. Remember: the french e with acute is Unicode 233 (0xE9), encoded \xC3 \xA9 in UTF-8. Another example of the bug: #-*- coding: UTF-8 -*- code = "u\"%s\"" % "\xc3\xa9" compiled = compile(code,'<string>',"single") exec compiled Result : u'\xc3\xa9' Excepted result : u'\xe9' After long hours of debuging (read Python documentation, debug Python with gdb, read Python C source code, ...) I found the origin of the bug: function parsestr() in Python/compile.c. This function translate a string to a unicode string (or a classic string). The problem is when the encoding declaration doesn't exist: the string isn't converted. Solution to the first code: #-*- coding: ascii -*- code = """#-*- coding: UTF-8 -*- u\"%s\"""" % "\xc3\xa9" compiled = compile(code,'<string>',"single") exec compiled Proposition: u"..." and unicode("...") should use sys.stdin.encoding by default. They will work as unicode("...", sys.stdin.encoding). Or easier, the compiler should use sys.stdin.encoding and not ascii as default encoding. Sorry if someone already reported this bug. And, is it a bug or a feature ? ;-) Bye, Haypo (who just have subscribed to the mailing list)

3 4

Re: [Python-Dev] Simplify the file-like-object interface
by Michael Chermside Sept. 14, 2005

Sept. 14, 2005

Andrew Durdin writes: > Another area where I think this approach can help is with the > text/binary file distinction. file() could always open files as > binary, and there could be a convenience function textfile(name, mode) > which would simply return textstream(file(name, mode)). This would > remove the need for "a" or "b" in the mode parameter, and make it > easier to keep text- and binary-file access separate in one's mind: I think you are suffering from the (rather common) misconception that all files are binary, and the definition of "text file" is a binary file which should be interpreted as containing characters in some encoding. In unix, the above is true. One of the fundamental decisions in Unix was to treat all files (and lots of other vaguely file-like things) as undiferentiated streams of bytes. But this is NOT true on many other operating systems. It is not, for example, true on Windows. Many operating systems make a distinction between two basic types of files... "text files" which are line-oriented and contain "text", and "binary files" which contain streams of bytes. This distinction is supported by the basic file operations in the C library. To open a text file in binary mode is technically an error (although in many OSs you'll get away with it). -- Michael Chermside

3 2

IDLE development
by Arthur Sept. 13, 2005

Sept. 13, 2005

Moam writes - >Hello, >More than a year and a half ago, I posted a big patch to IDLE which >adds support for completion and much better calltips, along with some >other improvements. I had also tried to have a little input to the IDLE development process. Suggesting on the idle-dev list it seemed to me that a trivial patch to the existing code would provide functionality to allow customization via user- defined domain-specific syntax highlighting files in a user's home directory. So that if one wished one could add appropriate syntax highlighting for say, Numeric. I left open the possibility that I was mistaken in my opinion that it was trivial or necessary or desirable. I was unable to assess whether the lack of my ability to get a yes, no or sideways was a result of the fact my suggestion and analysis was absurd, or something else. One of the "something else" possibilities was a broken process. Still can't assess it. Art

1 0

Re: [Python-Dev] Replacement for print in Python 3.0
by Paul F. Dubois Sept. 13, 2005

Sept. 13, 2005

Remove the print statement....I laughed until my sides hurt. Hello? Try dating girls and talking to normal people, geek boys. We scientists still use these for debugging. We never 'move on' very far from the tutorial. The salient feature about print statements is that they live to be put in and commented out 10 minutes later, without some import being required or other enabling object being around. Easy things should be easy. Hard things should be possible. I don't believe the person who said the trailing comma case mixed up anybody, not for more than 10 seconds anyway. OK, now that I've offended everyone, I'll go back into retirement. But I *am* laughing at you.

37 153

PEP 3000 and iterators
by Lisandro Dalcin Sept. 12, 2005

Sept. 12, 2005

PEP 3000 says (http://www.python.org/peps/pep-3000.html) : Core language - Return iterators instead of lists where appropriate for atomic type methods (e.g. dict.keys(), dict.values(), dict.items(), etc.) Built-in Namespace - Make built-ins return an iterator where appropriate (e.g. range(), zip(), etc.) - Relevant functions should consume iterators (e.g. min(), max()) To be removed: - xrange(): use range() instead [1] Any possibility to add one (or more) __future__ statement to implicitly get this behavior? Any suggestion about naming? -- Lisandro Dalcín --------------- Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC) Instituto de Desarrollo Tecnológico para la Industria Química (INTEC) Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET) PTLC - Güemes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594

8 15

unintentional and unsafe use of realpath()
by Peter Jones Sept. 12, 2005

Sept. 12, 2005

Hi, In Python 2.4.1, Python/sysmodule.c includes a function PySys_SetArgv(). One of the things it does is attempt to resolve symbolic links into absolute paths. Currently, it uses readlink() if configure found that your system supports it, and then it tries to do the same thing again using realpath() if you system supports that. This seems wrong; there's really no reason to do both. So here's a patch to move the realpath() usage into a #else following the HAVE_READLINK test: --- Python-2.4.1/Python/sysmodule.c.readlink 2005-09-10 14:05:26.000000000 -0400 +++ Python-2.4.1/Python/sysmodule.c 2005-09-10 14:06:00.000000000 -0400 @@ -1211,7 +1211,7 @@ } } } -#endif /* HAVE_READLINK */ +#else /* HAVE_READLINK */ #if SEP == '\\' /* Special case for MS filename syntax */ if (argc > 0 && argv0 != NULL) { char *q; @@ -1244,6 +1244,7 @@ #endif p = strrchr(argv0, SEP); } +#endif /* HAVE_READLINK */ if (p != NULL) { #ifndef RISCOS n = p + 1 - argv0; Another problem (which I have not fixed) is that when realpath() is used, in some cases MAXPATHLEN is smaller than the system's PATH_MAX/pathconf(path, _PC_PATH_MAX). When that happens, the realpath() usage is a potential buffer overflow, which can be selectively aggravated using a carefully constructed symbolic link. This is the case currently on Fedora Core (Linux) at least, and possibly other OSes. Recent versions of gcc include a feature to check for this kind of bug at runtime, and if you build python with gcc+glibc and the gcc command line arguments "-Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector" it will give you a failure that looks something like: case $MAKEFLAGS in \ *-s*) LD_LIBRARY_PATH=/home/pjones/build/BUILD/Python-2.4.1: CC='gcc -pthread' LDSHARED='gcc -pthread -shared' OPT='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m32 -march=i386 -mtune=pentium4 -fasynchronous-unwind-tables -D_GNU_SOURCE -fPIC -I/usr/kerberos/include ' ./python -E ./setup.py -q build;; \ *) LD_LIBRARY_PATH=/home/pjones/build/BUILD/Python-2.4.1: CC='gcc -pthread' LDSHARED='gcc -pthread -shared' OPT='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m32 -march=i386 -mtune=pentium4 -fasynchronous-unwind-tables -D_GNU_SOURCE -fPIC -I/usr/kerberos/include ' ./python -E ./setup.py build;; \ esac *** buffer overflow detected ***: ./python terminated ======= Backtrace: ========= /lib/libc.so.6(__chk_fail+0x41)[0x590495] /lib/libc.so.6(__ptsname_r_chk+0x0)[0x590ac0] /home/pjones/build/BUILD/Python-2.4.1/libpython2.4.so.1.0(PySys_SetArgv +0x1a1)[0x228cf5] /home/pjones/build/BUILD/Python-2.4.1/libpython2.4.so.1.0(Py_Main +0x671)[0x22bedd] ./python(main+0x2a)[0x804859a] /lib/libc.so.6(__libc_start_main+0xdf)[0x4c74ff] ./python[0x80484ed] ======= Memory map: ======== 00185000-0026e000 r-xp 00000000 03:03 278531 /home/pjones/build/BUILD/Python-2.4.1/libpython2.4.so.1.0 0026e000-00295000 rwxp 000e8000 03:03 278531 /home/pjones/build/BUILD/Python-2.4.1/libpython2.4.so.1.0 00295000-00298000 rwxp 00295000 00:00 0 00495000-004ae000 r-xp 00000000 03:03 1966150 /lib/ld-2.3.90.so 004ae000-004af000 r-xp 00018000 03:03 1966150 /lib/ld-2.3.90.so 004af000-004b0000 rwxp 00019000 03:03 1966150 /lib/ld-2.3.90.so 004b2000-005d7000 r-xp 00000000 03:03 1966261 /lib/libc-2.3.90.so 005d7000-005d9000 r-xp 00124000 03:03 1966261 /lib/libc-2.3.90.so 005d9000-005db000 rwxp 00126000 03:03 1966261 /lib/libc-2.3.90.so 005db000-005dd000 rwxp 005db000 00:00 0 005df000-00602000 r-xp 00000000 03:03 1966272 /lib/libm-2.3.90.so 00602000-00603000 r-xp 00022000 03:03 1966272 /lib/libm-2.3.90.so 00603000-00604000 rwxp 00023000 03:03 1966272 /lib/libm-2.3.90.so 00606000-00608000 r-xp 00000000 03:03 1966270 /lib/libdl-2.3.90.so 00608000-00609000 r-xp 00001000 03:03 1966270 /lib/libdl-2.3.90.so 00609000-0060a000 rwxp 00002000 03:03 1966270 /lib/libdl-2.3.90.so 006f7000-00705000 r-xp 00000000 03:03 1966283 /lib/libpthread-2.3.90.so 00705000-00706000 r-xp 0000d000 03:03 1966283 /lib/libpthread-2.3.90.so 00706000-00707000 rwxp 0000e000 03:03 1966283 /lib/libpthread-2.3.90.so 00707000-00709000 rwxp 00707000 00:00 0 0073f000-00743000 r-xp 00000000 03:03 1442230 /home/pjones/build/BUILD/Python-2.4.1/Modules/stropmodule.so 00743000-00745000 rwxp 00004000 03:03 1442230 /home/pjones/build/BUILD/Python-2.4.1/Modules/stropmodule.so 00a22000-00a2b000 r-xp 00000000 03:03 1966127 /lib/libgcc_s-4.0.1-20050906.so.1 00a2b000-00a2c000 rwxp 00009000 03:03 1966127 /lib/libgcc_s-4.0.1-20050906.so.1 00df7000-00df9000 r-xp 00000000 03:03 1968751 /lib/libutil-2.3.90.so 00df9000-00dfa000 r-xp 00001000 03:03 1968751 /lib/libutil-2.3.90.so 00dfa000-00dfb000 rwxp 00002000 03:03 1968751 /lib/libutil-2.3.90.so 00e33000-00e34000 r-xp 00e33000 00:00 0 [vdso] 08048000-08049000 r-xp 00000000 03:03 278551 /home/pjones/build/BUILD/Python-2.4.1/python 08049000-0804a000 rw-p 00000000 03:03 278551 /home/pjones/build/BUILD/Python-2.4.1/python 08086000-0810b000 rw-p 08086000 00:00 0 [heap] b7e69000-b7f2c000 rw-p b7e69000 00:00 0 b7f2d000-b7fb1000 rw-p b7f2d000 00:00 0 b7fc4000-b7fc5000 rw-p b7fc4000 00:00 0 bffaf000-bffc5000 rw-p bffaf000 00:00 0 [stack] /bin/sh: line 1: 9510 Aborted (core dumped) LD_LIBRARY_PATH=/home/pjones/build/BUILD/Python-2.4.1: CC='gcc -pthread' LDSHARED='gcc -pthread -shared' OPT='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m32 -march=i386 -mtune=pentium4 -fasynchronous-unwind-tables -D_GNU_SOURCE -fPIC -I/usr/kerberos/include ' ./python -E ./setup.py -q build make: *** [sharedmods] Error 134 -- Peter

2 2

Re: [Python-Dev] PEP 3000 and iterators
by Delaney, Timothy (Tim) Sept. 12, 2005

Sept. 12, 2005

James Y Knight wrote: > Just to be clear, I do not want nor expect this. I wish to be able to > specifically modify code with full knowledge of what has changed in > Py3.0 such that it will work with both Py2.X and Py3.0. If you want these things to work in 2.x and 3.0, just use iter(dict_instance) and list(dict_instance) as appropriate. Tim Delaney

2 1

Tools directory (Was RE: Replacement for print in Python 3.0)
by Tony Meyer Sept. 12, 2005

Sept. 12, 2005

[finding Tools/i18n/pygettext.py] > You're right, I think Tools is probably a bad place for > anything. If it's not part of the stdlib, I'll likely never > find it. Agreed. Maybe with the introduction of -m in Python 2.4, some of the Tools/ scripts could be put in __main__ sections of appropriate modules? So that "python -m gettext" would be equivilant to "python Tools/i18n/pygettext.py"? (However, pyggettext.py is 22KB, which is a big addition to the module; not everything in Tools/Scripts might be used enough for this, or have an appopriate module to be put in either). Are there other ideas about how Tools/ could be improved? Either moving things, or making it more likely that people will look there for scripts? =Tony.Meyer

5 4

Re: [Python-Dev] Replacement for print in Python 3.0
by Michael Chermside Sept. 12, 2005

Sept. 12, 2005

Guido writes: > Is it worth doing this and completely dropping the %-based formats in > Py3k? (Just asking -- it might be if we can get people to get over the > shock of $ becoming first class ;-). In my opinion, YES -- it's worth seriously considering it. A single, well-designed solution for string interpolation (with syntactic support if needed to make it very easy to use) is FAR better than having one good solution and another legacy solution. Just the awkwardness of the trailing s in "%(foo)s" is enough to motivate a search for something better. But this presuposes that there IS a single well-designed solution. PEP 292 templates are an excellent start, but they are not that solution. The largest problem is the lack of a means for formatting numbers. People should think hard about good solutions. He continues: > I proposed ${varname%fmt} earlier but it prevents you to extend the > varname syntax to arbitrary expressions, which I think is an extension > that will get lots of requests. I certainly agree that we should keep open the syntactic possibility to allow arbitrary Python expressions between "${" and "}" in an interpolation string even though it may not be supported today. I favor idea (Barry's?) of using "${<value>:<format>:<more-format>}" where <value> is an identifier (but someday might allow expressions), and <format> and <more-format> behave like the % interpolation modifiers today. I would have suggested it myself, but somehow I failed to realize that slice literals are allowed only within subscripts and thus do not conflict with this use. -- Michael Chermside

9 12