Mailman 3 December 2008 - Python-Dev

Calling the GC less often when there are lots of long-lived objects
by Antoine Pitrou 17 Dec '08

17 Dec '08

Hello, There are recurring complaints about the garbage collector degrading performance when lots of objects are created in a row. In issue #4074, I've proposed a patch which basically implements Martin's suggestion in http://mail.python.org/pipermail/python-dev/2008-June/080579.html to base the decision to do a full collection on the ratio between the number of objects surviving the (n-1) generation collection and the number of long-lived objects. I've also added a condition so that this new behaviour is only triggered when there are more than 10000 long-lived objects -- therefore, cycles will still get collected quickly in lightweight programs. In Gregory's simple test of storing many tuples in a list, the behaviour has indeed changed from exponential to linear. Is anybody opposed to the principle of this proposal? Antoine.

5 7

Trap SIGSEGV and SIGFPE
by Victor Stinner 16 Dec '08

16 Dec '08

Hi, I published a new version of my fault handler: it installs an handler for signals SIGFPE and SIGSEGV. Using it, it's possible to catch them and continue the execution of your Python program. Example: try: call_evil_code() except MemoryError: print "A segfault? Haha, I don't care!" print "continue the execution" (yes, it's possible to continue the execution after a segmentation fault!) Handled errors: - Segmentation fault: * invalid memory read * invalid memory write * stack overflow (stack pointer outside the stack memory) - SIGFPE * division by zero * floating point error? Such errors may occurs from external libraries (written in C)... or Python builtin libraries (eg. imageop). The handler is now only used in Py_EvalFrameEx(), but it could be used anywhere. The patch uses sigsetjmp() in Py_EvalFrameEx() to set a "check point", and siglongjmp() in the signal handler to go back to the check point. It also uses a separated stack for the signal handler, because on stack overflow you can not use the stack (ex: unable to call any function!). With MAXDEPTH=100, the memory footprint is ~20 KB. If you call Py_EvalFrameEx() more than MAXDEPTH times, the handler will go back to the frame #MAXDEPTH on error (you loose the last entries in the Python traceback). sigsetjmp()/siglongjmp() should be available on many OS. I just know that it works perfectly on Linux. sigaltstack() is needed to recover after a stack overflow, but other errors can be catched without it. I didn't run any benchmark yet, but it would be interresting ;-) Changing MAXDEPTH constant may changes the speed with many recursive calls (eg. MAXDEPTH=1 only set a check for the first call to Py_EvalFrameEx()). I would appreciate a review, especially for the patch in Python/ceval.c. -- Victor Stinner aka haypo http://www.haypocalc.com/blog/

16 31

[ANN] EuroPython 2009 – Call for Participation!
by Martin P. Hellwig 16 Dec '08

16 Dec '08

On behalf of the EuroPython 2009 organisation it is my privilege and honour to announce the 'Call for Participation' for EuroPython 2009! EuroPython is the conference for the communities around Python, including the Django, Zope and Plone communities. This years conference will be held in Birmingham, UK from Monday 29th June to Saturday 4th July 2009. Talk & Themes Do you have something you wish to present at EuroPython? Go to http://www.europython.eu/talks/cfp/ for this years themes and submissions criteria, the deadline is on 5th April 2009. Other Talks, Activities and Events Have you got something which does not fit the above? Visit http://www.europython.eu/talks/ . Help Us Out We could use a hand any contribution is welcome, please take a look at http://www.europython.eu/contact/ . Sponsors An unique opportunity to affiliate with the prestigious EuroPython conference! http://www.europython.eu/sponsors/ Spread the Word Improve our publicity by distributing this announcement in your corner of the community, please coordinate this with the organizers: http://www.europython.eu/contact/ General Information For more information about the conference, please visit http://www.europython.eu/ Looking forward to see you! The EuroPython Team

1 0

Reindenting the C code base?
by Antoine Pitrou 16 Dec '08

16 Dec '08

Hello, I remember there were some talks of reindenting the C code base (from tabs to 4-space indents) after py3k is released, but I can't find the discussion thread again. Was a decision ever taken about it? Regards Antoine.

15 25

Re: [Python-Dev] [Python-3000] python-3000 list is closed
by skip＠pobox.com 16 Dec '08

16 Dec '08

Martin> The mailing list python-3000(a)python.org is now closed. All Martin> further discussion of Python 3.x takes place on Martin> python-dev(a)python.org. Maybe set up a simple email alias reflecting python-3000 to python-dev? Skip

4 3

[ANN] Python 2.4.6 and 2.5.3, release candidate 1
by "Martin v. Löwis" 15 Dec '08

15 Dec '08

On behalf of the Python development team and the Python community, I'm happy to announce the release candidates of Python 2.4.6 and 2.5.3. 2.5.3 is the last bug fix release of Python 2.5. Future 2.5.x releases will only include security fixes. According to the release notes, over 100 bugs and patches have been addressed since Python 2.5.1, many of them improving the stability of the interpreter, and improving its portability. 2.4.6 includes only a small number of security fixes. Python 2.6 is the latest version of Python, we're making this release for people who are still running Python 2.4. See the release notes at the website (also available as Misc/NEWS in the source distribution) for details of bugs fixed; most of them prevent interpreter crashes (and now cause proper Python exceptions in cases where the interpreter may have crashed before). Assuming no major problems crop up, a final release of Python 2.4.6 and 2.5.3 will follow in about a week's time. For more information on Python 2.4.6 and 2.5.3, including download links for various platforms, release notes, and known issues, please see: http://www.python.org/2.4.6 http://www.python.org/2.5.3 Highlights of the previous major Python releases are available from the Python 2.5 page, at http://www.python.org/2.4/highlights.html http://www.python.org/2.5/highlights.html Enjoy this release, Martin Martin v. Loewis martin(a)v.loewis.de Python Release Manager (on behalf of the entire python-dev team)

4 6

Re: [Python-Dev] How to force export of a particular symbol from python.exe?
by Jim Murphy 15 Dec '08

15 Dec '08

Martin: You wrote: "That's not the issue. Had pymath.o been linked into python, it's symbols would have been exported (is that proper use of English tenses?)" Yes, it's a proper and idiomatic use of the subjunctive mood, which many native (American) English speakers manage to mangle. I also noticed you wrote the following a few emails later on the python-dev list: "Using that would require to split pymath.c into multiple files." My ear tells me that either "that would require splitting pymath,c ..." or "that would require one to split pymat.c ..." is much more grammatical than "that would require to split ...," but I can't cite a rule. It is frequently acceptable to use either the infinitive or the gerund form of a verb, which would imply that "to split" should be interchangeable with "splitting," but I believe that some verbs have preferences for one form over the other. My ear seems to be thinking of the template "require someone to do something," and rebels at hearing the "to" without a "someone." That's the best excuse for a rule I could come up with. I actually spent a half-hour trying to find rules on the uses in English of infinitives versus gerunds and did not find anything definitive. I realize now, to my disgust, that English usage is very badly afflicted with "special casing." Jim Murphy 326 Sunnyview Lane Ithaca, New York 14850-6258 Tel (home): +1 607-319-4161

4 3

Psyco for -OO or -O
by Lie Ryan 15 Dec '08

15 Dec '08

I'm sure probably most of you knows about psyco[1], the optimizer. Python has an -O and -OO flag that is intended to be optimization flag, but we know that currently it doesn't do much. Why not add psyco as standard library and let -O or -OO invoke psyco? [1] http://psyco.sourceforge.net/index.html

4 4

sys.stdout.write encoding failure
by Alexander Belopolsky 15 Dec '08

15 Dec '08

There is currently a unit test in the trunk that fails in verbose mode: $ ./python.exe Lib/test/test_doctest.py -v ... UnicodeEncodeError: 'ascii' codec can't encode characters in position 338-339: ordinal not in range(128) Apparently, the problem is that stdout cannot encode non-ascii characters: >>> sys.stdout.write(u'f\xf6\xf6') Traceback (most recent call last): File "<stdin>", line 1, in <module> UnicodeEncodeError: 'ascii' codec can't encode characters in position 1-2: ordinal not in range(128) which is strange because >>> sys.stdout.encoding 'UTF-8' and print has no problem with the same string: >>> print u'f\xf6\xf6' föö Where does 'ascii' codec come from?

2 1

Python-3.0, unicode, and os.environ
by Toshio Kuratomi 15 Dec '08

15 Dec '08

I opened up bug http://bugs.python.org/issue4006 a while ago and it was suggested in the report that it's not a bug but a feature and so I should come here to see about getting the feature changed :-) I have a specific problem with os.environ and a somewhat less important architectural issue with the unicode/bytes handling in certain os.* modules. I'll start with the important one: Currently in python3 there's no way to get at environment variables that are not encoded in the system default encoding. My understanding is that this isn't a problem on Windows systems but on *nix this is a huge problem. environment variables on *nix are a sequence of non-null bytes. These bytes are almost always "characters" but they do not have to be. Further, there is nothing that requires that the characters be in the same encoding; some of the characters could be in the UTF-8 character set while others are in latin-1, shift-jis, or big-5. These mixed encodings can occur for a variety of reasons. Here's an example that isn't too contrived :-) Swallow is a multi-user shell server hosted at a university in Japan. The OS installed is Fedora 10 where the encoding of all filenames provided by the OS are UTF-8. The administrator of the OS has kept this convention and, among other things has created a directory to mount and NFS directory from another computer. He calls that "ネットワーク" ("network" in Japanese). Since it's utf-8, that gets put on the filesystem as '\xe3\x83\x8d\xe3\x83\x83\xe3\x83\x88\xe3\x83\xaf\xe3\x83\xbc\xe3\x82\xaf' Now the administrators of the fileserver have been maintaining it since before Unicode was invented. Furthermore, they don't want to suffer from the space loss of using utf-8 to encode Japanese so they use shift-jis everywhere. They have a directory on the nfs share for programs that are useful for people on the shell server to access. It's called "プログラム" ("programs" in Japanese) Since they're using shift-jis, the bytes on the filesystem are: '\x83v\x83\x8d\x83O\x83\x89\x83\x80' The system administrator of the shell server adds the directory of programs to all his user's default PATH variables so then they have this: PATH=/bin:/usr/bin:/usr/local/bin:/mnt/\xe3\x83\x8d\xe3\x83\x83\xe3\x83\x88\xe3\x83\xaf\xe3\x83\xbc\xe3\x82\xaf/\x83v\x83\x8d\x83O\x83\x89\x83\x80 (Note: python syntax, In the unix shell you'd likely have octal instead of hex) Now comes the problematic part. One of the user's on the system wants to write a python3 program that needs to determine if a needed program is in the user's PATH. He tries to code it like this:: #!/usr/bin/python3.0 import os for directory in os.environ['PATH']: programs = os.listdir(directory) That code raises a KeyError because python3 has silently discarded the PATH due to the shift-jis encoded path elements. Much more importantly, there's no way the programmer can handle the KeyError and actually get the PATH from within python. In the bug report I opened, I listed four ways to fix this along with the pros and cons: 1) return mixed unicode and byte types in os.environ and os.getenv - I think this one is a bad idea. It's the easiest for simple code to deal with but it's repeating the major problem with python2's Unicode handling: mixing unicode and byte types unpredictably. 2) return only byte types in os.environ - This is conceptually correct but the most annoying option. Technically we're receiving bytes from the C libraries and the C libraries expect bytes in return. But in the common case we will be dealing with things in one encoding so this causes needless effort to the application programmer in the common case. 3) silently ignore non-decodable value when accessing os.environ['PATH'] as we do now but allow access to the full information via os.environ[b'PATH'] and os.getenvb(). - This mirrors the practice of os.listdir('.') vs os.listdir(b'.') and os.getcwd() vs os.getcwdb(). 4) raise an exception when non-decodable values are *accessed* and continue as in #3. This means that os.environ wouldn't be a simple dict as it would need to decode the values when keys are accessed (although it could cache the values). - This mirrors the practice of open() which is to decode the value for the common case but throw an exception and allow the programmer to decide what to do if all values are not decodable. Either #3 or #4 will solve the major problem and both have precedent in python3's current implementation. The difference between them is whether to throw an exception when a non-decodable value is encountered. Here's why I think that's appropriate: One of the things I enjoy about python is the informative tracebacks that make debugging easy. I think that the ease of debugging is lost when we silently ignore an error. If we look at the difference in coding and debugging for problems with files that aren't encoded in the default encoding (where a traceback is issued) and os.listdir() when filenames aren't in the default encoding (where the filenames are silently ignored), I think we'll see that:: #!/usr/bin/python3.0 # Code with two unicode problems: import os, sys directory = sys.stdin.readline().strip() for filename in os.listdir(directory): myfile = open(filename, 'r') print('%s: %s' % [os.path.join(directory, filename), myfile.readline()]) myfile.close() Let's say I write the above code and test it on a directory that's all encoded in the default encoding. I release it to the world. Someone uses it on a system that has files and filenames with mixed encodings. They immediately get a traceback like this: File "./test.py", line 7, in <module> print(myfile.readline()) [...] UnicodeDecodeError: 'utf8' codec can't decode bytes in position 24-26: invalid data With that information I can diagnose that my program is failing to read a line from a file because the file is not written in the default encoding (utf8 in this case). It points out that myfile on line 7 of test.py is the file object that has issues. I quickly fix it by doing this: + unknown_encoded_files = [] [...] + try: - print(myfile.readline()) + print('%s: %s' % [os.path.join(directory, filename), myfile.readline()]) + except UnicodeDecodeError: + unknown_encoded_files.append(filename) myfile.close() +if unknown_encoded_files: + print('These files are not in the default encoding:\n %s' % '\n '.join(unknown_encoded_files)) Very simple. The traceback has all the information I need to fix this. A little later I get another report from that user that my code is failing to list the first line of all the files in their home directory. This time there's no traceback to point out which of my files is failing, just that some files are being ignored. I ask for the list of files in the directory and get back: é.txt ñ.txt I create those files in a directory and they're processed fine. I tell the user that and ask if there's anything special about what's in the files or anything that makes them different. No... they're both text files on his machine. One was created there, though, and the other was copied from another machine. Hmm.. do the filenames show up mangled by any chance? Yes, one of them does but he knows it's correct since it shows up correctly on his machine at home. Ah ha! That seems to point at an encoding problem. But where? After writing a test and perusing my code for a while, I find my os.listdir() call. directory has to be converted to bytes for this to work. So I change the code like so: - for filename in os.listdir(directory): + for filename in os.listdir(directory.encode()): [...] - unknown_encoded_files.append(filename) + unknown_encoded_files.append(str(filename, errors='replace')) The code for the fix is simple but the debugging to find the problem is not. Raising an exception instead of silently failing is much better for getting code that works correctly. The bug report I opened suggests creating a PEP to address this issue. I think that's a good idea for whether os.listdir() and friends should be changed to raise an exception but not having any way to get at some environment variables seems like it's just a bug that needs to be addressed. What do other people think on both these issues? -Toshio

29 156