Mailman 3 September 2008 - Python-Dev

when is path==NULL?
by Ulrich Eckhardt 30 Sep '08

30 Sep '08

Hi! I'm looking at trunk/Python/sysmodule.c, function PySys_SetArgv(). In that function, there is code like this: PyObject* path = PySys_GetObject("path"); ... if (path != NULL) { ... } My intuition says that if path==NULL, something is very wrong. At least I would expect to get 'None', but never NULL, except when out of memory. So, for the case that path==NULL', I would simply invoke Py_FatalError("no mem for sys.path"), similarly to the other call there. Sounds reasonable? Uli -- Sator Laser GmbH Geschäftsführer: Thorsten Föcking, Amtsgericht Hamburg HR B62 932 ************************************************************************************** Visit our website at <http://www.satorlaser.de/> ************************************************************************************** Diese E-Mail einschließlich sämtlicher Anhänge ist nur für den Adressaten bestimmt und kann vertrauliche Informationen enthalten. Bitte benachrichtigen Sie den Absender umgehend, falls Sie nicht der beabsichtigte Empfänger sein sollten. Die E-Mail ist in diesem Fall zu löschen und darf weder gelesen, weitergeleitet, veröffentlicht oder anderweitig benutzt werden. E-Mails können durch Dritte gelesen werden und Viren sowie nichtautorisierte Änderungen enthalten. Sator Laser GmbH ist für diese Folgen nicht verantwortlich. **************************************************************************************

4 4

Re: [Python-Dev] [Python-3000] New proposition for Python3 bytes filename issue
by Adam Olsen 30 Sep '08

30 Sep '08

On Tue, Sep 30, 2008 at 12:22 AM, Georg Brandl <g.brandl(a)gmx.net> wrote: > Victor Stinner schrieb: >> Le Monday 29 September 2008 18:45:28 Georg Brandl, vous avez écrit : >>> If I had to choose, I'd still argue for the modified UTF-8 as filesystem >>> encoding (if it were UTF-8 otherwise), despite possible surprises when a >>> such-encoded filename escapes from Python. >> >> If I understand correctly this solution. The idea is to change the default >> file system encoding, right? Eg. if your filesystem is UTF-8, use ISO-8859-1 >> to make sure that UTF-8 conversion will never fail. > > No, that was not what I meant (although it is another possibility). As I wrote, > Martin's proposal that I support here is using the modified UTF-8 codec that > successfully roundtrips otherwise invalid UTF-8 data. > > You seem to forget that (disregarding OSX here, since it already enforces > UTF-8) the majority of file names on Posix systems will be encoded correctly. > >> Let's try with an ugly directory on my UTF-8 file system: >> $ find >> .. >> ../têste >> ../ô >> ../a?b >> ../dossié >> ../dossié/abc >> ../dir?name >> ../dir?name/xyz >> >> Python3 using encoding=ISO-8859-1: >>>>> import os; os.listdir(b'.') >> [b't\xc3\xaaste', b'\xc3\xb4', b'a\xffb', b'dossi\xc3\xa9', b'dir\xffname'] >>>>> files=os.listdir('.'); files >> ['tÃªste', 'Ã´', 'aÿb', 'dossiÃ(c)', 'dirÿname'] >>>>> open(files[0]).close() >>>>> os.listdir(files[-1]) >> ['xyz'] >> >> Ok, I have unicode filenames and I'm able to open a file and list a directory. >> The problem is now to display correctly the filenames. >> >> For me "unicode" sounds like "text (characters) encoded in the correct >> charset". In this case, unicode is just a storage for *bytes* in a custom >> charset. > >> How can we mix <custom unicode (bytes encoded in ISO-8859-1)> with <real >> unicode>? Eg. os.path.join('dossiÃ(c)', "fichié") : first argument is encoded >> in ISO-8859-1 whereas the second argument is encoding in Unicode. It's >> something like that: >> str(b'dossi\xc3\xa9', 'ISO-8859-1') + '/' + 'fichi\xe9' >> >> Whereas the correct (unicode) result should be: >> 'dossié/fichié' >> as bytes in ISO-8859-1: >> b'dossi\xc3\xa9/fichi\xc3\xa9' >> as bytes in UTF-8: >> b'dossi\xe9/fichi\xe9' > > With the filenames decoded by UTF-8, your files named têste, ô, dossié will > be displayed and handled correctly. The others are *invalid* in the filesystem > encoding UTF-8 and therefore would be represented by something like > > u'dir\uXXffname' where XX is some private use Unicode namespace. It won't look > pretty when printed, but then, what do other applications do? They e.g. display > a question mark as you show above, which is not better in terms of readability. > > But it will work when given to a filename-handling function. Valid filenames > can be compared to Unicode strings. > > A real-world example: OpenOffice can't open files with invalid bytes in their > name. They are displayed in the "Open file" dialog, but trying to open fails. > This regularly drives me crazy. Let's not make Python not work this way too, > or, even worse, not even display those filenames. The only way to display that file would be to transform it into some other valid unicode string. However, as that string is already valid, you've just made any files named after it impossible to open. If you extend unicode then you're unable to display that extended name[1]. I think Guido's right on this one. If I have to choose between openoffice crashing or skipping my file, I'd vastly prefer it skip it. A warning would be a nice bonus (from python or from openoffice), telling me there's a buggered file I should go fix. Renaming the file is the end solution. [1] You could argue that Unicode should add new scalars to handle all currently invalid UTF-8 sequences. They could then output to their original forms if in UTF-8, or a mundane form in UTF-16 and UTF-32. However, I suspect "we don't want to add validation to linux" will not be a very persuasive argument. -- Adam Olsen, aka Rhamphoryncus

2 2

Status of MS Windows CE port
by Ulrich Eckhardt 30 Sep '08

30 Sep '08

Greetings! I'm currently trying to assess the effort required for a CE port. I'm already aware of the project at http://pythonce.sourceforge.net, but I find it a waste of time to have two separate projects which then require syncing changes back and forth. Questions: - What are the feelings towards a port in the mainline? I'm aware that you are trying to get 2.6 and 3.0 out of the door and are probably not willing to risk such changes to the release code right now, but in general? - Is anyone already working on this? I know that Brad Clements made an attempt some years ago but didn't get it integrated, are there any others? Maybe even any ongoing work? - I see that you are using Subversion and Bazaar, would it be possible to get a feature branch for the CE port? I'm thinking about the effort going on at pythonce.sf.net and some further porting that I need for my specific platform. IMHO both would be better off in the same repository as your upstream sources, since merging changes back and forth is much easier that way. Thanks! Uli -- Sator Laser GmbH Geschäftsführer: Thorsten Föcking, Amtsgericht Hamburg HR B62 932 ************************************************************************************** Visit our website at <http://www.satorlaser.de/> ************************************************************************************** Diese E-Mail einschließlich sämtlicher Anhänge ist nur für den Adressaten bestimmt und kann vertrauliche Informationen enthalten. Bitte benachrichtigen Sie den Absender umgehend, falls Sie nicht der beabsichtigte Empfänger sein sollten. Die E-Mail ist in diesem Fall zu löschen und darf weder gelesen, weitergeleitet, veröffentlicht oder anderweitig benutzt werden. E-Mails können durch Dritte gelesen werden und Viren sowie nichtautorisierte Änderungen enthalten. Sator Laser GmbH ist für diese Folgen nicht verantwortlich. **************************************************************************************

4 11

Re: [Python-Dev] Patch for an initial support of bytes filename in Python3
by Brett Cannon 30 Sep '08

30 Sep '08

On Mon, Sep 29, 2008 at 5:47 PM, Victor Stinner <victor.stinner(a)haypocalc.com> wrote: > Hi, > > See attached patch: python3_bytes_filename.patch > Patches should go on the tracker, not the mailing list. Otherwise it will just get lost. -Brett

1 0

Re: [Python-Dev] Real segmentation fault handler
by Ralf W. Grosse-Kunstleve 30 Sep '08

30 Sep '08

FWIW: I didn't have much luck translating segfaults into exceptions. It (seemed) to work on some platforms, but not others; this was in the context of C++. In my experience, it is more useful to generate Python and C stack traces and bail out. I also do this for floating-point exceptions. The handlers are installed at runtime from a low-level extension module: http://cctbx.svn.sourceforge.net/viewvc/cctbx/trunk/boost_adaptbx/meta_ext.… Example output is below. It works under Linux and partially under Mac OS X. Ralf % boost_adaptbx.segmentation_fault Now dereferencing null-pointer ... show_stack(1): /net/chevy/raid1/rwgk/dist/boost_adaptbx/command_line/segmentation_fault.py(10) run show_stack(2): /net/chevy/raid1/rwgk/dist/boost_adaptbx/command_line/segmentation_fault.py(14) <module> libc backtrace (18 frames, most recent call last): /net/chevy/raid1/rwgk/bintbx_py252/base/bin/python [0x4118e9] /lib64/libc.so.6(__libc_start_main+0xf4) [0x363241e074] /net/chevy/raid1/rwgk/bintbx_py252/base/bin/python(Py_Main+0x935) [0x4123c5] /net/chevy/raid1/rwgk/bintbx_py252/base/bin/python(PyRun_SimpleFileExFlags+0x1a0) [0x4a8860] /net/chevy/raid1/rwgk/bintbx_py252/base/bin/python(PyRun_FileExFlags+0x10e) [0x4a85ce] /net/chevy/raid1/rwgk/bintbx_py252/base/bin/python(PyEval_EvalCode+0x32) [0x487402] /net/chevy/raid1/rwgk/bintbx_py252/base/bin/python(PyEval_EvalCodeEx+0x81f) [0x4873bf] /net/chevy/raid1/rwgk/bintbx_py252/base/bin/python(PyEval_EvalFrameEx+0x6bc1) [0x486541] /net/chevy/raid1/rwgk/bintbx_py252/base/bin/python(PyEval_EvalFrameEx+0x2bb9) [0x482539] /net/chevy/raid1/rwgk/bintbx_py252/base/bin/python(PyObject_Call+0x13) [0x415ae3] /net/chevy/raid1/rwgk/bintbx_py252/lib/libboost_python.so [0x2aaaaba7c6f7] /net/chevy/raid1/rwgk/bintbx_py252/lib/libboost_python.so(boost::python::handle_exception_impl(boost::function0<void>)+0x28) [0x2aaaaba87148] /net/chevy/raid1/rwgk/bintbx_py252/lib/libboost_python.so(boost::function0<void>::operator()() const+0x19e) [0x2aaaaba8816e] /net/chevy/raid1/rwgk/bintbx_py252/lib/libboost_python.so [0x2aaaaba7fef8] /net/chevy/raid1/rwgk/bintbx_py252/lib/libboost_python.so(boost::python::objects::function::call(_object*, _object*) const+0x7d) [0x2aaaaba7fb5d] /net/chevy/raid1/rwgk/bintbx_py252/lib/boost_python_meta_ext.so(boost::python::objects::caller_py_function_impl<boost::python::detail::caller<char (*)(char const*), boost::python::default_call_policies, boost::mpl::vector2<char, char const*> > >::operator()(_object*, _object*)+0x29) [0x2aaaab8470a9] /net/chevy/raid1/rwgk/bintbx_py252/lib/boost_python_meta_ext.so [0x2aaaab843790] /lib64/libc.so.6 [0x3632430f30] Segmentation fault (Python and libc call stacks above) % boost_adaptbx.divide_by_zero Now dividing by zero (in C++) ... show_stack(1): /net/chevy/raid1/rwgk/dist/boost_adaptbx/command_line/divide_by_zero.py(10) run show_stack(2): /net/chevy/raid1/rwgk/dist/boost_adaptbx/command_line/divide_by_zero.py(14) <module> libc backtrace (18 frames, most recent call last): /net/chevy/raid1/rwgk/bintbx_py252/base/bin/python [0x4118e9] /lib64/libc.so.6(__libc_start_main+0xf4) [0x363241e074] /net/chevy/raid1/rwgk/bintbx_py252/base/bin/python(Py_Main+0x935) [0x4123c5] /net/chevy/raid1/rwgk/bintbx_py252/base/bin/python(PyRun_SimpleFileExFlags+0x1a0) [0x4a8860] /net/chevy/raid1/rwgk/bintbx_py252/base/bin/python(PyRun_FileExFlags+0x10e) [0x4a85ce] /net/chevy/raid1/rwgk/bintbx_py252/base/bin/python(PyEval_EvalCode+0x32) [0x487402] /net/chevy/raid1/rwgk/bintbx_py252/base/bin/python(PyEval_EvalCodeEx+0x81f) [0x4873bf] /net/chevy/raid1/rwgk/bintbx_py252/base/bin/python(PyEval_EvalFrameEx+0x6bc1) [0x486541] /net/chevy/raid1/rwgk/bintbx_py252/base/bin/python(PyEval_EvalFrameEx+0x2bb9) [0x482539] /net/chevy/raid1/rwgk/bintbx_py252/base/bin/python(PyObject_Call+0x13) [0x415ae3] /net/chevy/raid1/rwgk/bintbx_py252/lib/libboost_python.so [0x2aaaaba7c6f7] /net/chevy/raid1/rwgk/bintbx_py252/lib/libboost_python.so(boost::python::handle_exception_impl(boost::function0<void>)+0x28) [0x2aaaaba87148] /net/chevy/raid1/rwgk/bintbx_py252/lib/libboost_python.so(boost::function0<void>::operator()() const+0x19e) [0x2aaaaba8816e] /net/chevy/raid1/rwgk/bintbx_py252/lib/libboost_python.so [0x2aaaaba7fef8] /net/chevy/raid1/rwgk/bintbx_py252/lib/libboost_python.so(boost::python::objects::function::call(_object*, _object*) const+0x7d) [0x2aaaaba7fb5d] /net/chevy/raid1/rwgk/bintbx_py252/lib/boost_python_meta_ext.so(boost::python::objects::caller_py_function_impl<boost::python::detail::caller<double (*)(double const&, double const&), boost::python::default_call_policies, boost::mpl::vector3<double, double const&, double const&> > >::operator()(_object*, _object*)+0x12a) [0x2aaaab84759a] /net/chevy/raid1/rwgk/bintbx_py252/lib/boost_python_meta_ext.so [0x2aaaab8437a4] /lib64/libc.so.6 [0x3632430f30] Floating-point error (Python and libc call stacks above) ----- Original Message ---- From: Victor Stinner <victor.stinner(a)haypocalc.com> To: python-dev(a)python.org Sent: Monday, September 29, 2008 4:05:53 PM Subject: [Python-Dev] Real segmentation fault handler Hi, I would like to be able to catch SIGSEGV in my Python code! So I started to hack Python trunk to support this feature. The idea is to use a signal handler which call longjmp(), and add setjmp() at Py_EvalFrameEx() enter. See attached ("small") patch: segfault.patch Example read.py with the *evil* ctypes module of invalid memory read: ------------------- 8< -------------- from ctypes import string_at def fault(): text = string_at(1, 10) print("text = {0!r}".format(text)) def test(): print("test: 1") try: fault() except MemoryError, err: print "ooops!" print err print("test: 2") try: fault() except MemoryError, err: print "ooops!" print err print("test: end") def main(): test() if __name__ == "__main__": main() ------------------- 8< -------------- Result: ------------------- 8< -------------- $ python read.py test: 1 sizeof()=160 ooops! segmentation fault test: 2 sizeof()=160 ooops! segmentation fault test: end ------------------- 8< -------------- Example bug1.py of a stack overflow: ---------- loop = None, for i in xrange(10**5): loop = loop, None ---------- Result: ---------- $ python -i bug1.py >>> print loop (((((((((...Traceback (most recent call last): File "<stdin>", line 1, in <module> MemoryError: segmentation fault >>> ---------- Python is able to restore a valid state (stack/heap) after a segmentation fault and raise a classical Python exception (I choosed MemoryError, but it could be a specific exception). On my computer (Ubuntu Gutsy/i386), each segfault_frame takes sizeof(sigjmpbuf) + sizeof(void*) = 160 bytes, allocated on the stack. I don't know if it's huge or not, but that will limit the number of recursive calls. The feature can be optional if we add a configure option and some #ifdef/#endif. A dedicated stack is needed to be call the signal handler on stack overflow error. I choosed 4 KB, but since I only call longjmp(), smaller stack might also works. Does other VM support such feature? JVM, Mono, .NET, etc. ? I had the idea of catching SIGSEGV after reading the issue 1069092 (stack overflow because of too many recursive calls). -- Victor Stinner aka haypo http://www.haypocalc.com/blog/

1 0

New proposition for Python3 bytes filename issue
by Victor Stinner 29 Sep '08

29 Sep '08

Hi, After reading the previous discussion, here is new proposition. Python 2.x and Windows are not affected by this issue. Only Python3 on POSIX (eg. Linux or *BSD) is affected. Some system are broken, but Python have to be able to open/copy/move/remove files with an "invalid filename". The issue can wait for Python 3.0.1 / 3.1. Windows ------- On Windows, we might reject bytes filenames for all file operations: open(), unlink(), os.path.join(), etc. (raise a TypeError or UnicodeError) POSIX OS -------- The default behaviour should be to use unicode and raise an error if conversion to unicode fails. It should also be possible to use bytes using bytes arguments and optional arguments (for getcwd). - listdir(unicode) -> unicode and raise an error on invalid filename - listdir(bytes) -> bytes - getcwd() -> unicode - getcwd(bytes=True) -> bytes - open(): accept bytes or unicode os.path.*() should accept operations on bytes filenames, but maybe not on bytes+unicode arguments. os.path.join('directory', b'filename'): raise an error (or use *implicit* conversion to bytes)? When the user wants to display a filename to the screen, he can uses: text = str(filename, fs_encoding, "replace") -- Victor Stinner aka haypo http://www.haypocalc.com/blog/

4 5

',' precedence in documentation
by C. Titus Brown 29 Sep '08

29 Sep '08

Hi folks, over on the pygr list, we were recently discussing a mildly confusing edit I made: assert 'seq1' in self.db, self.db.keys() This was interpreted by someone as being assert 'seq1' in (self.db, self.db.keys()) which is very different from the actual meaning, assert ('seq1' in self.db), self.db.keys() where 'self.db.keys()' serves as the message to be printed out when the assertion fails. Apart from questions of why I was apparently out to confuse people with this edit, the question of ',' precedence came up. Looking at http://docs.python.org/ref/summary.html there's no mention of ',' binding and tuple creation vs the other operators. I'm not really sure how to think of it but it seems like this is something that could be cleared up in the docs with a brief note somewhere. Thoughts? cheers, --titus -- C. Titus Brown, ctb(a)msu.edu

16 28

Broken link to NASM download location
by Ulrich Eckhardt 29 Sep '08

29 Sep '08

Hi! In trunk/PCbuild/readme.txt it says NASM is available from kernel.org. The project has moved to Sourceforge though, please replace the link with http://nasm.sf.net Thanks! Uli -- Sator Laser GmbH Geschäftsführer: Thorsten Föcking, Amtsgericht Hamburg HR B62 932 ************************************************************************************** Visit our website at <http://www.satorlaser.de/> ************************************************************************************** Diese E-Mail einschließlich sämtlicher Anhänge ist nur für den Adressaten bestimmt und kann vertrauliche Informationen enthalten. Bitte benachrichtigen Sie den Absender umgehend, falls Sie nicht der beabsichtigte Empfänger sein sollten. Die E-Mail ist in diesem Fall zu löschen und darf weder gelesen, weitergeleitet, veröffentlicht oder anderweitig benutzt werden. E-Mails können durch Dritte gelesen werden und Viren sowie nichtautorisierte Änderungen enthalten. Sator Laser GmbH ist für diese Folgen nicht verantwortlich. **************************************************************************************

2 1

Re: [Python-Dev] [Python-3000] Filename as byte string in python 2.6 or 3.0?
by Victor Stinner 29 Sep '08

29 Sep '08

Le Monday 29 September 2008 06:43:55, vous avez écrit : > It will make users happy, and it's simple enough to implement for > python 3.0. I dislike your argument. A "quick and dirty hack" is always faster to implement than a real solution, but we may hits later new issues if we don't choose the right solution. -- Victor Stinner aka haypo http://www.haypocalc.com/blog/

1 0

Reminder: Python 2.6 final next Wednesday
by Barry Warsaw 29 Sep '08

29 Sep '08

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 This is a reminder that Python 2.6 final is schedule for release next Wednesday, October 1st. Once again, I've gone through the release blocker issues and knocked anything that doesn't specifically affect 2.6 to deferred blocker. This leaves us with 7 open blocking issues. Please spend some time over the next several days reviewing patches, making comments and working toward closing these issues. Email me directly if you have any questions, or ping me on irc. Unfortunately, my 'net connection may be a little bit flakey until Wednesday, but I will do my best to get online and follow up as needed. Please pitch in to help get Python 2.6 released on time! Thanks, - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (Darwin) iQCVAwUBSN1hJnEjvBPtnXfVAQIQfQQAgYr0tuzJhm3LZX/1SaCwRJcX09UvNH1I CjZHs2TKS22MjF9d3mmBgcEJPl9AwGE+6EF6OiSgrsNRoRtnN0MMT3nQo+deRkan P3jUgMFJMFkA7Uq5MmuNnEnKZXa/bsu/8Om/4wqHqvtDXbUQkZPfyE8BFwBzJJSM Aa6Wp3wieFs= =r4iD -----END PGP SIGNATURE-----

4 3