Mailman 3 September 2008 - Python-Dev

when is path==NULL?
by Ulrich Eckhardt Sept. 30, 2008

Sept. 30, 2008

Hi! I'm looking at trunk/Python/sysmodule.c, function PySys_SetArgv(). In that function, there is code like this: PyObject* path = PySys_GetObject("path"); ... if (path != NULL) { ... } My intuition says that if path==NULL, something is very wrong. At least I would expect to get 'None', but never NULL, except when out of memory. So, for the case that path==NULL', I would simply invoke Py_FatalError("no mem for sys.path"), similarly to the other call there. Sounds reasonable? … [View More]

4 4

Re: [Python-Dev] [Python-3000] New proposition for Python3 bytes filename issue
by Adam Olsen Sept. 30, 2008

Sept. 30, 2008

On Tue, Sep 30, 2008 at 12:22 AM, Georg Brandl <g.brandl(a)gmx.net> wrote: > Victor Stinner schrieb: >> Le Monday 29 September 2008 18:45:28 Georg Brandl, vous avez écrit : >>> If I had to choose, I'd still argue for the modified UTF-8 as filesystem >>> encoding (if it were UTF-8 otherwise), despite possible surprises when a >>> such-encoded filename escapes from Python. >> >> If I understand correctly this solution. The idea is to change … [View More]the default >> file system encoding, right? Eg. if your filesystem is UTF-8, use ISO-8859-1 >> to make sure that UTF-8 conversion will never fail. > > No, that was not what I meant (although it is another possibility). As I wrote, > Martin's proposal that I support here is using the modified UTF-8 codec that > successfully roundtrips otherwise invalid UTF-8 data. > > You seem to forget that (disregarding OSX here, since it already enforces > UTF-8) the majority of file names on Posix systems will be encoded correctly. > >> Let's try with an ugly directory on my UTF-8 file system: >> $ find >> .. >> ../têste >> ../ô >> ../a?b >> ../dossié >> ../dossié/abc >> ../dir?name >> ../dir?name/xyz >> >> Python3 using encoding=ISO-8859-1: >>>>> import os; os.listdir(b'.') >> [b't\xc3\xaaste', b'\xc3\xb4', b'a\xffb', b'dossi\xc3\xa9', b'dir\xffname'] >>>>> files=os.listdir('.'); files >> ['tÃªste', 'Ã´', 'aÿb', 'dossiÃ(c)', 'dirÿname'] >>>>> open(files[0]).close() >>>>> os.listdir(files[-1]) >> ['xyz'] >> >> Ok, I have unicode filenames and I'm able to open a file and list a directory. >> The problem is now to display correctly the filenames. >> >> For me "unicode" sounds like "text (characters) encoded in the correct >> charset". In this case, unicode is just a storage for *bytes* in a custom >> charset. > >> How can we mix <custom unicode (bytes encoded in ISO-8859-1)> with <real >> unicode>? Eg. os.path.join('dossiÃ(c)', "fichié") : first argument is encoded >> in ISO-8859-1 whereas the second argument is encoding in Unicode. It's >> something like that: >> str(b'dossi\xc3\xa9', 'ISO-8859-1') + '/' + 'fichi\xe9' >> >> Whereas the correct (unicode) result should be: >> 'dossié/fichié' >> as bytes in ISO-8859-1: >> b'dossi\xc3\xa9/fichi\xc3\xa9' >> as bytes in UTF-8: >> b'dossi\xe9/fichi\xe9' > > With the filenames decoded by UTF-8, your files named têste, ô, dossié will > be displayed and handled correctly. The others are *invalid* in the filesystem > encoding UTF-8 and therefore would be represented by something like > > u'dir\uXXffname' where XX is some private use Unicode namespace. It won't look > pretty when printed, but then, what do other applications do? They e.g. display > a question mark as you show above, which is not better in terms of readability. > > But it will work when given to a filename-handling function. Valid filenames > can be compared to Unicode strings. > > A real-world example: OpenOffice can't open files with invalid bytes in their > name. They are displayed in the "Open file" dialog, but trying to open fails. > This regularly drives me crazy. Let's not make Python not work this way too, > or, even worse, not even display those filenames. The only way to display that file would be to transform it into some other valid unicode string. However, as that string is already valid, you've just made any files named after it impossible to open. If you extend unicode then you're unable to display that extended name[1]. I think Guido's right on this one. If I have to choose between openoffice crashing or skipping my file, I'd vastly prefer it skip it. A warning would be a nice bonus (from python or from openoffice), telling me there's a buggered file I should go fix. Renaming the file is the end solution. [1] You could argue that Unicode should add new scalars to handle all currently invalid UTF-8 sequences. They could then output to their original forms if in UTF-8, or a mundane form in UTF-16 and UTF-32. However, I suspect "we don't want to add validation to linux" will not be a very persuasive argument. -- Adam Olsen, aka Rhamphoryncus [View Less]

2 2

Status of MS Windows CE port
by Ulrich Eckhardt Sept. 30, 2008

Sept. 30, 2008

Greetings! I'm currently trying to assess the effort required for a CE port. I'm already aware of the project at http://pythonce.sourceforge.net, but I find it a waste of time to have two separate projects which then require syncing changes back and forth. Questions: - What are the feelings towards a port in the mainline? I'm aware that you are trying to get 2.6 and 3.0 out of the door and are probably not willing to risk such changes to the release code right now, but in general? - Is … [View More]

4 11

Re: [Python-Dev] Patch for an initial support of bytes filename in Python3
by Brett Cannon Sept. 30, 2008

Sept. 30, 2008

On Mon, Sep 29, 2008 at 5:47 PM, Victor Stinner <victor.stinner(a)haypocalc.com> wrote: > Hi, > > See attached patch: python3_bytes_filename.patch > Patches should go on the tracker, not the mailing list. Otherwise it will just get lost. -Brett

1 0

Re: [Python-Dev] Real segmentation fault handler
by Ralf W. Grosse-Kunstleve Sept. 30, 2008

Sept. 30, 2008

FWIW: I didn't have much luck translating segfaults into exceptions. It (seemed) to work on some platforms, but not others; this was in the context of C++. In my experience, it is more useful to generate Python and C stack traces and bail out. I also do this for floating-point exceptions. The handlers are installed at runtime from a low-level extension module: http://cctbx.svn.sourceforge.net/viewvc/cctbx/trunk/boost_adaptbx/meta_ext.… Example output is below. It works under Linux and … [View More]partially under Mac OS X. Ralf % boost_adaptbx.segmentation_fault Now dereferencing null-pointer ... show_stack(1): /net/chevy/raid1/rwgk/dist/boost_adaptbx/command_line/segmentation_fault.py(10) run show_stack(2): /net/chevy/raid1/rwgk/dist/boost_adaptbx/command_line/segmentation_fault.py(14) <module> libc backtrace (18 frames, most recent call last): /net/chevy/raid1/rwgk/bintbx_py252/base/bin/python [0x4118e9] /lib64/libc.so.6(__libc_start_main+0xf4) [0x363241e074] /net/chevy/raid1/rwgk/bintbx_py252/base/bin/python(Py_Main+0x935) [0x4123c5] /net/chevy/raid1/rwgk/bintbx_py252/base/bin/python(PyRun_SimpleFileExFlags+0x1a0) [0x4a8860] /net/chevy/raid1/rwgk/bintbx_py252/base/bin/python(PyRun_FileExFlags+0x10e) [0x4a85ce] /net/chevy/raid1/rwgk/bintbx_py252/base/bin/python(PyEval_EvalCode+0x32) [0x487402] /net/chevy/raid1/rwgk/bintbx_py252/base/bin/python(PyEval_EvalCodeEx+0x81f) [0x4873bf] /net/chevy/raid1/rwgk/bintbx_py252/base/bin/python(PyEval_EvalFrameEx+0x6bc1) [0x486541] /net/chevy/raid1/rwgk/bintbx_py252/base/bin/python(PyEval_EvalFrameEx+0x2bb9) [0x482539] /net/chevy/raid1/rwgk/bintbx_py252/base/bin/python(PyObject_Call+0x13) [0x415ae3] /net/chevy/raid1/rwgk/bintbx_py252/lib/libboost_python.so [0x2aaaaba7c6f7] /net/chevy/raid1/rwgk/bintbx_py252/lib/libboost_python.so(boost::python::handle_exception_impl(boost::function0<void>)+0x28) [0x2aaaaba87148] /net/chevy/raid1/rwgk/bintbx_py252/lib/libboost_python.so(boost::function0<void>::operator()() const+0x19e) [0x2aaaaba8816e] /net/chevy/raid1/rwgk/bintbx_py252/lib/libboost_python.so [0x2aaaaba7fef8] /net/chevy/raid1/rwgk/bintbx_py252/lib/libboost_python.so(boost::python::objects::function::call(_object*, _object*) const+0x7d) [0x2aaaaba7fb5d] /net/chevy/raid1/rwgk/bintbx_py252/lib/boost_python_meta_ext.so(boost::python::objects::caller_py_function_impl<boost::python::detail::caller<char (*)(char const*), boost::python::default_call_policies, boost::mpl::vector2<char, char const*> > >::operator()(_object*, _object*)+0x29) [0x2aaaab8470a9] /net/chevy/raid1/rwgk/bintbx_py252/lib/boost_python_meta_ext.so [0x2aaaab843790] /lib64/libc.so.6 [0x3632430f30] Segmentation fault (Python and libc call stacks above) % boost_adaptbx.divide_by_zero Now dividing by zero (in C++) ... show_stack(1): /net/chevy/raid1/rwgk/dist/boost_adaptbx/command_line/divide_by_zero.py(10) run show_stack(2): /net/chevy/raid1/rwgk/dist/boost_adaptbx/command_line/divide_by_zero.py(14) <module> libc backtrace (18 frames, most recent call last): /net/chevy/raid1/rwgk/bintbx_py252/base/bin/python [0x4118e9] /lib64/libc.so.6(__libc_start_main+0xf4) [0x363241e074] /net/chevy/raid1/rwgk/bintbx_py252/base/bin/python(Py_Main+0x935) [0x4123c5] /net/chevy/raid1/rwgk/bintbx_py252/base/bin/python(PyRun_SimpleFileExFlags+0x1a0) [0x4a8860] /net/chevy/raid1/rwgk/bintbx_py252/base/bin/python(PyRun_FileExFlags+0x10e) [0x4a85ce] /net/chevy/raid1/rwgk/bintbx_py252/base/bin/python(PyEval_EvalCode+0x32) [0x487402] /net/chevy/raid1/rwgk/bintbx_py252/base/bin/python(PyEval_EvalCodeEx+0x81f) [0x4873bf] /net/chevy/raid1/rwgk/bintbx_py252/base/bin/python(PyEval_EvalFrameEx+0x6bc1) [0x486541] /net/chevy/raid1/rwgk/bintbx_py252/base/bin/python(PyEval_EvalFrameEx+0x2bb9) [0x482539] /net/chevy/raid1/rwgk/bintbx_py252/base/bin/python(PyObject_Call+0x13) [0x415ae3] /net/chevy/raid1/rwgk/bintbx_py252/lib/libboost_python.so [0x2aaaaba7c6f7] /net/chevy/raid1/rwgk/bintbx_py252/lib/libboost_python.so(boost::python::handle_exception_impl(boost::function0<void>)+0x28) [0x2aaaaba87148] /net/chevy/raid1/rwgk/bintbx_py252/lib/libboost_python.so(boost::function0<void>::operator()() const+0x19e) [0x2aaaaba8816e] /net/chevy/raid1/rwgk/bintbx_py252/lib/libboost_python.so [0x2aaaaba7fef8] /net/chevy/raid1/rwgk/bintbx_py252/lib/libboost_python.so(boost::python::objects::function::call(_object*, _object*) const+0x7d) [0x2aaaaba7fb5d] /net/chevy/raid1/rwgk/bintbx_py252/lib/boost_python_meta_ext.so(boost::python::objects::caller_py_function_impl<boost::python::detail::caller<double (*)(double const&, double const&), boost::python::default_call_policies, boost::mpl::vector3<double, double const&, double const&> > >::operator()(_object*, _object*)+0x12a) [0x2aaaab84759a] /net/chevy/raid1/rwgk/bintbx_py252/lib/boost_python_meta_ext.so [0x2aaaab8437a4] /lib64/libc.so.6 [0x3632430f30] Floating-point error (Python and libc call stacks above) ----- Original Message ---- From: Victor Stinner <victor.stinner(a)haypocalc.com> To: python-dev(a)python.org Sent: Monday, September 29, 2008 4:05:53 PM Subject: [Python-Dev] Real segmentation fault handler Hi, I would like to be able to catch SIGSEGV in my Python code! So I started to hack Python trunk to support this feature. The idea is to use a signal handler which call longjmp(), and add setjmp() at Py_EvalFrameEx() enter. See attached ("small") patch: segfault.patch Example read.py with the *evil* ctypes module of invalid memory read: ------------------- 8< -------------- from ctypes import string_at def fault(): text = string_at(1, 10) print("text = {0!r}".format(text)) def test(): print("test: 1") try: fault() except MemoryError, err: print "ooops!" print err print("test: 2") try: fault() except MemoryError, err: print "ooops!" print err print("test: end") def main(): test() if __name__ == "__main__": main() ------------------- 8< -------------- Result: ------------------- 8< -------------- $ python read.py test: 1 sizeof()=160 ooops! segmentation fault test: 2 sizeof()=160 ooops! segmentation fault test: end ------------------- 8< -------------- Example bug1.py of a stack overflow: ---------- loop = None, for i in xrange(10**5): loop = loop, None ---------- Result: ---------- $ python -i bug1.py >>> print loop (((((((((...Traceback (most recent call last): File "<stdin>", line 1, in <module> MemoryError: segmentation fault >>> ---------- Python is able to restore a valid state (stack/heap) after a segmentation fault and raise a classical Python exception (I choosed MemoryError, but it could be a specific exception). On my computer (Ubuntu Gutsy/i386), each segfault_frame takes sizeof(sigjmpbuf) + sizeof(void*) = 160 bytes, allocated on the stack. I don't know if it's huge or not, but that will limit the number of recursive calls. The feature can be optional if we add a configure option and some #ifdef/#endif. A dedicated stack is needed to be call the signal handler on stack overflow error. I choosed 4 KB, but since I only call longjmp(), smaller stack might also works. Does other VM support such feature? JVM, Mono, .NET, etc. ? I had the idea of catching SIGSEGV after reading the issue 1069092 (stack overflow because of too many recursive calls). -- Victor Stinner aka haypo http://www.haypocalc.com/blog/ [View Less]

1 0

New proposition for Python3 bytes filename issue
by Victor Stinner Sept. 30, 2008

Sept. 30, 2008

Hi, After reading the previous discussion, here is new proposition. Python 2.x and Windows are not affected by this issue. Only Python3 on POSIX (eg. Linux or *BSD) is affected. Some system are broken, but Python have to be able to open/copy/move/remove files with an "invalid filename". The issue can wait for Python 3.0.1 / 3.1. Windows ------- On Windows, we might reject bytes filenames for all file operations: open(), unlink(), os.path.join(), etc. (raise a TypeError or UnicodeError) … [View More]

4 5

',' precedence in documentation
by C. Titus Brown Sept. 30, 2008

Sept. 30, 2008

Hi folks, over on the pygr list, we were recently discussing a mildly confusing edit I made: assert 'seq1' in self.db, self.db.keys() This was interpreted by someone as being assert 'seq1' in (self.db, self.db.keys()) which is very different from the actual meaning, assert ('seq1' in self.db), self.db.keys() where 'self.db.keys()' serves as the message to be printed out when the assertion fails. Apart from questions of why I was apparently out to confuse people with this edit, … [View More]

16 28

Broken link to NASM download location
by Ulrich Eckhardt Sept. 29, 2008

Sept. 29, 2008

Hi! In trunk/PCbuild/readme.txt it says NASM is available from kernel.org. The project has moved to Sourceforge though, please replace the link with http://nasm.sf.net Thanks! Uli -- Sator Laser GmbH Geschäftsführer: Thorsten Föcking, Amtsgericht Hamburg HR B62 932 ************************************************************************************** Visit our website at <http://www.satorlaser.de/> ***********************************************************************… [View More]

2 1

Re: [Python-Dev] [Python-3000] Filename as byte string in python 2.6 or 3.0?
by Victor Stinner Sept. 29, 2008

Sept. 29, 2008

Le Monday 29 September 2008 06:43:55, vous avez écrit : > It will make users happy, and it's simple enough to implement for > python 3.0. I dislike your argument. A "quick and dirty hack" is always faster to implement than a real solution, but we may hits later new issues if we don't choose the right solution. -- Victor Stinner aka haypo http://www.haypocalc.com/blog/

1 0

Reminder: Python 2.6 final next Wednesday
by Barry Warsaw Sept. 29, 2008

Sept. 29, 2008

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 This is a reminder that Python 2.6 final is schedule for release next Wednesday, October 1st. Once again, I've gone through the release blocker issues and knocked anything that doesn't specifically affect 2.6 to deferred blocker. This leaves us with 7 open blocking issues. Please spend some time over the next several days reviewing patches, making comments and working toward closing these issues. Email me directly if you have any … [View More]

4 3