[ python-Bugs-1293741 ] doctest runner cannot handle non-ascii characters

[SourceForge.net]

Bugs item #1293741, was opened at 2005-09-17 07:41
Message generated for change (Comment added) made by tim_one
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1293741&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Extension Modules
Group: Python 2.4
Status: Open
Resolution: None
Priority: 5
Submitted By: GRISEL (ogrisel)
>Assigned to: Nobody/Anonymous (nobody)
Summary: doctest runner cannot handle non-ascii characters 

Initial Comment:
The doctest module fails when the expected result
string has non-ascii charcaters even if the # -*-
coding: XXX -*- line is properly set.

The enclosed code sample produce the following error:

Traceback (most recent call last):
  File "test_iso-8859-15.py", line 41, in ?
    _test()
  File "test_iso-8859-15.py", line 26, in _test
    tried, failed = runner.run(t)
  File "/usr/lib/python2.4/doctest.py", line 1376, in run
    return self.__run(test, compileflags, out)
  File "/usr/lib/python2.4/doctest.py", line 1259, in __run
    if check(example.want, got, self.optionflags):
  File "/usr/lib/python2.4/doctest.py", line 1475, in
check_output
    if got == want:
UnicodeDecodeError: 'ascii' codec can't decode byte
0xe9 in position 8: ordinal not in range(128)



----------------------------------------------------------------------

>Comment By: Tim Peters (tim_one)
Date: 2006-04-23 21:21

Message:
Logged In: YES 
user_id=31435

Unassigned myself -- don't know enough about encodings.

----------------------------------------------------------------------

Comment By: Bjorn Tillenius (bjoti)
Date: 2006-02-16 06:41

Message:
Logged In: YES 
user_id=1032069

I'm quite sure that you can use non-ASCII characters in 
your doctest, given that it's a unicode string. So if you 
make your docstring a unicode string, it should work. That 
is:

u"""Docstring containing non-ASCII characters.
...
"""


----------------------------------------------------------------------

Comment By: GRISEL (ogrisel)
Date: 2005-09-18 06:25

Message:
Logged In: YES 
user_id=795041

Unfortunateny that patch does not fix my problem. The patch
at bug #1080727 fixes the problem for doctests written in
external reST files (testfile and DocFileTest functions). My
problem is related to internal docstring encoding (testmod
for instance). However, Bjorn Tillenius says:
"""
If one writes doctests within documentation strings of
classes and
functions, it's possible to use non-ASCII characters since
one can
specify the encoding used in the source file.
"""
So according to him, docstrings' doctests with non-ascii
characters should work by default. So maybe my system setup
is somewhat broken. Could somebody please confirm/infirm
this by running the attached sample script on his system?

My system config:
LANG=fr_FR at euro (on linux)
python 2.4.1 with:  sys.getdefaultencoding() == 'ascii' 
and locale.getpreferredencoding() == 'ISO-8859-15'
$ file test_iso-8859-15.py
test_iso-8859-15.py: ISO-8859 English text


----------------------------------------------------------------------

Comment By: Tim Peters (tim_one)
Date: 2005-09-17 13:42

Message:
Logged In: YES 
user_id=31435

Please try the patch at

http://www.python.org/sf/1080727

and report back on whether it solves your problem (attaching 
comments to the patch report would be most useful).

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1293741&group_id=5470