[Moin-user] Re: Re: Image formatters and UTF8

[Jiba]

On Sun, 5 Feb 2006 11:02:12 +0100
Alexander Schremmer <2005a at usenet.alexanderweb.de> wrote:

> On Sat, 4 Feb 2006 13:17:05 +0100, Jiba wrote:
> 
> > Actually the formatter must return unicode object (because the cache
> > system concatenate the returned string with other unicode strings).
> 
> Which cache system do you mean? Of course you should not return Unicode
> strings from your formatter. The content-type is set based on your mimetype
> parameter.


I use the same cache system than for text_html, i.e. caching.py.

My formatter returns Unicode ; if it don't, I get the following error :

Traceback (most recent call last):
  File "/home/jiba/src/MoinMoin/request.py", line 1020, in run
    handler(self.page.page_name, self)
  File "/home/jiba/src/MoinMoin/wikiaction.py", line 825, in do_format
    Page(request, pagename, formatter=Formatter(request)).send_page(request)
  File "/home/jiba/src/MoinMoin/Page.py", line 1241, in send_page
    start_line=pi_lines)
  File "/home/jiba/src/MoinMoin/Page.py", line 1327, in send_page_content
    code = self.makeCache(request, parser)
  File "/home/jiba/src/MoinMoin/Page.py", line 1378, in makeCache
    text = request.redirectedOutput(parser.format, formatter)
  File "/home/jiba/src/MoinMoin/request.py", line 627, in redirectedOutput
    function(*args, **kw)
  File "/home/jiba/src/MoinMoin/parser/wiki.py", line 1077, in format
    formatted_line = self.scan(scan_re, line)
  File "/home/jiba/src/MoinMoin/parser/wiki.py", line 873, in scan
    return u''.join(result)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 0: ordinal not in range(128)


The call to u"".join(...) force Unicode for all string, which is not possible for a string that represent a PNG or PDF raw data (since string may containt special non ASCII characters).
I get the same error without cache (but at a different traceback):


Traceback (most recent call last):
  File "/home/jiba/src/MoinMoin/request.py", line 1020, in run
    handler(self.page.page_name, self)
  File "/home/jiba/src/MoinMoin/wikiaction.py", line 825, in do_format
    Page(request, pagename, formatter=Formatter(request)).send_page(request)
  File "/home/jiba/src/MoinMoin/Page.py", line 1241, in send_page
    start_line=pi_lines)
  File "/home/jiba/src/MoinMoin/Page.py", line 1320, in send_page_content
    self.format(parser)
  File "/home/jiba/src/MoinMoin/Page.py", line 1337, in format
    parser.format(self.formatter)
  File "/home/jiba/src/MoinMoin/parser/wiki.py", line 1077, in format
    formatted_line = self.scan(scan_re, line)
  File "/home/jiba/src/MoinMoin/parser/wiki.py", line 873, in scan
    return u''.join(result)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 0: ordinal not in range(128)


Maybe we should use "".join(...) if (and only if) all string are non-unicode.

Jiba