[Twisted-Python] print unicode
Hello! I'm using Twisted 10.0 and as usually sometime print debug infos with myunicodestr.encode('UTF-8') which are saved to logfile, but since using twisted 10 I'm getting UnicodeEncodeError: 'ascii' codec can't encode characters... type(myunicodestr) returns <type 'unicode'> What is the problem here? Thanks!
On Wednesday 05 May 2010, Pet wrote:
I'm using Twisted 10.0 and as usually sometime print debug infos with myunicodestr.encode('UTF-8') which are saved to logfile, but since using twisted 10 I'm getting
UnicodeEncodeError: 'ascii' codec can't encode characters...
UTF-8 uses the full 8 bits of a byte, while ASCII only uses 7, so writing Unicode encoded as UTF-8 to an ASCII stream is not valid. I think recent Python versions are more strict about what is written to stdout/stderr than older versions, it might not be related to Twisted itself. You can specify a different encoding for stdin/out/err by setting the PYTHONIOENCODING environment variable. Bye, Maarten
On Wed, May 5, 2010 at 2:05 PM, Maarten ter Huurne <maarten@treewalker.org> wrote:
On Wednesday 05 May 2010, Pet wrote:
I'm using Twisted 10.0 and as usually sometime print debug infos with myunicodestr.encode('UTF-8') which are saved to logfile, but since using twisted 10 I'm getting
UnicodeEncodeError: 'ascii' codec can't encode characters...
UTF-8 uses the full 8 bits of a byte, while ASCII only uses 7, so writing Unicode encoded as UTF-8 to an ASCII stream is not valid.
I think recent Python versions are more strict about what is written to stdout/stderr than older versions, it might not be related to Twisted itself. You can specify a different encoding for stdin/out/err by setting the PYTHONIOENCODING environment variable.
Hi Maarten! Thanks for help! Unfortunately, my Python installation is 2.5.2 and PYTHONIOENCODING is introduced in 2.6 Pet
Bye, Maarten
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On 05/05/10 13:31, Pet wrote:
On Wed, May 5, 2010 at 2:05 PM, Maarten ter Huurne <maarten@treewalker.org> wrote:
On Wednesday 05 May 2010, Pet wrote:
I'm using Twisted 10.0 and as usually sometime print debug infos with myunicodestr.encode('UTF-8') which are saved to logfile, but since using twisted 10 I'm getting
UnicodeEncodeError: 'ascii' codec can't encode characters...
UTF-8 uses the full 8 bits of a byte, while ASCII only uses 7, so writing Unicode encoded as UTF-8 to an ASCII stream is not valid.
I think recent Python versions are more strict about what is written to stdout/stderr than older versions, it might not be related to Twisted itself. You can specify a different encoding for stdin/out/err by setting the PYTHONIOENCODING environment variable.
Hi Maarten!
Thanks for help! Unfortunately, my Python installation is 2.5.2 and PYTHONIOENCODING is introduced in 2.6
I think this is highly dependent on your OS environment. For example: Python 2.4.3 (#1, Oct 23 2006, 14:19:47) [GCC 4.1.1 20060525 (Red Hat 4.1.1-1)] o Type "help", "copyright", "credits" or "
import sys sys.getdefaultencoding() 'ascii' print unichr(163) £
print unichr(163) Traceback (most recent call last): File "<stdin>", line 1, in ? UnicodeEncodeError: 'ascii' codec can't encode character u'\xa3' in
[pjm3@wildfire ~]$ echo $LANG en_GB.UTF-8 ...but: LANG=C python Python 2.4.3 (#1, Oct 23 2006, 14:19:47) [GCC 4.1.1 20060525 (Red Hat 4.1.1-1)] on linux2 Type "help", "copyright", "credits" or "license" for more information. position 0: ordinal not in range(128) ...i.e. here I can just print unicode characters, with nothing particularly special, provided my environment variables are set right.
On Wed, May 5, 2010 at 2:47 PM, Phil Mayers <p.mayers@imperial.ac.uk> wrote:
On 05/05/10 13:31, Pet wrote:
On Wed, May 5, 2010 at 2:05 PM, Maarten ter Huurne <maarten@treewalker.org> wrote:
On Wednesday 05 May 2010, Pet wrote:
I'm using Twisted 10.0 and as usually sometime print debug infos with myunicodestr.encode('UTF-8') which are saved to logfile, but since using twisted 10 I'm getting
UnicodeEncodeError: 'ascii' codec can't encode characters...
UTF-8 uses the full 8 bits of a byte, while ASCII only uses 7, so writing Unicode encoded as UTF-8 to an ASCII stream is not valid.
I think recent Python versions are more strict about what is written to stdout/stderr than older versions, it might not be related to Twisted itself. You can specify a different encoding for stdin/out/err by setting the PYTHONIOENCODING environment variable.
Hi Maarten!
Thanks for help! Unfortunately, my Python installation is 2.5.2 and PYTHONIOENCODING is introduced in 2.6
I think this is highly dependent on your OS environment. For example:
Python 2.4.3 (#1, Oct 23 2006, 14:19:47) [GCC 4.1.1 20060525 (Red Hat 4.1.1-1)] o Type "help", "copyright", "credits" or " >>> import sys >>> sys.getdefaultencoding() 'ascii' >>> print unichr(163) £
on python console it works for me, but not in application, if the string to be printed is fetched from db which is utf-8
[pjm3@wildfire ~]$ echo $LANG en_GB.UTF-8
...but:
LANG=C python Python 2.4.3 (#1, Oct 23 2006, 14:19:47) [GCC 4.1.1 20060525 (Red Hat 4.1.1-1)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> print unichr(163) Traceback (most recent call last): File "<stdin>", line 1, in ? UnicodeEncodeError: 'ascii' codec can't encode character u'\xa3' in position 0: ordinal not in range(128)
...i.e. here I can just print unicode characters, with nothing particularly special, provided my environment variables are set right.
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Wed, 2010-05-05 at 13:45 +0200, Pet wrote:
Hello!
I'm using Twisted 10.0 and as usually sometime print debug infos with myunicodestr.encode('UTF-8') which are saved to logfile, but since using twisted 10 I'm getting
UnicodeEncodeError: 'ascii' codec can't encode characters...
type(myunicodestr) returns <type 'unicode'>
What is the problem here?
This works fine for me (Twisted trunk): $ python2.5 -c "import sys; from twisted.python import log; \ log.startLogging(file('/tmp/log', 'w')); print \ u'\u1234'.encode('UTF-8')" $ cat /tmp/log 2010-05-05 08:48:40-0400 [-] Log opened. 2010-05-05 08:48:40-0400 [-] ሴ Can you include a minimal reproducing example?
On Wed, May 5, 2010 at 2:49 PM, Itamar Turner-Trauring <itamar@itamarst.org> wrote:
On Wed, 2010-05-05 at 13:45 +0200, Pet wrote:
Hello!
I'm using Twisted 10.0 and as usually sometime print debug infos with myunicodestr.encode('UTF-8') which are saved to logfile, but since using twisted 10 I'm getting
UnicodeEncodeError: 'ascii' codec can't encode characters...
type(myunicodestr) returns <type 'unicode'>
What is the problem here?
This works fine for me (Twisted trunk):
$ python2.5 -c "import sys; from twisted.python import log; \ log.startLogging(file('/tmp/log', 'w')); print \ u'\u1234'.encode('UTF-8')" $ cat /tmp/log 2010-05-05 08:48:40-0400 [-] Log opened. 2010-05-05 08:48:40-0400 [-] ሴ
Can you include a minimal reproducing example?
If I print as you do it works, but my string is fetched from database and only then it fails
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Wed, May 5, 2010 at 4:15 PM, Pet <petshmidt@googlemail.com> wrote:
On Wed, May 5, 2010 at 2:49 PM, Itamar Turner-Trauring <itamar@itamarst.org> wrote:
On Wed, 2010-05-05 at 13:45 +0200, Pet wrote:
Hello!
I'm using Twisted 10.0 and as usually sometime print debug infos with myunicodestr.encode('UTF-8') which are saved to logfile, but since using twisted 10 I'm getting
UnicodeEncodeError: 'ascii' codec can't encode characters...
type(myunicodestr) returns <type 'unicode'>
What is the problem here?
This works fine for me (Twisted trunk):
$ python2.5 -c "import sys; from twisted.python import log; \ log.startLogging(file('/tmp/log', 'w')); print \ u'\u1234'.encode('UTF-8')" $ cat /tmp/log 2010-05-05 08:48:40-0400 [-] Log opened. 2010-05-05 08:48:40-0400 [-] ሴ
Can you include a minimal reproducing example?
If I print as you do it works, but my string is fetched from database and only then it fails
It's pretty weird. I've send as parameter {'s': u'c\u0142a'} to twisted xml-rpc server after it was restarted and it has printed param['s'].encode('UTF-8') without errors. Immidiately after that I've send the same request again and it failed to print it. I've restarted the server again and at the first request it prints without errors, all other requests raise exceptions. So it has nothing to do with database.
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Wed, May 5, 2010 at 4:29 PM, Pet <petshmidt@googlemail.com> wrote:
On Wed, May 5, 2010 at 4:15 PM, Pet <petshmidt@googlemail.com> wrote:
On Wed, May 5, 2010 at 2:49 PM, Itamar Turner-Trauring <itamar@itamarst.org> wrote:
On Wed, 2010-05-05 at 13:45 +0200, Pet wrote:
Hello!
I'm using Twisted 10.0 and as usually sometime print debug infos with myunicodestr.encode('UTF-8') which are saved to logfile, but since using twisted 10 I'm getting
UnicodeEncodeError: 'ascii' codec can't encode characters...
type(myunicodestr) returns <type 'unicode'>
What is the problem here?
This works fine for me (Twisted trunk):
$ python2.5 -c "import sys; from twisted.python import log; \ log.startLogging(file('/tmp/log', 'w')); print \ u'\u1234'.encode('UTF-8')" $ cat /tmp/log 2010-05-05 08:48:40-0400 [-] Log opened. 2010-05-05 08:48:40-0400 [-] ሴ
Can you include a minimal reproducing example?
If I print as you do it works, but my string is fetched from database and only then it fails
It's pretty weird. I've send as parameter {'s': u'c\u0142a'} to twisted xml-rpc server after it was restarted and it has printed param['s'].encode('UTF-8') without errors. Immidiately after that I've send the same request again and it failed to print it. I've restarted the server again and at the first request it prints without errors, all other requests raise exceptions. So it has nothing to do with database.
Now, I'm getting Exception with File "/usr/local/tw10/lib/python2.5/site-packages/Twisted-10.0.0-py2.5-linux-x86_64.egg/twisted/python/log.py", line 555, in write d = (self.buf + data).split('\n') exceptions.UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 4: ordinal not in range(128)
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Wed, 2010-05-05 at 16:47 +0200, Pet wrote:
Now, I'm getting Exception with
File "/usr/local/tw10/lib/python2.5/site-packages/Twisted-10.0.0-py2.5-linux-x86_64.egg/twisted/python/log.py", line 555, in write d = (self.buf + data).split('\n') exceptions.UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 4: ordinal not in range(128)
Are you logging/printing unencoded unicode strings (i.e. type(s) == unicode)? Twisted does not support that.
On Wed, May 5, 2010 at 4:54 PM, Itamar Turner-Trauring <itamar@itamarst.org> wrote:
On Wed, 2010-05-05 at 16:47 +0200, Pet wrote:
Now, I'm getting Exception with
File "/usr/local/tw10/lib/python2.5/site-packages/Twisted-10.0.0-py2.5-linux-x86_64.egg/twisted/python/log.py", line 555, in write d = (self.buf + data).split('\n') exceptions.UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 4: ordinal not in range(128)
Are you logging/printing unencoded unicode strings (i.e. type(s) == unicode)? Twisted does not support that.
No, this exception occurs if I do print myunicodestring.encode('UTF-8') As I said before, it doesn't happen at first request after server restart.
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
participants (4)
-
Itamar Turner-Trauring -
Maarten ter Huurne -
Pet -
Phil Mayers