UnicodeEncodeError when not running script from IDE
magpettersson at gmail.com
Tue Feb 12 16:29:00 CET 2013
> Are you sure you are writing the same data? That would mean that pydev
> changes the default encoding -- which is evil.
> A portable approach would be to use codecs.open() or io.open() instead of
> the built-in:
> import io
> with io.open(filepath, "a") as f:
> io.open() uses UTF-8 by default, but you can specify other encodings with
> io.open(filepath, mode, encoding=whatever).
Interesting. Pydev must be doing something behind the scenes because when i changed open() to io.open() i get error inside of eclipse now:
File "C:\python27\lib\encodings\cp1252.py", line 19, in encode
UnicodeEncodeError: 'charmap' codec can't encode character u'\u53c8' in position 32: character maps to <undefined>
io.open(filepath, "a", encoding="UTF-8") as f:
Then it works in eclipse. But I seem to be having an encoding problem all over the place that works in eclipse but dosnt work outside of eclipse pydev.
Here is the flow of my data, im terrible at using unicode/encode/decode so could use some help here:
# self.kanji.text() comes from a kanji letter written into a pyqt4 QLineEdit
kanji = unicode(self.kanji.text())
card = kanji_anki.scrapeKanji(kanji,tags)
def scrapeKanji(kanji, tags="", onlymeaning=False):
baseurl = unicode("http://www.romajidesu.com/kanji/")
url = unicode(baseurl+kanji)
#test to write out url to disk, works outside of eclipse now
#getting webpage works fine in eclipse, prints "Oh no..." in terminal
page = urllib2.urlopen(url)
print "OH no website dont work"
#Code that does some scraping and returns a string containing kanji letters
with io.open(filepath, "a") as f:
for card in cardlist:
More information about the Python-list