[pytest-dev] tox-2.0.0: plugin support, platforms, env isolation

[Florian Bruhin]

* Florian Schulze <mail at florian-schulze.net> [2015-05-21 10:16:03 +0200]:
> On 21 May 2015, at 9:17, raphael at hackebrot.de wrote:
> 
> >Hi testing folks,
> >
> >I have been running into a slightly confusing problem on my Cookiecutter
> >dev env with the latest tox version and click.
> >
> >The changes regarding ENV isolation happen to break our tox suite on
> >Python3 envs because Python is making a guess on the preferred encoding
> >which happens to be 'ascii' which in turn causes a 'RuntimeError' in
> >click.
> >
> >Please see the according PR and the gist included.
> >https://github.com/audreyr/cookiecutter/pull/453
> >
> >Given that this is a deliberate change in tox 2.0, I was thinking whether
> >we should spread the word (possibly reddit, twitter?!) so other projects
> >affected by this change are aware and update their tox config accordingly.
> >
> >Please let me know your thoughts and I'll be happy to contribute.
> 
> I just hit the same issue, but for me it happens during installation because
> of unicode in the long_description. With passenv = LC_ALL, LANG it works.
> Question is, should these envvars be added by default or does everyone have
> to update their tox.ini?

The same thing broke my pep257 checks (docstring convention checker)
as well as the pyflakes checks for pytest itself:

https://bitbucket.org/pytest-dev/pytest/issue/747/flakes-test-fail

I'm still not sure if it's a good idea to include LC_ALL/LANG by
default though, even with stuff breaking. As mentioned by someone else
in some mailinglist thread earlier, it's nice to know when tests
depend on those variables.

After all, there still could be people running the tests with a broken
system locale and no UTF-8 support (I've gotten bugreports because of
that!).

In the usual cases, I think the code should be fixed instead:

- pep257 and flake8 should use tokenize.detect_encoding[1] /
  tokenize.open (or equivalent code for < 3.2) to honour PEP 263
  encoding markers[2].

  (I got bitten by this when trying to run pep257 on Python files
  containing UTF-8 Unicode chars on Windows)

- Code reading/writing files only used by itself should always specify
  an explicit encoding, probably  encoding='utf-8'  [3]

  (I got bitten by this by writing a cookie file with Unicode in
  cookies without explicit encoding - it worked on Linux but crashed
  on Windows because it used latin1)

- Code writing files for other applications should either know the
  encoding the other part needs, or fallback to an explicit default
  (i.e. utf-8).

- Code reading files from other applications should either know the
  encoding the other part writes, or find out by some kind of
  meta-data, or do a more elaborated guess (e.g. using chardet[4]).

[1] https://docs.python.org/3.4/library/tokenize.html#tokenize.detect_encoding
[2] https://www.python.org/dev/peps/pep-0263/
[3] http://utf8everywhere.org/
[4] https://pypi.python.org/pypi/chardet

Does this sound reasonable? I know it's the much more painful way
compared to basically ignoring those issues (by passing the locale
through, which is most likely UTF8-aware on Linux). But I think it's
better to fix the real issues wherever possible.

Florian

-- 
http://www.the-compiler.org | me at the-compiler.org (Mail/XMPP)
   GPG: 916E B0C8 FD55 A072 | http://the-compiler.org/pubkey.asc
         I love long mails! | http://email.is-not-s.ms/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/pytest-dev/attachments/20150521/7bdb5888/attachment.sig>