[pytest-dev] My gripes with pytest and tox

[Florian Bruhin]

Hi,

What's following here are a few thoughts about things I miss or
dislike about pytest, plugins I use, and tox. This is a list of things
I personally would like to get fixed during the sprint, but of course
that's not the only thing I want to work on, then I wouldn't need a
sprint with other people. ;)

Let me mention again this is my *personal* view of things. You might
disagree with some of it. That's okay. Let's not turn this into a
bike-shedding for now, though feel free to reply privately if needed.

This also is not a rant. It's just a reminder for myself and a todo
list. All of this software is awesome! Those are all minor annoyances.

Some of those items have issues open, some of them I haven't told
anyone about yet. A few of them might not be solvable at all.

I thought for some while whether to make this public or not, but
figured, hey, why not.

In no particular order:

pytest
    - doesn't have an easy way to do printf-style debugging
      (without -s)
    - prints the exception before all other report output instead of
      after
    - segfaults in my testsuite when using pdb (haven't investigated
      yet, pdb++ works)
    - blows up in weird ways with ImportErrors in plugins
    - too much boilerplate with parametrized fixtures
    - too much boilerplate with parametrized tests (what about keyword
      arguments?)
    - has a weirdly structured documentation
    - is inconsistent with types and attributes in all of its API
    - has an unintuitive warning system, for devs and users
    - doesn't work on Windows with colors (for me), prints black on
      black
    - hides python warnings

pytest-qt
    - doesn't have tests yet for its modeltest branch, which is why it
      isn't merged yet
    - doesn't have a way to check arguments of signals
    - doesn't work with PySide2 (which is in development)

pytest-bdd
    - doesn't have an easy way to setup/teardown things before/after
      all tests in a .feature file
    - doesn't have a way of parametrizing tests I can get in my head
    - doesn't support markers with arguments in .feature files
    - doesn't have gherkin reporting

    thinking about writing an alternative which integrates an existing
    python bdd framework (behave or lettuce) to profit from everything
    those offer

pytest-catchlog
    - is still not in pytest-dev or core or renamed to -logging
    - still doesn't have pytest-logging functionality integrated
    - has no easy way to check a list of log messages
    - has no way to fail tests on unexpected error logging messages

pytest-xdist
    - breaks my testsuite in ways I don't understand
    - doesn't seem to have an easy way for per-job fixtures

pytest-xvfb
    - is broken with Qt4

plugin ideas
    - good profiling of tests:
      What's slow in my testsuite?
      What's slow in my application?
    - new BDD plugin maybe, see above

tox
    - takes too long until my tests start
    - doesn't detect changes in requirements.txt files
    - doesn't support comments in deps= with newer pip's
    - is too verbose
      (too much noise with single-line output, needs -q)
    - is too quiet (with -v I want to see pip output)

Again - no offense to anyone, and I don't intend to start a
bike-shedding ;)

Looking forward to the sprint!

Florian

-- 
http://www.the-compiler.org | me at the-compiler.org (Mail/XMPP)
   GPG: 916E B0C8 FD55 A072 | http://the-compiler.org/pubkey.asc
         I love long mails! | http://email.is-not-s.ms/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/pytest-dev/attachments/20160610/4985f37b/attachment.sig>