Mailman 3 April 2018 - pytest-dev

request to add pytest-echo to pytest-dev plugins
by Stefano May 2, 2018

May 2, 2018

Hi, I did a very small plugin (pytest-echo), let me know if it can be added to the pytest-dev organization. Code can be found at https://github.com/saxix/pytest-echo Best, Stefano

3 7

Creating a complex PyTest program, wondering about the best way to handle collection errors, etc.
by Nicholas Williams April 28, 2018

April 28, 2018

Hello, all. This is my first post to the mailing list, FWIW. I've been a PyTest user for some years, but now I'm working on writing a rather complex PyTest plugin to facilitate testing our service oriented architecture at $work. The purpose of the plugin is to collect additional tests (for which we are using `pytest_collection_modifyitems`) from a specialized file format that PyTest wouldn't normally understand. I've got it working beautifully, but I would love to hammer out some finer details to really make it a clean-running plugin. With that in mind, I'm looking for some suggestions / best practices / advice here on how best to handle these situations: Collection Errors The author of one of these specialized tests, which we call "test plans," could possibly do a few things wrong that cause more than just making their test fail. For example, they could get their file syntax incorrect, which results in a parsing error. At the moment, I catch the parsing error (we're using PyParsing to parse our test plans) and raise a different exception about the problem (and I've tried AssertionError), but everything I do results in a stack trace dumped with "INTERNALERROR>" and the immediate, abnormal termination of the PyTest process. This contrasts to, for an analog example, a syntax error in a regular Python test file, which instead results in a "ERROR collecting" output, while all other unaffected tests still run and the PyTest process terminates normally. What I'm looking for here is the "right way" to make my plugin report an "ERROR collecting" instead of causing an "INTERNALERROR>" when such a problem occurs collecting these specialized tests. Improving the Test Error Report When tests have errors or failures, PyTest prints out a report of those errors, and with each one includes an exhaustive stack trace. For our specialized tests, with some clever manipulation of the test data added to `items` in `pytest_collection_modifyitems`, I've managed to at least get it to display a meaningful name for the test that failed so that it's at least _possible_ for the developer to figure out which test, in which test file, failed. However, almost the entire stack trace is meaningless and full of plugin code instead of containing information useful to debugging the test. Additionally, I would _love_ to be able to add the actual test file and line number for the failure to the report. Would this be possible? Or, at least, would it be possible to filter the plugin code out of the stack trace so that it doesn't distract the developer from simply viewing the assertion error details? What approach should I take to achieve this? Thanks in advance for any help you can provide, Nick

3 9

pytest 3.5.1 released
by Bruno Oliveira April 24, 2018

April 24, 2018

pytest 3.5.1 has just been released to PyPI. This is a bug-fix release, being a drop-in replacement. To upgrade:: pip install --upgrade pytest The full changelog is available at http://doc.pytest.org/en/latest/changelog.html. Thanks to all who contributed to this release, among them: * Brian Maissy * Bruno Oliveira * Darren Burns * David Chudzicki * Floris Bruynooghe * Holger Kohr * Irmen de Jong * Jeffrey Rackauckas * Rachel Kogan * Ronny Pfannschmidt * Stefan Scherfke * Tim Strazny * Семён Марьясин Happy testing, The pytest Development Team

1 0

new pytest-dev plugin proposal
by davide moro April 23, 2018

April 23, 2018

Hi, I asked permission to my company for transferring the ownership of github repositories that might be interesting for the testing community to the pytest-dev organization and they said yes. Talking before with Bruno we thought that the following one might be a candidate: * https://github.com/tierratelematics/pytest-play There is a a cookiecutter scaffolding tool based on pytest providing a working selenium setup, examples with pytest-bdd (it is not a pytest plugin): * https://github.com/tierratelematics/cookiecutter-qa Let me know which of them could be added to the pytest-dev organization. Thanks. Bests, davide -- Linkedin: https://it.linkedin.com/in/davidemoro82 Twitter: https://twitter.com/davidemoro Blog: http://davidemoro.blogspot.it/

3 5

Adding plugins to pytest-dev
by Floris Bruynooghe April 21, 2018

April 21, 2018

Hi all, While just looking at https://github.com/pytest-dev/pytest/blob/master/CONTRIBUTING.rst#submittin… I had some observations: 1. It says one of the objectives of having plugins under pytest-dev is "Sharing some of the maintenance responsibility (in case a maintainer no longer wishes to maintain a plugin)". I found this somewhat surprisingly worded. I thought it was more "protect the pytest community in case a maintainer goes AWOL". I think that's a significant difference, maintainers aren't supposed to think they can move the plugin there if they no longer feel like actively maintaining it. 2. The other thing this says is "We recommend that each plugin has at least three people who have the right to release to PyPI." I think that's somewhat light and does not ensure a package can be released if a maintainer does go AWOL. Could we change that to something like "each plugin must have at least three people from https://github.com/orgs/pytest-dev/teams/core/members who have the right to release to PyPI" [0]? Practically we should probably include a list of pypi usernames to that paragraph since there's not mapping otherwise. Any opinions on these? Cheers, Floris [0] Bonus points for whoever contributes a teams functionality to warehouse!

3 7

Unsubscribing from pytest-dev/pytest notifications
by Florian Bruhin April 19, 2018

April 19, 2018

Hi, (I accidentally sent this to pytest-dev(a)pytest.org rather than @python.org originally. It didn't bounce, but I have no idea where it went :D) this is a difficult step to take, but during three sleepless nights because of health issues last week (don't worry, it's getting better now) I realized a lot of things (also see this thread on Twitter: [1]) [1] https://twitter.com/the_compiler/status/984356385214394368 I really wish GitHub had a way to subscribe to issue labels, but until that happens, I need to unsubscribe from pytest-dev/pytest (which is what I just did) - I'm spending too much time working on things in pytest I'm somewhat interested in, but not really passionate about. This is probably best explained with some back story: I got into pytest because I started using it with qutebrowser[2] (which is still my main OSS project), started contributing on plugins (mostly pytest-qt) because I was missing some features, and got into the pytest community. Somehow I ended up first getting regarded as a contributor, later as a core contributor (or maintainer) - while I still feel like I have no idea of pytest's code base :D [2] https://www.qutebrowser.org/ There's a list of things I'm still very much interested in - if something happens in those areas, please don't hesitate to contact/mention/request a review from me, I'd very much like to hear about those: - Giving trainings and talks (probably what I liked doing most so far!) - Anything related to user-facing changes: The API, output, etc. I am still an avid user of pytest, and thanks to trainings and other involvement feel like I have a picture of what other users want, too. - The community, which is awesome! I'll still stay on this mailinglist and the IRC channel, and I'd very much like to go to another development sprint in the future. - Testing (pre-)releases - Some plugins, especially pytest-qt - Any code I've written Things I'm explicitly not interested in (and where I never really did anything anyways): - Anything internal and complex in the codebase: Markers, fixtures, pytest.code, capturing, etc. etc. - you get the idea. I'd like to keep my maintainer/commit/whatever flag, and I might revisit this in 1-2 years, where I might have more time on my hand for open-source work. If there's consensus that I should drop it, I'm not going to say no either, though. (If you want to answer something along the lines of "get well soon" - those replies are very much appreciated, but please consider sending them to me directly instead of the mailinglist, to cut down "noise" for others) Florian -- https://www.qutebrowser.org | me(a)the-compiler.org (Mail/XMPP) GPG: 916E B0C8 FD55 A072 | https://the-compiler.org/pubkey.asc I love long mails! | https://email.is-not-s.ms/

3 2

Approval testing plugin for pytest
by Vasily Kuznetsov April 18, 2018

April 18, 2018

Hi everyone! It would be nice to have an approval testing <http://approvaltests.com/> plugin for pytest, but it seems there's no perfect solution at the moment. A quick google search gave me this <https://github.com/approvals/ApprovalTests.Python> and this <https://gist.github.com/npryce/9c4dc2e0ea7eaf6e27fc>, but that's kind of far from ideal. What I would like to have is the ability to mark a test as approval test and then the framework would compare the return value of the test function to the approved output, produce error diffs, allow approving new output, etc. It would also play nice with fixtures, parametrization and other pytest features so that a separate approved result would be generated for each test item, stored in a file with some useful name and all of this would be transparent for the developer. A diffing tool for approving the outputs could be specified via configuration parameter or an environment variable. Does this sounds like something that you have seen? If not, I will probably try to build it (most likely from scratch, since not much of the functionality of ApprovalTest.Python would be useful here and I feel like I would struggle against their approval naming code) and would appreciate any advice. Cheers, Vasily

2 2

preparing a set of breaking changes
by RonnyPfannschmidt April 11, 2018

April 11, 2018

Hi everyone, i wold like to get started with a long needed spring cleaning of things that are unfixable stange/broken since dozens of years * removal of yield tests, ever since collection and test running are no longer connected, their setupstate has been fundamentally broken for anything but test so simple, one should use parametrize * removal of the pytest namespace hook and setting up the pytest config in a way that is actially correct on xdist (currently pytest-xdist is unable to create config objects on workers in a consistent manner, and we leak data in strange ways * removal of the dozns and dozens of compat properties/hackish aliases across the codebase -> getting rid of customclasses on collectors * after the mark merge: introduce an actual FunctionDefinition node that's visible in the collection tree and ensure metafunc uses that as base for parameterization, * detangle node construction - the amount of spaghetti, in particular wrt sessions, fs collectors & co has an unbearable amount of stones in it for unit and functional testing -- Ronny

5 5

Printing plugin help and skip tests
by Ringo De Smet April 5, 2018

April 5, 2018

Hello, I'm working on a custom (private) plugin vary tailored to our product. The plugin adapts its feature set based on the abilities of the internal server it connects to. I was thinking of implementing an option in my plugin, e.g. `--show-our-superduper-help`. This could go out to the server, fetch it's config and print the abilities which can be used in the test suite for that server. I looked at how the regular `--help` is implemented: https://github.com/pytest-dev/pytest/blob/71a7b3c062d51c61ee8b1ac479b1260d5… It uses the HelpAction which raises a PrintHelp exception to shortcut on running the test suite. Not really fond of implementing something similar. How could I get a similar behaviour for only printing the (server specific) config of my plugin? Thanks, *Ringo De Smet* ringo.de.smet(a)ontoforce.com

2 1