[pytest-dev] another "i have had it moment" and a stern request ro review our interaction with backward compatibility

[Zach Kanzler]

Hi, y'all!

I share a similar sentiment as Oliver and Sylvain — pytest has changed my
life so much for the better, and I don't have a difficult time showing
others how dead simple it is to get started. Like Sylvain said: it's a tad
more difficult to show the power of fixtures at first glance, but the mark
of a great tool is how it develops alongside one's understanding of it. And
also, I've bundled a bunch of great fixtures together in a mixin to make
DRF API testing sexy. It's been a breeze.

I would hope the ease of using pytest would extend to its own development.

I've put in my own fair share of workarounds and hackaroos — honestly, I
loved pytest so much I never stopped to consider whether the internals
might be disorganized; I just merrily went on my way. But hearing there may
not have been more formal design kinda fits the bill :P

I think Sylvain hit a great point: fixtures and parametrization are the
killer features to me as well. Though I might be inconvenienced if I have
to change a bunch of hacks just so I can upgrade to a new pytest version
which fixes a bug I experience, it's as I interpret you say, Ronny: having
a clean, minimal, hopefully-globally-consistent API is such a pleasure to
work with, and has the side benefit that conversations with it needn't be
hours-long. "Hmm, I have to do X. I thiiiiiink that would be *here*. Oh,
hell yeah! That's where it was." Not only do you spend less time battling,
but ya feel more powerful — when the limit's of pytest's default features
are reached, instead of wondering what testing compromise is reasonable,
one can immediately reach for a plugin touching deep internals.

On the volunteer contributions point: I don't feel so held back from
contributing now, after reading this email chain, but I used to hold back
from even considering pull requests, because the code seemed dense and
wandering and wavering at times... yet pytest works so wonderfully for me,
so I figured there must be a reason to the rhyme. I presumed there was a
grander design at play — one I might hinder by becoming a triage/review
burden.

The documentation *has* become fantastic. At times I've found examples of
flows I'd never even considered trying, which have become routine to me
now. The documentation has one advantage over releases: no one's losing
hair over publishing documentation. When it's possible to experiment — when
worries over the past don't consume the present, it's strange how things
improve. We've all had those friends who seem content to stay their same
miserable selves, year after year.

Some of us may have even had those friends who have a new "thing" every
time you see them. I'll hold off on examples there, 'cause I'm sure to
offend someone when it ain't the point :P "Backwards compatibility" is a
great ideal, but even Microsoft cut the cord at a point. It can't hinder
progress, lest a project stall for all eternity — not just due to appeasing
that ideal, but for your reasons, Ronny: it's demotivating. It seems to me
that forward, even breaking movement is the only way to reach eventual
consistency... so long as a "fuck you, old fogeys" doesn't cause all nodes
to disconnect— er, as long as a sudden breaking change with no recourse
causes all users to jump ship.

Anyway, I can't undersell how much pytest has changed my philosophy of
development. Because of pytest, I'm able to teach newer devs testing not as
"writing tests", but as "describing how things work, in Python". So little
needs explaining to kindle the spark.

I don't want pytest's spark to die out because 80 million use cases must be
appeased. The past is just a goodbye.

That's my two dollars
Zanch "they4kman" Kranzanzler

On Thu, Nov 8, 2018 at 3:56 AM Sylvain MARIE <sylvain.marie at se.com> wrote:

> Hi all
>
> I could not agree more with Oliver - I discovered pytest two years ago and
> it was really a "wow' moment - I quickly moved away from NoseTest :)
>
> Parameters and Fixtures are to me THE features that makes it so powerful
> and easy to use, and that enable anyone to invent new usage patterns that
> others did not even think about before. The issue for new users especially
> the ones coming from traditional unit testing (e.g. junit) is that it is
> quite difficult to understand how powerful they are at first glance.
>
> It is however very easy to "pre-bundle" usage patterns with more
> user-friendly names. That's what I did with pytest-cases and pytest-steps
> (with the help of the great 'decorator' lib to manipulate the signatures a
> bit). This de-coupling between a very compact and generic core (with the
> smallest possible amount of API and functionality to ensure
> maintainability), and multiple, composable, "usage-oriented" plugins, seems
> to me the best reasonable long-term choice. Of course that does not prevent
> doing a bit of cleaning and design improvement in the internals, to improve
> its compactness.
>
> Best;
>
> --
> Sylvain
>
>
> De : pytest-dev [mailto:pytest-dev-bounces+sylvain.marie=se.com at python.org]
> De la part de Oliver Bestwalter
> Envoyé : mercredi 7 novembre 2018 22:34
> À : Bruno Oliveira <nicoddemus at gmail.com>
> Cc : pytest-dev at python.org
> Objet : Re: [pytest-dev] another "i have had it moment" and a stern
> request ro review our interaction with backward compatibility
>
> [External email: Use caution with links and attachments]
> ________________________________________
>
> Hi Ronny,
>
> I'd like to add my 2 cents from the perspective of an enthusiastic pytest
> user, who dabbles in a lot of projects using pytest and who occasionally
> teaches it:
>
> I use pytest since 2010 and can still remember a time when I was able to
> trigger the dreaded <INTERNALERROR> quite regularly and it was often really
> hard for me to figure out what went wrong, leading to arcane workarounds in
> my test suites.
>
> Since I am a bit involved in the project and pay more attention, I
> remember
>
> * one incident where I triggered an INTERNAL error during normal usage in
> a fresh release and the problem was easy to spot and fix (
> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fpytest-dev%2Fpytest%2Fissues%2F2788&data=02%7C01%7Csylvain.marie%40se.com%7C59e7ab289a2a4698bc9208d644f8fa26%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C636772233479207352&sdata=%2BuNmxKZxmKNgLJ9eWFPvzL1C2fZpES9MGxYEiDp11ak%3D&reserved=0
> ).
> * one breaking change that was also easy to fix: the change in the logging
> behaviour and that was also perfectly o.k. because bottom line is that the
> logging behaviour overall now is far better because of this
> * one hard to reproduce race condition due to the introduction of tmp_path
> (thanks for that btw), which also was fixed faster than I was even able to
> find some time to have a closer look (
> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fpytest-dev%2Fpytest%2Fissues%2F4181&data=02%7C01%7Csylvain.marie%40se.com%7C59e7ab289a2a4698bc9208d644f8fa26%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C636772233479217361&sdata=9wKOnAy1azrWVCvHxHWOjN4SETPmZD3XGvQk%2BpbOXhs%3D&reserved=0
> ).
>
> All in all pretty painless and easy to work around and usually fixed
> really quickly.
>
> In my experience the subset of functionality/plugins that by far the most
> test suites use are rock solid, have a good API and hardly ever break.
> Also: the quality of the "user experience" and the documentation has
> improved vastly in the last few years.
>
> Just today I had the chance to pair program some tests to introduce a
> colleague to pytest. Seeing the expression of delight in their face, when
> they realized how easy it is to get started and when they started to
> comprehend the power of fixtures and parametrization is priceless.
>
> Long story short: you folks are amazing and pytest is something to be
> extremely proud of despite all the pain that is always involved when trying
> to pay off technical debt without breaking the word.
>
> The release automation that is in place now makes more frequent releases
> easier and from my own experience I agree with Brunos suggestion to make
> even more frequent releases that contain smaller changesets and also to not
> shy away from more frequent major releases, if that makes the transition
> easier.
>
> Thanks that you care and thanks for all your work
> Oliver
>
> On Wed, 7 Nov 2018 at 20:56 Bruno Oliveira <mailto:nicoddemus at gmail.com>
> wrote:
> Hi Ronny,
> On Wed, Nov 7, 2018 at 7:12 AM Ronny Pfannschmidt <mailto:
> opensource at ronnypfannschmidt.de> wrote:
>
> ...
>
>
> This is an accumulative cost in many ways - in particular for a project
> driven primarily by volunteers - every hour we sacrafice to this altar
> of technical debt is an hour lost for improving the project and/or
> working on actual features.
>
> I feel your frustration. I agree that the pytest codebase needs some
> refactorings in order for it to be maintainable and allow us to move
> forward.
>
> From my perspective this issue directly grows out of driving a large
> part of pytests development by examples and tests, but precluding
> stepping back and doing actual design.
>
> The history of marks makes a perfect example of such an horror storry.
> Each iteration adding a new minimal element while adding a huge factor
> to the structural error as it grew.
>
> What I think often happens is that we can't foresee that a minimal
> increment might lead to a large technical debt in the future. Once we put
> something in the open, we try very hard to not change that behavior (even
> if considered incorrect now), which is the main point of your email.
>
> I really want to hammer down there that "minimal" is very different
> from "minimal viable" - leaving design uncontested for too long is an
> ramp up for unsustainable technical debt.
>
> Its aslo critical to note that for **me** there is a inherent
> dishonesty in trying to stay backward compatible and not putting
> forward a design and development process that deeply supports it -
> right now we can observe a state of fragility from pytest where it
> feels like every feature release triggers some regression - while at
> the same time we keep and keep shipping features that are structurally
> and fundamentally broken since years (yield tests, config init, ...).
>
> I agree about the fact that every feature release we end up breaking
> something unintentionally. Of course that sometimes will happen, but I feel
> that happens more often in the murky areas of the code which have grown
> organically over the years.
>
> This setup, which gives me the impression it is "designed" to make the
> project fail its user (aka its broken to begin with and now it will
> break some more), is a massive emotional drain. It painfully twists
> around the sense of responsibility simply due to the perceived inherent
> base-level of failure - and **I** want to do and feel better.
>
> However with the current flow of releases and with our backward
> compatibility policies in place i am under a very general impression
> that i cant really do anything about it, and even if i try it would
> drag on for years to generate even basic positive results - this
> impression is a killer to my motivation - as it **feels** like it would
> take years to even get a initial iteration out to the users - and the
> process of fixing the issues in general would drag on over not only
> years - but decades.
>
> A timeflow like that would commpletely disconnect me from perceptions
> of reward or archivement  - in turn making it even less likely to
> succeed or even start.
>
> About our backward compatibility policies, we don't have any minimal time
> restriction for major releases, we only state that we will issue
> deprecation warnings for two feature releases before actually
> changing/removing a feature.
>
> Having said that, I see that it is possible for us to have major releases
> more often (a few per year even). As we have learned already, frequent
> releases which cause few incompatibilities are preferred over infrequent
> releases which cause a lot of incompatibilities, as it will affect less
> users and makes it easy to pin point problems.
>
> My gut feeling is that those backward incompatible releases won't be that
> bad in the end: we only change little used features or porting the code to
> the new way of doing things is easy to apply to existing code. Of course
> some friction always happen.
>
> Dragging deprecated features over many releases, specially if they get in
> the way of new implementations, is a certain way to needlessly increase the
> burden of us maintainers, as you point out.
>
> This is a open source project that volunteer driven - fixing deeper
> issues should connect to feeling an archivement - but with out current
> setup what seems to be in for **me** is more like dread, fear and
> depression - a general defeat.
>
> Thats not something i want to allow anymore.
>
> **I** want to feel good about my work on pytest.
> **I** want to feel the archivements i have on pytest.
>
> and because of that
>
> **I** have to make sure the project can support that.
>
> So **I** want to invite all of you,
>
>
> Lets make pytest a project again where
>
> * we claim suport for backward compatibility and our actions and
> results show for it
> * we can enjoy the act of adding new features and enhancing the
> ecosystem
> * we can have better and quicker feedback, not just from the fabulous
> people that work on it - but also the users.
>
>
> **I** strongly beleive that this is doable, but it will require cutting
> some bad ends.
>
> We need to enhance layering, it doesnt have to be perfect but we do
> have to be able to keep things appart sanely and we need to have
> objects in valid states per default (invalid states should only be
> possible deliberately, not accidentially)
>
> We need to talk/write about design more - the difference between
> minimal and viable after all only shows itself in the heat and fruction
> of different perspectives and oppinions clashing.
>
> We need a conversation and interaction with our advanced users, so they
> dont end up on old and dead pytest versions (like pypy for example).
>
> We need a pervasive chain reason to bring trough the period of
> papercuts where we do shift around layering so the ecosystem can keep
> up (saving the structural integrity of pytest at the expense of
> destroying the ecosystem would be a dissaster).
>
> I agree with the points above Ronny, let's make it happen. :)
>
> Cheers,
> Bruno.
> _______________________________________________
> pytest-dev mailing list
> mailto:pytest-dev at python.org
>
> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.python.org%2Fmailman%2Flistinfo%2Fpytest-dev&data=02%7C01%7Csylvain.marie%40se.com%7C59e7ab289a2a4698bc9208d644f8fa26%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C636772233479217361&sdata=dHBvhBLA60OOzRjx3TI4bHZFtLxmdhwactGyJehyWe0%3D&reserved=0
>
> ______________________________________________________________________
> This email has been scanned by the Symantec Email Security.cloud service.
> ______________________________________________________________________
> _______________________________________________
> pytest-dev mailing list
> pytest-dev at python.org
> https://mail.python.org/mailman/listinfo/pytest-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pytest-dev/attachments/20181114/741a8138/attachment-0001.html>