[Python-Dev] How far to go with user-friendliness

[Florian Bruhin]

* Ron Adam <ron3200 at gmail.com> [2015-07-20 12:57:08 -0400]:
> >It's "unsafe" because tests which:
> >
> >1) are using the assert_* methods of a mock, and
> >2) where the programmer did a typo (assert_called() instead of
> >    assert_called_with() for example)
> >
> >do silently pass.
> 
> And further down, you say...
> 
> >Compare it with the behavior of a normal object - if you call a method
> >which doesn't exist, it raises AttributeError.
> >
> >This isn't possible with Mock objects, as they are designed to support
> >*any*  call, and you can check the calls have happened after the fact.
> 
> 
> And the docs say...
> 
> """
> spec: This can be either a list of strings or an existing object (a class or
> instance) that acts as the specification for the mock object. If you pass in
> an object then a list of strings is formed by calling dir on the object
> (excluding unsupported magic attributes and methods). Accessing any
> attribute not in this list will raise an AttributeError.
> """
> 
> So calling a method that doesn't exist on a mocked object will raise an
> AttributeError if it is given a spec.
> 
> But if you don't give it a spec, then a mispelling of *any* method will pass
> silently.  So it's not a problem limited to "assert" methods.
> 
> It seems the obvious and best solution is to always use a spec.

I agree - I always use mocks with autospec/spec, and I recommend doing
that - I actually plan to write a plugin for pylint to enforce this.

Still mistyping the assert methods seems to be a common issue, since
(according to some other mail) a couple of bugs in OpenStack were
found this way.

But yeah - if your code under test has a typo, and you don't use
spec/autospec, you might not notice as well - though in my experience
you *usually* do, since the returned mock object doesn't behave in the
way you expect it to.

But yeah - always using (auto)spec is probably the best solution.

> >>> m.assert_me()
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "/media/hda2/home/ra/Dev/python-dev/python3.5/cpython-master/Lib/unittest/mock.py",
> line 583, in __getattr__
>     raise AttributeError(name)
> AttributeError: assert_me
> 
> 
> Why is AttributeError raised here?  Why are methods beginning with assert
> special?  (or "unsafe")

Some things I can think of:

- It's more likely that you use assert_called() instead of
  assert_called_with() accidentally than that you do a typo in your
  code under test.

- If you do a typo in your code under test, a linter is more likely to
  find it than with mocks, because of their nature.

- Other tests (even if they're manual ones) should probably discover
  the typo in your real code. The always-passing assert won't.

Florian

-- 
http://www.the-compiler.org | me at the-compiler.org (Mail/XMPP)
   GPG: 916E B0C8 FD55 A072 | http://the-compiler.org/pubkey.asc
         I love long mails! | http://email.is-not-s.ms/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20150720/d842bd7f/attachment.sig>