Re: [Twisted-Python] Problems with inlineCallback, Deferred, yield and Python 3 in buildbot

26 Jan 2017


      On Tue, Jan 24, 2017 at 12:48 PM, Craig Rodrigues 
wrote:
...
On Mon, Jan 23, 2017 at 5:10 PM, Jean-Paul Calderone <
exarkun@twistedmatrix.com> wrote:
...
On Mon, Jan 23, 2017 at 7:08 PM, Craig Rodrigues 
wrote:
...
I did some more debugging.  The callstack is quite deep. :/
I used this command to trigger the error:
trial  buildbot.test.unit.test_process_buildrequestdistributor.Tes
tMaybeStartBuilds.test_slow_db
I found in this function in buildbot's BuildRequestsEndpoint.get()
function ( https://github.com/buildbot/buildbot/blob/master/master/buil
dbot/data/buildrequests.py#L146 ) there is this line:
*        defer.returnValue(            [(yield self.db2data(br)) for br
in buildrequests])*
On Python 2, this line returns:
  an object list
  each list entry is a dictionary of name/value pairs that looks like:
[{'buildrequestid': 10, 'complete': False, 'waited_for': False,
'claimed_at': None, 'results': -1, 'claimed': False, 'buildsetid': 11,
'complete_at': None, 'submitted_at': datetime.datetime(1970, 1, 2, 12, 6,
40, tzinfo=tzutc()), 'builderid': 77, 'claimed_by_masterid': None,
'priority': 0}, {'buildrequestid': 11, 'complete': False, 'waited_for':
False, 'claimed_at': None, 'results': -1, 'claimed': False, 'buildsetid':
11, 'complete_at': None, 'submitted_at': datetime.datetime(1970, 1, 2, 13,
30, tzinfo=tzutc()), 'builderid': 77, 'claimed_by_masterid': None,
'priority': 0}]
On Python 3, this returns:
  an object 
Yep.
On Python 2, [(yield self.db2data(br)) for br in buildrequests] is a list
comprehension.  It will have len(buildrequests) elements and each will be
the value sent back in to the generator via the yield expression.
On Python 3, the same expression is a list of one element which is a
generator expression.
This form is much clearer, I think, and behaves as intended on both
versions of Python:
results = []
    for br in buildrequests:
        results.append((yield self.db2data(br)))
    defer.returnValue(results)
Wow, thanks!  That fixed it.  I submitted those fixes upstream to buildbot.
I'm not so familiar with generator expressions and Deferred's so have
been playing
around with things to try to understand.
I went back to the problem code, and changed:
*        defer.returnValue(            [(yield self.db2data(br)) for br in
buildrequests])*
to:
*       defer.returnValue(*
*           list([**(yield self.db2data(br)) for br in buildrequests])*
and ran the code under Python 3 and got a return value of:
[ Deferred ]
instead of:
           [ {......} ]
where the Deferred.results value was the dictionary.
How does Python 2 directly go to returning the dictionary out of the
Deferred,
while Python 3 returns the Deferred inside the list?  self.db2data()
returns a Deferred.
Thanks.
--
Craig
On the python-dev mailing list, Joe Jevnik gave a really excellent
explanation of what
is going on here, down to the Python bytecode level:

https://mail.python.org/pipermail/python-dev/2017-January/147244.html


--
Craig