On Jan 24, 2017, at 12:48 PM, Craig Rodrigues <rodrigc@crodrigues.org> wrote:

On Mon, Jan 23, 2017 at 5:10 PM, Jean-Paul Calderone <exarkun@twistedmatrix.com> wrote:

On Mon, Jan 23, 2017 at 7:08 PM, Craig Rodrigues <rodrigc@crodrigues.org> wrote:

I did some more debugging.  The callstack is quite deep. :/
I used this command to trigger the error:

trial  buildbot.test.unit.test_process_buildrequestdistributor.TestMaybeStartBuilds.test_slow_db

I found in this function in buildbot's BuildRequestsEndpoint.get() function ( https://github.com/buildbot/buildbot/blob/master/master/buildbot/data/buildrequests.py#L146 ) there is this line:

        defer.returnValue(
            [(yield self.db2data(br)) for br in buildrequests])

On Python 2, this line returns:

  an object list

  each list entry is a dictionary of name/value pairs that looks like:

[{'buildrequestid': 10, 'complete': False, 'waited_for': False, 'claimed_at': None, 'results': -1, 'claimed': False, 'buildsetid': 11, 'complete_at': None, 'submitted_at': datetime.datetime(1970, 1, 2, 12, 6, 40, tzinfo=tzutc()), 'builderid': 77, 'claimed_by_masterid': None, 'priority': 0}, {'buildrequestid': 11, 'complete': False, 'waited_for': False, 'claimed_at': None, 'results': -1, 'claimed': False, 'buildsetid': 11, 'complete_at': None, 'submitted_at': datetime.datetime(1970, 1, 2, 13, 30, tzinfo=tzutc()), 'builderid': 77, 'claimed_by_masterid': None, 'priority': 0}]

On Python 3, this returns:

  an object <generator object BuildRequestsEndpoint.get.<locals>.<listcomp>

  of type <class 'generator'>

Yep.

On Python 2, [(yield self.db2data(br)) for br in buildrequests] is a list comprehension.  It will have len(buildrequests) elements and each will be the value sent back in to the generator via the yield expression.

On Python 3, the same expression is a list of one element which is a generator expression.

This form is much clearer, I think, and behaves as intended on both versions of Python:

    results = []

    for br in buildrequests:

        results.append((yield self.db2data(br)))

    defer.returnValue(results)

Wow, thanks!  That fixed it.  I submitted those fixes upstream to buildbot.

 I'm not so familiar with generator expressions and Deferred's so have been playing

around with things to try to understand.

I went back to the problem code, and changed:

        defer.returnValue(
            [(yield self.db2data(br)) for br in buildrequests])

to:

       defer.returnValue(

           list([(yield self.db2data(br)) for br in buildrequests])

and ran the code under Python 3 and got a return value of:

           [ Deferred ]

instead of:

           [ {......} ]

where the Deferred.results value was the dictionary.

How does Python 2 directly go to returning the dictionary out of the Deferred,

while Python 3 returns the Deferred inside the list?  self.db2data() returns a Deferred.