Hello folks,

I recently stumbled upon twisted and was wondering if it could suit my needs. On one hand, I want to use python but on another hand there are all these scalability concerns with this language so, I though I would pick the brains of the community. So.. a flask based app would look something like this.

similar_types = ['foo', 'bar', 'baz']


def long_computation(rec_type):
     # some long computation
     return recs

@app.route('/fetch_similar_users/<user_id>'
def fetch_similar_users(user_id)
        r = json.loads(requests.get('url/to/fetch/%s'%user_id).text)
        recs = {}
        for stype in similar_types:
             recs[stype] = long_computation(rec_type)
       return recs

Now, I tried to "twistify" but it failed.

@defer.inlinecallbacks

def long_computation(rec_type):
     # some long computation
     defer.returnValue(recs)

@defer.inlinecallbacks

def fetch_data(user_id):

r = yieldjson.loads(requests.get('url/to/fetch/%s'%user_id).text)

defer.returnValue(r)

@defer.inlinecallbacks

def fetch_recs(user_id):

data = yield fetch_data(user_id)

recs = {}

for stype in similar_types:

d = defer.ToThread(fetch_data, *(stype))

rec = yield d

recs[stype] = rec

defer.returnValue(recs)

I wrapped all the above in twisted render_Get method.. but then I did a load test with locust (https://docs.locust.io/en/latest/what-is-locust.html) framework.

It choked. As the time progressed, the response time increased.

I am guessing, things are still blocking.

Can you please help me look into the right place. Why exactly am I seeing increase in response time as the time progresses. I am guessing things are still working in "blocking" fashion but i thought the above should run things in async.

Thanks