[Tutor] Threading

[Oleg Oltar]

Hi!

I need to open about 1200 urls from my database. And to check that those
urls are really exists.

I used urllib2.urlopen for it. But it's a little bit slow. I thought that
it's might be a good idea to do it in a threads. So it can add some
performance to my code.

Unfortunately I can't get started with the treading module.  There are no
simple examples in the python docs. Not sure how to start.


What I want to have:
for my list of urls I want to start few threads, each one of them should do
some validation.

My code without threads:

def queryDb(query, db_name = sys.argv[1]):
    db = MySQLdb.connect(db = db_name, read_default_file="~/.my.cnf")
    c = db.cursor()
    c.execute(query)
    return c


def openUrl(url):
    error = ''

    try:
        urllib2.urlopen(url)
    except (urllib2.URLError, urllib2.HTTPError,httplib.BadStatusLine),
error:
        pass

    return str(error)

def isValuePresent(dbField):
    if str(dbField).find('None') is not -1:
        return False
    else:
        return True

def displayResults(resultTable):
    for query in resultTable:
        print query

def processUrl(url, guid, parent_guid):


def testUrls():
    error = ''
    resultTable = []
    errorTable = ['None']
    products = queryDb(query = """SELECT url, guid, parent from product;""")
    i = 0

    for product in products.fetchall():

        url = product[0]
        guid = product[1]
        parent_guid = product[2]
        error_code = ''
        #resultTable = {}

        #print product[0]


        if isValuePresent(url) is False:
            #print "!!!!!", "this one is none", product[0]

            """Checking if this is a variant and it has parent with url"""
            if isValuePresent(parent_guid) is True:
                #print parent_guid, isValuePresent(parent_guid)
                for parent in queryDb(query="""SELECT url from product where
guid = '%s';""" %parent_guid).fetchall():
                    #print parent[0]
                    '''Checking if parent product contains url'''
                    if isValuePresent(parent[0]) is True:
                        url = guid + "the parent url is: " + parent[0]
                        error = openUrl(parent[0])
                    else:

                        error = 'The variant with guid ' + guid + ' and his
parent do not contain urls!'
            """So the product is parent, and it's expected that it's child
will contain guids"""
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/tutor/attachments/20080911/0b23aaa1/attachment.htm>