using python to post data to a form

[Karim]

On 04/04/2011 01:01 PM, Corey Richardson wrote:
> On 04/04/2011 01:36 AM, Littlefield, Tyler wrote:
>> Hello:
>> I have some data that needs to be fed through a html form to get
>> validated and processed and the like. How can I use python to send data
>> through that form, given a specific url? the form says it uses post, but
>> I"m not really sure what the difference is. would it just be:
>> http://mysite.com/bla.php?foo=bar&bar=foo?
>> If so, how do I do that with python?
>>
> import urllib
> import urllib2
>
> url = "http://www.foo.com/"
> data = {"name": "Guido", "status": "BDFL"}
>
> data = urllib.urlencode(data)
> request = urllib2.Request(url, data)
> response = urllib2.urlopen(request)
>
> page = response.read()
>
> So yeah, passing in a Request object to urlopen that has some
> urlencode'ed data in it.

Real life example:
I query for bugs:

def make_form_data(query=None):
     """Factory function to create a post form query to submit on a html 
webpage.

     @param query - the query string from the existing query list of the 
query webpage.
     """
     return {
        'init'           : EMPTY,
        'NextForm'       : EMPTY,
        'LastForm'       : FORM_QUERY,
        'ACTION'         : ACTION,
        'class'          : CLASS,
        'personalQuery'  : query,
        'sharedQuery'    : EMPTY,
        '.cgifields'     : CONFIG_QUERY,
        '.cgifields'     : SHARED_QUERY
        }

def authentication_setup(username=None, password=None, url=None):
     """Setup an authentication for a super-url (root url) on a given 
securised web server.

     @param username  - String
     @param password  - String
     @param url       - String
     """
     # Password Manager creation
     pwd_manager = urllib2.HTTPPasswordMgrWithDefaultRealm()

     # As we set the first parameter to None
     # the Password Manager will always use
     # the same combination username/password
     # for the urls for which 'url' is a super-url.
     pwd_manager.add_password(None, url, username, password)

     # Authentication Handler creation from the Password Manager.
     auth_handler = urllib2.HTTPBasicAuthHandler(pwd_manager)

     # Opener creation from the Authentication Handler.
     opener = urllib2.build_opener(auth_handler)

     # Tous les appels a urllib2.urlopen vont maintenant utiliser le handler
     # Ne pas mettre le protocole l'URL, ou
     # HTTPPasswordMgrWithDefaultRealm sera perturbe.
     # Vous devez (bien sur) l'utiliser quand vous recuperez la page.
     urllib2.install_opener(opener)
     # l'authentification est maintenant geree automatiquement pour nous

def post_request(url=None, data=None, headers=None):
     """Post a request form on a given url web server.

     @param url       - String
     @param data      - Dictionnary
     @param headers   - Dictionnary

     @return response The web page (file) object.
     """
     if headers is None:
         headers = {
                   'User-Agent'   : __file__,
                   'Content-Type' : 'application/x-www-form-urlencoded',
                   'Accept'       : 'text/html'
                    }

     query   = urllib.urlencode(data) if data is not None else ''
     request = urllib2.Request(url, query, headers)

     response = urllib2.urlopen(request)
     #print('INFO', response.info())
     return response

AND MAIN CODE:

  try:
         webpage = post_request(url=MAIN_URL, data=make_form_data(query))
         html    = webpage.read()
         print('Authentication Information: access granted.')

except URLError, e:
         print('Authentication Information: {msg}.'.format(msg=e))
         sys.exit(1)

That's all folks Authentication+Posting a request.
make_form_data() is the most important.
To find cgi data dict you can use ClientForm.py (google it!) it is a 
good helper to find form data.

Regards
Karim