How to script client-side HTTP transactions?

[Ramon Felciano]

Hi --

I'm trying automate a set of HTTP client-side steps via a Python script. The
problem I'm running into is handling *client-side* cookies. In particular,
the flow of events I want is something like this:

    1. Generate a POST to the server login form
    2. Server returns an OK and sets a cookie (e.g. username + password)
    3. Generate a POST to the server main form, including any cookies needed
    4. Second post completes successfully.

The problem I'm having is that I can't figure out how to use the various
Python web-related libraries to handle client-side cookies. There are
libraries for handler server-side/CGI aspects of cookies, but not much
support for HTTP clients. The scripts fail outright, or the server returns
an error page usually reserved for clients that don't support cookies.

What I'm hoping for is some type of HTTPClient object that encapsulates
transactions to an HTTP server, including getting and setting cookies (e.g.
figures out which cookies should be sent to which HTTP server). This
HTTPClient object would also take care of notifying the server that it is a
cookie-capable client. As a bonus, it would be nice to have this client be
able to emulate various browser settings since some CGIs fail if they think
you aren't running browser X.

I've done this with Perl, which has library that makes it easy to manage
cookies across HTTP requests and responses (e.g. take the cookies from an
HTTP Response and copy them to a new HTTP Request). I like Python better
now, and was hoping to continue to use it. Any suggestions for how to handle
this?

Thank you in advance for your time.

Ramon M. Felciano