[Tutor] retrieving httponly cookies on accessing webpage with urllib2

[xbmuncher]

Thanks for the insight Kent. The problem wasn't because of the httpOnly
cookies, I just wasn't requesting the right page. I did get it to send me
the httpOnly cookies when I requested the right page. Apparently, the right
page was a redirection of the original page. When I used to use cURL in PHP
I remember that it had an option to automatically follow redirects. How can
I do this with urllib2?

For the headers, I did verify that it sent the headers as I had formed them
(just not in the right order) with wireshark. So my method does work, I also
got this method from here:
http://www.voidspace.org.uk/python/articles/urllib2.shtml#headers

On Fri, Oct 17, 2008 at 7:29 AM, Kent Johnson <kent37 at tds.net> wrote:

> On Thu, Oct 16, 2008 at 11:40 PM, xbmuncher <xboxmuncher at gmail.com> wrote:
> > I'm trying to mimic my firefox browser in requesting a webpage with
> python.
>
> > So I tried trusty ol' urllib2 to request it in python:
> > import urllib2
> >
> >
> > url = 'http://www.website.com'
> >
> > #headers
> > h = {
> > 'User-Agent' : 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:
> 1.9.0.3)
> <snip>
> > }
> > #request page
> > reqObj = urllib2.Request(url, None, h)
> > urlObj = urllib2.urlopen(reqObj)
>
> It doesn't work to set the User-Agent header this way. See
> http://personalpages.tds.net/~kent37/kk/00010.html#e10request-headers<http://personalpages.tds.net/%7Ekent37/kk/00010.html#e10request-headers>
> for a recipe.
>
> > #read response
> > print urlObj.read()
>
> What content do you get? Is it an error message? What does wireshark
> show for this request?
>
> > Notice the content length is considerably smaller, and no cookies are
> sent
> > to me like they were in firefox. I know only a little about httpOnly
> > cookies, but that it is some kind of special cookie that I suppose has
> > something to do with python not being able to access it like firefox. All
> I
> > want to do is have python receive the same cookies that firefox did, how
> can
> > I do this? I read somewhere that httpOnly cookies were implemented in the
> > python cookie module:
> > http://glyphobet.net/blog/blurb/285
> > ....yet the other cookies aren't being sent either...
>
> I don't think that has anything to do with your problem. httpOnly is
> set by the server and interpreted by the browser. The Python change
> was to allow httpOnly to be set by servers written in Python.
>
> Kent
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/tutor/attachments/20081018/78392b04/attachment.htm>