[Baypiggies] Adding testing to screen-scraping code

[Robert Zuber]

Asheesh,

An approach I have used in the past for http unit testing is to  
replace a lower-level client library with a mock version, rather than  
implementing a fake server.  I'm not familiar with the libraries you  
are using, but it looks like urllib2 would be a good candidate.   
Unless you are specifically concerned with testing the interaction of  
your code with urllib2, you can replace it during your unit tests  
with one that returns predefined results without having to cross the  
network.  Since you will be intercepting the requests, you'll also be  
able to look at those and confirm that they include what you are  
expecting.

Hope that helps,
Rob.


On Dec 16, 2007, at 9:59 PM, Asheesh Laroia wrote:

> On Sun, 16 Dec 2007, Charles Merriam wrote:
>
>> Sorry, if I fail to understand.  When I don't understand, I break
>> things down into small steps where
>>
>> 1.  You use a web hosted service which processes transcations from
>> HTTP forms.  No AJAX or JavaScript.
>> 2.  You have a scripts to automate this service to provide you a
>> better or more programmable interface.
>> 3.  You want regression tests for your scripts to ensure that the  
>> same
>> page from the server will cause exactly the same output from your
>> scripts.
>
> Yes.
>
>> This means you probably want:
>> 4.  A local HTTP server that checks for an exact set of inputs, and
>> then serves up a predetermined page if they match.  The page would
>> also include the right URL and the like.
>> 5.  A tool that makes snapshotting all those pages and linking them
>> together easy.
>
> That's right.
>
>> I don't have a solution yet.  Is this the problem?
>
> Yes.
>
> I'm also interested in hearing about related tools that solve
> similar problems if no one knows of tools that solve exactly my  
> problem.
>
> Thanks for rephrasing the question in a concise way! (-:
>
> -- Asheesh.
>
> --
> "It says he made us all to be just like him.  So if we're dumb,  
> then god is
> dumb, and maybe even a little ugly on the side."
>  		-- Frank Zappa
> _______________________________________________
> Baypiggies mailing list
> Baypiggies at python.org
> To change your subscription options or unsubscribe:
> http://mail.python.org/mailman/listinfo/baypiggies