download all mib files from a web page

[Chris Rebert]

On Wed, May 27, 2009 at 9:29 AM, powah <wong_powah at yahoo.ca> wrote:
> I want to download all mib files from the web page:
> http://www.juniper.net/techpubs/software/junos/junos94/swconfig-net-mgmt/juniper-specific-mibs-junos-nm.html#jN18E19
>
> All mib filenames are of this format:
> www.juniper.net/techpubs ... .txt
>
> I write this program but has the following error.
> Please help.
> Thanks.
>
> [code]
> #!/usr/bin/env python
> import urllib2,os,urlparse
> url="http://www.juniper.net/techpubs/software/junos/junos94/swconfig-
> net-mgmt/juniper-specific-mibs-junos-nm.html#jN18E19"
> page=urllib2.urlopen(url)
> f=0
> links=[]
> data=page.read().split("\n")
> for item in data:
>    if "www.juniper.net/techpubs" in item:
>        httpind=item.index("www.juniper.net/techpubs")
>        item=item[httpind:]
>        #print "item " + item
>        ind=item.index("<")
>        links.append(item[:ind]) #grab all links
> # download all links
> for link in links:
>    print "link " + link
>    filename=link.split("/")[-1]
>    print "downloading ... " + filename
>    u=urllib2.urlopen(link)
>    p=u.read()
>    open(filename,"w").write(p)
> [/code]
>
> $ ~/python/downloadjuniper.py
> link www.juniper.net/techpubs/software/junos/junos94/swconfig-net-mgmt/mib-jnx-user-aaa.txt
> downloading ... mib-jnx-user-aaa.txt
> Traceback (most recent call last):
>  File "/home/powah/python/downloadjuniper.py", line 20, in ?
>    u=urllib2.urlopen(link)
>  File "/usr/lib/python2.4/urllib2.py", line 130, in urlopen
>    return _opener.open(url, data)
>  File "/usr/lib/python2.4/urllib2.py", line 350, in open
>    protocol = req.get_type()
>  File "/usr/lib/python2.4/urllib2.py", line 233, in get_type
>    raise ValueError, "unknown url type: %s" % self.__original
> ValueError: unknown url type:
> www.juniper.net/techpubs/software/junos/junos94/swconfig-net-mgmt/mib-jnx-user-aaa.txt

You need to ensure that all URL strings include the protocol to use,
i.e. "http://"

Cheers,
Chris
-- 
http://blog.rebertia.com