A fun python CLI program for all to enjoy!
MRAB
python at mrabarnett.plus.com
Fri May 6 16:30:17 EDT 2016
On 2016-05-06 20:10, DFS wrote:
> getAddresses.py
>
> Scrapes addresses from www.usdirectory.com and stores them in a SQLite
> database, or writes them to text files for mailing labels, etc
>
> Now, just by typing 'fast food Taco Bell <city> 10 db all' you can find
> out how many Taco Bells are within 10 miles of you, and store all the
> addresses in your own address database.
>
> No more convoluted Googling, or hitting the 'Next Page' button, or
> fumbling with the Yellow Pages...
>
> Note: the db structure is flat on purpose, and the .csv files aren't
> quote delimited.
>
> Put the program in its own directory. It creates the SQLite database
> there, and writes files there, too.
>
> Reviews of code, bug reports, criticisms, suggestions for improvement,
> etc are all welcome.
>
OK, you asked for it... :-)
1. It's shorter and clearer not to compare with True or False:
if verbose:
and:
if not dupeRow:
2. You can print a blank line with an empty print statement:
print
3. When looking for unique items, a set is a better choice than a list:
addrCheck = set()
def addrUnique(addr):
if addr not in addrCheck:
x = True
addrCheck.add(addr)
else:
x = False
return x
4. Try string formatting instead multiple concatenation:
print "%s arguments" % argCnt
5. Strings have a .join method, and when you combine it with string slicing:
keyw = "+".join(sys.argv[1 : argCnt - 5])
6. Another example of string formatting:
search = "%s %s %s %s %s" % (keyw, cityzip, state, miles, addrWant)
7. It's recommended to use the 'with' statement when handling files:
with open(webfile, "w") as f:
if store == "csv":
f.write("Name,Address,CityStateZip\n")
If you don't want to use the 'with' statement, note that closing the
file is:
f.close()
It needs the "()"!
8. When using SQL, you shouldn't try to insert the values yourself; you
should use parametrised queries:
cSQL = "DELETE FROM addresses WHERE datasrc = ? AND search = ?;"
if verbose:
print cSQL
db.execute(cSQL, (datasrc, search))
conn.commit()
It'll insert the values where the "?" are and will do any necessary
quoting itself. (Actually, some drivers use "?", others use "%s", so if
it doesn't work with one, try the other.)
The way you wrote it, it would fail if a value contained a "'".
It's that kind of thing that leads to SQL injection attacks.
More information about the Python-list
mailing list