[Tutor] Downloading S3 Logs

[Lao Mao]

Hello,

I've written the below to get the previous day's logs from an Amazon S3
bucket.

#!/usr/bin/python
import time
from datetime import datetime
import boto

daily_s3_log = open("/tmp/s3logs", "w+")
now = datetime.now()
connection = boto.connect_s3()
bucket = connection.get_bucket("downloads.sekrit.co.uk")
todays_keys = []

for key in bucket:
  time_difference = (now -
datetime(*time.strptime(key.last_modified.split(".")[0], "%Y-%m-%dT%H:
%M:%S")[0:6])).days
  if time_difference < 1 and key.name.startswith("log/access_log"):
    todays_keys.append(key)

for key in todays_keys:
  key.get_file(daily_s3_log)

daily_s3_log.close()

This takes about 2 mins to download a day's logs (about 25M).

I'd appreciate any improvements or feedback on the above.

For example, would it make sense to make the first loop into a generator
function that yields the interesting keys?  Also is there a cleaner way to
do the date comparison in Python 2.4?

Thanks,

Laomao
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/tutor/attachments/20100211/564db211/attachment.htm>