[Tutor] Downloading S3 Logs

[Tino Dai]

On Thu, Feb 11, 2010 at 11:23 AM, Lao Mao <laomao1975 at googlemail.com> wrote:

> Hello,
>
> I've written the below to get the previous day's logs from an Amazon S3
> bucket.
>
> #!/usr/bin/python
> import time
> from datetime import datetime
> import boto
>
> daily_s3_log = open("/tmp/s3logs", "w+")
> now = datetime.now()
> connection = boto.connect_s3()
> bucket = connection.get_bucket("downloads.sekrit.co.uk")
> todays_keys = []
>
> for key in bucket:
>   time_difference = (now -
> datetime(*time.strptime(key.last_modified.split(".")[0], "%Y-%m-%dT%H:
> %M:%S")[0:6])).days
>   if time_difference < 1 and key.name.startswith("log/access_log"):
>     todays_keys.append(key)
>
> for key in todays_keys:
>   key.get_file(daily_s3_log)
>
> daily_s3_log.close()
>
> This takes about 2 mins to download a day's logs (about 25M).
>
> What I would do would be to profile your code:
You can find more profiling info @
http://docs.python.org/library/profile.html

-Tino
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/tutor/attachments/20100211/92372d04/attachment.htm>