[Tutor] Nested for loops, possibly?

[DaveA]

On February 5, 2015 8:27:29 AM EST, Bob Williams <linux at barrowhillfarm.org.uk> wrote:
>Hi,
>
>My script is running under Python 3.4.1 on a 64bit openSUSE linux
>system. It is a backup script making calls to rsync and btrfs-tools,
>and backing up several different paths. Here is the script, my question
>follows below:
>
>**Code**
>import datetime
>import glob
>import os, os.path
>import subprocess
>import sys
>
>if not os.getuid() == 0:
>    print("\n*** This script must be run as root. ***\n")
>    sys.exit()
>
>mnt_path = "/home/bob/A3"
>    
>subprocess.call(["mount", "LABEL=backup", mnt_path])
>if not os.path.ismount(mnt_path):
>    print("\nBackup drive is not mounted\nCheck if it is attached.\n")
>    sys.exit()
>else:
>    print("\nBackup drive mounted at", mnt_path, "\n")
>
>src_path = "/home/bob"
>
>today = datetime.datetime.now()
>fname = today.strftime("%y-%m-%d_%H-%M")
>
>doc_retain = datetime.timedelta(days=90)
>pic_retain = datetime.timedelta(days=90)
>misc_retain = datetime.timedelta(days=90)
>etc_retain = datetime.timedelta(days=30)
>mus_retain = datetime.timedelta(days=30)
>web_retain = datetime.timedelta(days=30)
>rep_retain = datetime.timedelta(days=30)
>
>doc_srcpath = os.path.join(src_path, "Documents")
>pic_srcpath = os.path.join(src_path, "Pictures")
>misc_srcpath = src_path
>etc_srcpath = "/etc"
>music_srcpath = os.path.join(src_path, "music")
>www_srcpath = "/srv/www"
>repo_srcpath = os.path.join(src_path, "download")
>
Three things bother me about this portion of code, Too much code at top-level, too many separate variables, too many global.The last point doesn't matter much, since they're constant.

I'd put these Values into a class, and make a list of instances. If you're not yet comfortable with that, it would also be possible to make a list per "paragraph", and use zip to combine them, as already suggested. 

The class would hold retain, srcpath, syncpath, snappath, etc. And your list would have 7 instances of that class currently, corresponding to your doc, pic, misc, ...

That list would be THE global, replacing these 35 or so. It would be populated something like:

def initialize (worklist=[]):
    worklist . append  (Job (90, src_path, "Documents", "documents", "docsnaps")
    worklist .append (Job (90, src_path,  "Pictures", ...
    ....
    return worklist

Now all the joins and globs are done in the Job initializer, just once.

.....th.join(mnt_path, "documents")
>pic_syncpath = os.path.join(mnt_path, "pictures")
>misc_syncpath = os.path.join(mnt_path, "miscellaneous")
>etc_syncpath = os.path.join(mnt_path, "etc")
>music_syncpath = os.path.join(mnt_path, "music")
>www_syncpath = os.path.join(mnt_path, "www")
>repo_syncpath = os.path.join(mnt_path, "repo")
>
>doc_snappath = os.path.join(mnt_path, "docsnaps", fname)
>pic_snappath = os.path.join(mnt_path, "picsnaps", fname)
>misc_snappath = os.path.join(mnt_path, "miscsnaps", fname)
>etc_snappath = os.path.join(mnt_path, "etcsnaps", fname)
>music_snappath = os.path.join(mnt_path, "musicsnaps", fname)
>www_snappath = os.path.join(mnt_path, "wwwsnaps", fname)
>repo_snappath = os.path.join(mnt_path, "reposnaps", fname)
>
>doc_snaplist = glob.glob(mnt_path + "/docsnaps/*")
>pic_snaplist = glob.glob(mnt_path + "/picsnaps/*")
>misc_snaplist = glob.glob(mnt_path + "/miscsnaps/*")
>etc_snaplist = glob.glob(mnt_path + "/etcsnaps/*")
>music_snaplist = glob.glob(mnt_path + "/musicsnaps/*")
>www_snaplist = glob.glob(mnt_path + "/wwwsnaps/*")
>repo_snaplist = glob.glob(mnt_path + "/reposnaps/*")
>
>def do_sync(source, dest):
>subprocess.call(['rsync', '-av', '--safe-links', '--delete-excluded',
>'-F', source, dest])
>    print("\n")
>
>def create_snaps(newsnap, snapdest):
> subprocess.call(['btrfs', 'subvolume', 'snapshot', newsnap, snapdest])
>    print("\n")
>
>def expire_snaps(snaplist, today, expiry_interval):
>    x = 0
>print("\nDeleting snapshots older than", str(expiry_interval)[:7],
>"...")
>    for i in range(0, len(snaplist)):
>snap_date = datetime.datetime.strptime(snaplist[i][-14:],
>"%y-%m-%d_%H-%M")
>        if today - snap_date >= expiry_interval:
>         subprocess.call(['btrfs', 'subvolume', 'delete', snaplist[i]])
>            x += 1
>    if x == 0:
>print("... No snapshots older than", str(expiry_interval)[:7], "found -
>nothing to do.")
>    else:
>        print("...", x, "snapshot(s) deleted.")
>    print("\n")
>
>def main():
>    print("Backing up ", src_path, "/Documents\n", sep='')
>    do_sync(doc_srcpath, doc_syncpath)
>    create_snaps(doc_syncpath, doc_snappath)
>    print("Documents backup completed.")
>    expire_snaps(doc_snaplist, today, doc_retain)

At this point main becomes something like 

    jobs = initialize ()
    for job in jobs:
           do_sync (job.srcpath, job.syncpath)
           create_snaps (job.syncpath,  job.snappath)
            expire_snaps (job.snaplist, ...
>
>    print("Backing up ", src_path, "/Pictures\n", sep='')
>    do_sync(pic_srcpath, pic_syncpath)
>    create_snaps(pic_syncpath, pic_snappath)
>    print("Pictures backup completed.")
>    expire_snaps(pic_snaplist, today, pic_retain)
>
>    print("Backing up Miscellaneous files\n")      
>    do_sync(misc_srcpath, misc_syncpath)
>    create_snaps(misc_syncpath, misc_snappath)
>    print("Miscellaneous backup completed.")
>    expire_snaps(misc_snaplist, today, misc_retain)
>
>    print("Backing up /etc\n")
>    do_sync(etc_srcpath, etc_syncpath)
>    create_snaps(etc_syncpath, etc_snappath)
>    print("Backup of /etc completed.")
>    expire_snaps(etc_snaplist, today, etc_retain)
>
>    print("Backing up ", src_path, "/music\n", sep='')
>    do_sync(music_srcpath, music_syncpath)
>    create_snaps(music_syncpath, music_snappath)
>    print("Music backup completed.")
>    expire_snaps(music_snaplist, today, mus_retain)
>
>    print("Backing up Web server\n")
>    do_sync(www_srcpath, www_syncpath)
>    create_snaps(www_syncpath, www_snappath)
>    print("Web server backup completed.")
>    expire_snaps(www_snaplist, today, web_retain)
>
>    print("Backing up Download repository\n")
>    do_sync(repo_srcpath, repo_syncpath)
>    create_snaps(repo_syncpath, repo_snappath)
>    print("Download repository backup completed.")
>    expire_snaps(repo_snaplist, today, rep_retain)
>
>    print("\nAll backups completed.")
>    print("\nUnmounting backup drive.\n")
>    subprocess.call(['umount', mnt_path])
>print("\n>>> Please power down the Quad external drive enclosure
><<<\n")
>
>if __name__ == "__main__":
>    main()
>**/Code**
>
>I would like to reduce all those repeated calls to do_sync() in main(),
>for example, to one by putting the *_srcpath and *_*syncpath variables
>into lists (eg. source_list and sync_list) and using a for loop to get
>the first item out of each list, then the second item, etc. Something
>like:
>
>for i in range(0, len(source_list)):
>    for j in range(0, len(sync_list)):
>        do_sync(source_list[i], sync_list[j])
>
>but this will get all the values of sync_list[j] for each value of
>source_list[i], which is not what I want.

If you're sure all the lists are the same length,  you can use zip. But I'd recommend a list of objects. 

>
>I hope this is clear enough to see my problem? I realise that the print
>statements will need some work, I'm just trying to get the
>functionality working.
>
>TIA
>
>Bob

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.