Linux and Python scripts, help needed

[Chris Gonnerman]

----- Original Message ----- 
From: "Ron Stephens" <rdsteph at earthlink.net>


> I am back from a trip, I have manually updated the mygale generated
> Pythonic web articels page on my web site. There are quite a few new
> links of interest.
> 
> However, the automated updating from my Linux box still did not work
> while I was gone. I will manually update once per day but I would 
> sure like to get the automaton working.
> 
> I am embarrassed to come back once more to ask for help on this 
> subject.
> I have used crontab to set up 
> a my_cron_file that runs five Python scripts to automatically run
> mygale.py and associated programs to 
> update the output.html file listing new web articles about Python 
> once per day.
> 
> My crontab file reads as suchas checked using crontab -l:
> 
> 30 4 * * * /usr/bin/python /home/ron/mygale.py get all
> 30 5 * * * /usr/bin/python /home/ron/urls.py
> 30 6 * * * /usr/bin/python /home/ron/urldb.py
> 00 7 * * * /usr/bin/python /home/ron/mygale.py -n 200 display all
> 30 7 * * * /usr/bin/python /home/ron/mygale-uploader.py
> 
> crond is running , as checked using
>   /etc/rc.d/init.d/crond  status which shows a running cron process 
> with a pid number.
> 
> Also,  ps aux | grep cron shows a /usr/sbin/cron process.
> 
> Finally, the  cron daemons log check /var/log/cron shows all 5 Python
> scripts properly called at their 
> appointed times each morning. No errors are at all are recorded in 
> the cron error log files.
> 
> Now, when I run these scripts exactly as typed above, from the Bash
> shell,
> everything works as planned and that's how I have been updating the
> online HTML page that shows the Pythonic articles on my web site.
> 
> The mystery is that, even though the log files show the Python 
> scripts called each day, no update occurs 
> on my web site . Indeed, when I check the output.html file in my
> /home/ron directory, it is not updated 
> at all by these cron processes. Even though the log file shows the
> scripts as being ran, it is as if they did not run at all, in that no
> output occurs.
>
> Yet when I run the same scripts manually, all is updated.
> 
> So this morning at 4:30 AM I sat up and ran a "top" process from the
> Bash shell, that shows all 
> running processes. Sure enough, at exactly 4:30 AM, a new Python 
> process was spawned. The new 
> Python process was matched with a new entry in the cron log file of 
> the mygale.py program being called.
> 
> I watched carefully, and the mygale.py process lasted for about 30
> seconds and then disappeared from 
> the "top" screen. I assume it terminated normally. Except that when I
> run the mygale.py get all script 
> manually  in the BASH shell, it takes about two minutes to complete.
> This is the only sign of any 
> problem, but still no error messages are generated anywhere I can 
> find.

cron errors are mailed to your system mailbox.  Have you tried typing
"mail" at the BASH prompt?

> Well, it seems to me that a series of processes on Linux or any
> operating system should be 
> deterministic. That is, things should make sense, when the exact same
> steps are taken, the result should 
> always be the same.

cron jobs run in a bit different environment than the command prompt;
this often leads to problems as you are noting.

Add this to the top of each script:

    import sys
    sys.stdout = open("/home/ron/.error-log","a")
    sys.stderr = sys.stdout
    print "\n(job name) Starting..."

then any error traceback goes to the file /home/ron/.error-log and you
can review them at your leasure.

> This procedure I am trying to do does not seem to make sense to me, 
> it seems mysterious. Linux should not be mysterious.
> 
> Can anyone help me?
> 
> By the way, shouldn't it be possible to cut and paste from the BASH
> shell to a text editor? It would save 
> me a lot of typing ;-))))

Depends on what terminal window you are using.  I'm not a KDE user,
but I think the normal KDE terminal window supports old-fashioned
cut-and-paste; highlight the text you want to copy, switch windows,
and select paste (no cut/copy step).  GNOME's terminal is positively
frustrating this way.

> The clipboard in general is a frustration; I can't yet make it work
> properly in many situations. Is it me 
> or does the KDE clipboard lack a lot of functionality compared to
> Windows clipboard? I can't even 
> copy and paste from K-EDIT, nor K-WORD, to Netscape Composer. Oh, 
> well.

Regarding Netscape Composer, I don't know; I use Mozilla now and
it generally works.

> One last frustration: Linux does hang, at least to the point that 
> this newbie can't recover from it. My 
> box hung when I tried to open a web presentation by Guido van Rossum
> about Python Conference 10 
> using K Presenter.  I could not even open a console nor close the
> offending KDE shell to get back to a command line. 

Using CTRL-ALT-F1 or CTRL-ALT-F2 didn't get you a console?  Sounds
bad.

> The box also hangs when I try to print using cups on my HP
> T45 printer. (I wish I could use my printer.) My downloaded Open 
> Office also hangs my computer just by trying to open it.

This has to be the worst Linux-related hang problem I've ever heard
of; you should be posting to a Linux group about this.

> Nonetheless, these hang-ups and crashes are not of pivotal importance 
> ot me. But mysterious behavior 
> that not does not allow me to run daily scripts to update my web site
> does bother me. Computers 
> should not be mysterious. They should be deterministic. Do A, then B,
> then C, get X for output.

Add the logging steps I've outlined and see what happens.

> Help ;-))))

Good luck.  You can mail me off-list if you need additional help.