python daemon - compress data and load data into MySQL by pyodbc

[MacRules]

Dennis Lee Bieber wrote:
> On Thu, 03 Sep 2009 14:43:40 -0400, MacRules <MacRules at nome.com>
> declaimed the following in gmane.comp.python.general:
> 
>> Oracle DB in data center 1 (LA, west coast)
>> MSSQL DB in data center 2 (DC, east coast)
> 
> 	Note that your thread subject line states MySQL... There is a big
> difference between MySQL and M$ SQL-Server.
> 
>> So network bandwidth is an issue, I prefer to have gzip fist and deliver 
>> the data.
>>
>> I need 2 python daemons or a web service here in the future.
>> I will enter the Oracle table name, user id and password.
>> So the task is dump out Oracle data (Linux) and insert that to MSSQL.
>>
> 	One: daemon implies something running in the background,
> stand-alone, and not associated with any "user" type account.  IOW,
> something like a server program...
> 
> 	Do you really intend to create a server application that sits around
> waiting for anyone to connect to it and send a properly formatted
> request? What safeguards do you intend to restrict this daemon from
> responding to hackers (after all, you're going to have to have an open
> port to accept requests from outside).
> 
> 	Second: said daemon will have to be running ON the end with the
> Oracle DBMS (or, at least, within its LAN which presumably has higher
> bandwidth than the long distance connection).
>  
>> I can try first with 1 daemon python. Take the Oracle data file, and let 
>> the daemon connects to MSSQL (with pyodbc) and load the data in.
> 
> 	I see three operations here:
> 
> 1	dumping data from Oracle
> 2	transferring data from LA to DC
> 3	importing data to whatever DBMS you are really using.
> 
> 	I believe, in a later post, you mention this is a one-time task. If
> so, WHY worry about network bandwidth... Just let it run overnight...
> 
> 	Question: does the Oracle server permit remote connections?
> (dangerous unless behind a VPN type connection)
> 
> 	If it does, I'd suggest the initial program should just open a
> connection both the Oracle server, and the local server. Request the
> contents of whatever needs to be transferred, collect the data, massage
> it into whatever format is needed for the local, and insert it to the
> local... Anything else will require being able to log into the remote
> machine and running programs ON it. In which case you could just FTP the
> data from one end to the other (assuming there is an FTP server active
> on one or the other end -- or a web server address you can stuff the
> files into and use HTTP to fetch them)

I have 1000 servers, and 1 time each, so it is 1000 times.
What I try to say, I do not need real-time push/pull here.

I know the manual operations, just want Python does automatically for me.

I like to see good examples on how to code daemon, signal, maybe 
multi-threading.

I wrote something like that in C way back. And good at shell scripts.