Read from database, write to another database, simultaneously

[johnf]

Bjoern Schliessmann wrote:

> Sean Davis wrote:
> 
>> The author of one of the python database clients mentioned that
>> using one thread to retrieve the data from the oracle database and
>> another to insert the data into postgresql with something like a
>> pipe between the two threads might make sense, keeping both IO
>> streams busy.
> 
> IMHO he's wrong. Network interaction is quite slow compared with CPU
> performance, so there's no gain (maybe even overhead due to thread
> management and locking stuff). That's true even on multiprocessor
> machines, not only because there's almost nothing to compute but
> only IO traffic. CMIIW.
> 
> Using multiplexing, you'll get good results with simple code without
> the danger of deadlocks. Have a look at asyncore (standard library)
> or the Twisted framework -- personally, I prefer the latter.
> 
> Regards,
> 
> 
> Björn
> 
Sean you can't win - everyone has a different idea!  You need to explain
that oracle has millions of records and it's possible to a pipe open to
feed the Postgres side.  

One thing I didn't get - is this a one time transfer or something that is
going to happen often.  

One time transfer live to the time issue.  

Johnf