String concatenation - which is the fastest way ?
stefan_ml at behnel.de
Fri Aug 12 05:25:06 EDT 2011
przemolicc at poczta.fm, 11.08.2011 16:39:
> On Thu, Aug 11, 2011 at 02:48:43PM +0100, Chris Angelico wrote:
>> On Thu, Aug 11, 2011 at 2:46 PM,<przemolicc at poczta.fm> wrote:
>>> This is the way I am going to use.
>>> But what is the best data type to hold so many rows and then operate on them ?
>> List of strings. Take it straight from your Oracle interface and work
>> with it directly.
> Can I use this list in the following way ?
> subprocess_1 - run on list between 1 and 10000
> subprocess_2 - run on list between 10001 and 20000
> subprocess_3 - run on list between 20001 and 30000
Sure. Just read the data as it comes in from the database and fill up a
chunk, then hand that on to a process. You can also distribute it in
smaller packets, just check what size gives the best throughput.
Still, I'd give each work parcel a number and then reorder the results
while collecting them, that allows you to vary the chunk size and the
process size independently, without having to wait for a process that
happens to take longer.
More information about the Python-list