Hi Rocky,
That's still not great, but it does confirm my suspicion that scaling will improve with bigger data. Hopefully, that will continue to even larger datasets. Additionally, if you have more than one dataset to analyze, you can also parallelize your loop over datasets with the parallel_objects and piter commands.
This may work better for you since it will use significantly less communication between processes. Also, again, you'll be limited by the performance of your filesystem. In practice, I find that 8-16 processes is the absolute most I can use efficiently in parallel on your average computing cluster.
Britton