Sorry typed the wrong address. Sorry for the spam... On Fri, Apr 17, 2015 at 11:47 PM, Wenlei Xie <wenlei.xie@gmail.com> wrote:

Hi,

I am wondering the mechanism that determines the number of partitions created by SparkContext.sequenceFile ?

For example, although my file has only 4 splits, Spark would create 16 partitions for it. Is it determined by the file size? Is there any way to control it? (Looks like I can only tune minPartitions but not maxPartitions)

Thank you!

Best, Wenlei

-- Wenlei Xie (谢文磊) Ph.D. Candidate Department of Computer Science 456 Gates Hall, Cornell University Ithaca, NY 14853, USA Email: wenlei.xie@gmail.com