[New-bugs-announce] [issue3448] Multi-process 2to3

[Nick Edds]

New submission from Nick Edds <nedds at uchicago.edu>:

Here is a working, multiprocess version of 2to3 with a few caveats.
First, you need to already have the processing module installed for this
to work. If we don't want to include processing in some way, I think I
can modify this to only import processing and use the Process method if
the user wants to run it with more than one process. Also, I know that
logger is supposed to be thread safe, so I am correct in assuming that
this means it is also multi-process safe? This fix delegates the fixing
of files to a designated number of processes, so on a multi-core
machine, it is significantly faster. It may be appropriate to add a test
suite for it, but I have not yet done so. I've tested it on my dual-core
laptop running Ubuntu and a quad-core mac and it appears to be working
fine. I don't know if the use of tempfile was the right choice, but it
seemed reasonable. Another possibility is to instead using a processing
Queue and pass it the output, filename, and input rather than calling
write_file. Also, the join timeout I used is completely arbitrary
because I don't really have a sense of what a reasonable value for it
should be.

----------
assignee: collinwinter
components: 2to3 (2.x to 3.0 conversion tool)
files: multiProcess.diff
keywords: patch
messages: 70277
nosy: collinwinter, nedds
severity: normal
status: open
title: Multi-process 2to3
type: performance
Added file: http://bugs.python.org/file10985/multiProcess.diff

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue3448>
_______________________________________