memory leak with re.match
Cameron Simpson
cs at zip.com.au
Tue Jul 4 18:46:08 EDT 2017
On 04Jul2017 17:01, Mayling ge <maylinge0903 at gmail.com> wrote:
> My function is in the following way to handle file line by line. There are
> multiple error patterns defined and need to apply to each line. I use
> multiprocessing.Pool to handle the file in block.
>
> The memory usage increases to 2G for a 1G file. And stays in 2G even after
> the file processing. File closed in the end.
>
> If I comment out the call to re_pat.match, memory usage is normal and
> keeps under 100Mb. [...]
>
> def line_match(lines, errors)
> for error in errors:
> try:
> re_pat = re.compile(error['pattern'])
> except Exception:
> print_error
> continue
> for line in lines:
> m = re_pat.match(line)
> # other code to handle matched object
[...]
> Notes: I omit some code as I think the significant difference is
> with/without re_pat.match(...)
Hmm. Does the handling code (omitted) keep the line or match object in memory?
If leaving out the "m = re_pat.match(line)" triggers the leak, and presuming
that line itself doesn't leak, then I would start to suspect the handling code
is not letting go of the match object "m" or of the line (which is probably
attached to the match object "m" to support things like m.group() and so
forth).
So you might need to show us the handling code.
Cheers,
Cameron Simpson <cs at zip.com.au>
More information about the Python-list
mailing list