I'd love to help, but a "rough composite" is hard to make guesses about, especially since you're talking about hard-to-predict memory-consumption behavior.
Can you attach a <
http://sscce.org/> which is actually runnable, for example, with a canned list of input URLs (or better yet with an included web server so the URLs can be localhost and more predictable), so we can debug and diagnose a running program instead of ideas about the outline of one?