[Python-ideas] Memory limits [was Re: Membership of infinite iterators]

[Steven D'Aprano]

On Wed, Oct 18, 2017 at 01:39:28PM +0300, Koos Zevenhoven wrote:

> I'm writing from my phone now, cause I was dumb enough to try list(count())

You have my sympathies -- I once, due to typo, accidentally ran 
something like range(10**100) in Python 2.

 
> But should it be fixed in list or in count?

Neither. There are too many other places this can break for it to be 
effective to try to fix each one in place.

e.g. set(xrange(2**64)), or tuple(itertools.repeat([1]))

Rather, I think we should set a memory limit that applies to the whole 
process. Once you try to allocate more memory, you get an MemoryError 
exception rather than have the OS thrash forever trying to allocate a 
terabyte on a 4GB machine.

(I don't actually understand why the OS can't fix this.)

Being able to limit memory use is a fairly common request, e.g. on 
Stackoverflow:

https://stackoverflow.com/questions/30269238/limit-memory-usage
https://stackoverflow.com/questions/2308091/how-to-limit-the-heap-size
https://community.webfaction.com/questions/15367/setting-max-memory-for-python-script

And Java apparently has a commandline switch to manage memory:

https://stackoverflow.com/questions/22887400/is-there-an-equivalent-to-java-xmx-for-python

The problems with the resources module are that its effectively an 
interface to ulimit, which makes it confusing and platform-specific; it 
is no help to Windows users; it isn't used by default; and not many 
people know about it. (I know I didn't until tonight.)
 
So here is my suggestion:

1. Let's add a function in sys to set the "maximum memory" available, 
for some definition of memory that makes the most sense on your 
platform. Ordinary Python programmers shouldn't have to try to decipher 
the ulimit interface.

2. Have a command line switch to set that value, e.g.:

    python3 -L 1073741824  # 1 GiB
    python3 -L 0  # machine-dependent limit
    python3 -L -1  # unlimited

where the machine-dependent limit is set by the interpreter, depending 
on the amount of memory it thinks it has available.

3. For the moment, stick to defaulting to -L -1 "unlimited", but with 
the intention to change to -L 0 "let the interpreter decide" in some 
future release, after an appropriate transition period.

On Linux, we can always run

   ulimit XXXX python3 

but honestly I never know which option to give (maximum stack size? 
maximum virtual memory? why is there no setting for maximum real 
memory?) and that doesn't help Windows users.


Thoughts?




-- 
Steve