[Python-ideas] Memory limits [was Re: Membership of infinite iterators]
steve at pearwood.info
Wed Oct 18 07:38:17 EDT 2017
On Wed, Oct 18, 2017 at 01:39:28PM +0300, Koos Zevenhoven wrote:
> I'm writing from my phone now, cause I was dumb enough to try list(count())
You have my sympathies -- I once, due to typo, accidentally ran
something like range(10**100) in Python 2.
> But should it be fixed in list or in count?
Neither. There are too many other places this can break for it to be
effective to try to fix each one in place.
e.g. set(xrange(2**64)), or tuple(itertools.repeat())
Rather, I think we should set a memory limit that applies to the whole
process. Once you try to allocate more memory, you get an MemoryError
exception rather than have the OS thrash forever trying to allocate a
terabyte on a 4GB machine.
(I don't actually understand why the OS can't fix this.)
Being able to limit memory use is a fairly common request, e.g. on
And Java apparently has a commandline switch to manage memory:
The problems with the resources module are that its effectively an
interface to ulimit, which makes it confusing and platform-specific; it
is no help to Windows users; it isn't used by default; and not many
people know about it. (I know I didn't until tonight.)
So here is my suggestion:
1. Let's add a function in sys to set the "maximum memory" available,
for some definition of memory that makes the most sense on your
platform. Ordinary Python programmers shouldn't have to try to decipher
the ulimit interface.
2. Have a command line switch to set that value, e.g.:
python3 -L 1073741824 # 1 GiB
python3 -L 0 # machine-dependent limit
python3 -L -1 # unlimited
where the machine-dependent limit is set by the interpreter, depending
on the amount of memory it thinks it has available.
3. For the moment, stick to defaulting to -L -1 "unlimited", but with
the intention to change to -L 0 "let the interpreter decide" in some
future release, after an appropriate transition period.
On Linux, we can always run
ulimit XXXX python3
but honestly I never know which option to give (maximum stack size?
maximum virtual memory? why is there no setting for maximum real
memory?) and that doesn't help Windows users.
More information about the Python-ideas