[Python-ideas] Memory limits [was Re: Membership of infinite iterators]

Nick Coghlan ncoghlan at gmail.com
Wed Oct 18 09:43:12 EDT 2017

On 18 October 2017 at 22:51, Stefan Krah <stefan at bytereef.org> wrote:

> On Wed, Oct 18, 2017 at 10:43:57PM +1000, Nick Coghlan wrote:
> > Per-process memory quotas *can* help avoid this, but enforcing them
> > requires that every process run in a resource controlled sandbox. Hence,
> > it's not a coincidence that mobile operating systems and container-based
> > server environments already work that way, and the improved ability to
> cope
> > with misbehaving applications is part of why desktop operating systems
> > would like to follow the lead of their mobile and server counterparts :)
> Does this also fall under the sandbox definition?
> $ softlimit -m 1000000000 python3

Yeah, Linux offers good opt-in tools for this kind of thing, and the
combination of Android and containerised server environments means they're
only getting better. But we're still some time away from it being routine
for your desktop to be well protected from memory management misbehaviour
in arbitrary console or GUI applications.

The resource module (which Steven mentioned in passing) already provides
opt-in access to some of those features from within the program itself:

For example:

    >>> import sys, resource
    >>> data = bytes(2**32)
    >>> resource.setrlimit(resource.RLIMIT_DATA, (2**31, sys.maxsize))
    >>> data = bytes(2**32)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    >>> resource.setrlimit(resource.RLIMIT_DATA, (sys.maxsize, sys.maxsize))
    >>> data = bytes(2**32)

(Bulk memory allocations start failing on my machine somewhere between
2**33 and 2**34, which is about what I'd expect, since it has 8 GiB of
physical RAM installed)


Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20171018/432620d8/attachment-0001.html>

More information about the Python-ideas mailing list