[Python-ideas] Memory limits [was Re: Membership of infinite iterators]

[Nick Coghlan]

On 18 October 2017 at 22:51, Stefan Krah <stefan at bytereef.org> wrote:

> On Wed, Oct 18, 2017 at 10:43:57PM +1000, Nick Coghlan wrote:
> > Per-process memory quotas *can* help avoid this, but enforcing them
> > requires that every process run in a resource controlled sandbox. Hence,
> > it's not a coincidence that mobile operating systems and container-based
> > server environments already work that way, and the improved ability to
> cope
> > with misbehaving applications is part of why desktop operating systems
> > would like to follow the lead of their mobile and server counterparts :)
>
> Does this also fall under the sandbox definition?
>
> $ softlimit -m 1000000000 python3
>

Yeah, Linux offers good opt-in tools for this kind of thing, and the
combination of Android and containerised server environments means they're
only getting better. But we're still some time away from it being routine
for your desktop to be well protected from memory management misbehaviour
in arbitrary console or GUI applications.

The resource module (which Steven mentioned in passing) already provides
opt-in access to some of those features from within the program itself:
https://docs.python.org/3/library/resource.html

For example:

    >>> import sys, resource
    >>> data = bytes(2**32)
    >>> resource.setrlimit(resource.RLIMIT_DATA, (2**31, sys.maxsize))
    >>> data = bytes(2**32)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    MemoryError
    >>> resource.setrlimit(resource.RLIMIT_DATA, (sys.maxsize, sys.maxsize))
    >>> data = bytes(2**32)

(Bulk memory allocations start failing on my machine somewhere between
2**33 and 2**34, which is about what I'd expect, since it has 8 GiB of
physical RAM installed)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-ideas/attachments/20171018/432620d8/attachment-0001.html>