[Python-ideas] Memory limits [was Re: Membership of infinite iterators]

eryk sun eryksun at gmail.com
Fri Oct 20 13:01:14 EDT 2017


On Thu, Oct 19, 2017 at 9:05 AM, Stephan Houben <stephanh42 at gmail.com> wrote:
>
> I (quickly) tried to get something to work using the win32 package,
> in particular the win32job functions.
> However, it seems setting
> "ProcessMemoryLimit" using win32job.SetInformationJobObject
> had no effect
> (i.e.  a subsequent win32job.QueryInformationJobObject
> still showed the limit as 0)?

Probably you didn't set the JOB_OBJECT_LIMIT_PROCESS_MEMORY flag.
Here's an example that tests the process memory limit using ctypes to
call VirtualAlloc, before and after assigning the current process to
the Job.

Note that the py.exe launcher runs python.exe in an anonymous Job
that's configured to kill on close (i.e. python.exe is killed when
py.exe exits) and for silent breakaway of child processes. In this
case, prior to Windows 8 (the first version to support nested Job
objects), assigning the current process to a new Job will fail, so
you'll have to run python.exe directly, or use a child process via
subprocess. I prefer the former, since a child process won't be
tethered to the launcher, which could get ugly for console
applications.

    import ctypes
    import winerror, win32api, win32job

    kernel32 = ctypes.WinDLL('kernel32', use_last_error=True)

    MEM_COMMIT = 0x1000
    MEM_RELEASE = 0x8000
    PAGE_READWRITE = 4

    kernel32.VirtualAlloc.restype = ctypes.c_void_p
    kernel32.VirtualAlloc.argtypes = (ctypes.c_void_p, ctypes.c_size_t,
        ctypes.c_ulong, ctypes.c_ulong)
    kernel32.VirtualFree.argtypes = (ctypes.c_void_p, ctypes.c_size_t,
        ctypes.c_ulong)

    hjob = win32job.CreateJobObject(None, "")

    limits = win32job.QueryInformationJobObject(hjob,
                win32job.JobObjectExtendedLimitInformation)
    limits['BasicLimitInformation']['LimitFlags'] |= (
        win32job.JOB_OBJECT_LIMIT_PROCESS_MEMORY)
    limits['ProcessMemoryLimit'] = 2**31
    win32job.SetInformationJobObject(hjob,
        win32job.JobObjectExtendedLimitInformation, limits)

    addr0 = kernel32.VirtualAlloc(None, 2**31, MEM_COMMIT,
                PAGE_READWRITE)
    if addr0:
        mem0_released = kernel32.VirtualFree(addr0, 0, MEM_RELEASE)

    win32job.AssignProcessToJobObject(hjob,
        win32api.GetCurrentProcess())

    addr1 = kernel32.VirtualAlloc(None, 2**31, MEM_COMMIT,
                PAGE_READWRITE)

Result:

    >>> addr0
    2508252315648
    >>> mem0_released
    1
    >>> addr1 is None
    True
    >>> ctypes.get_last_error() == winerror.ERROR_COMMITMENT_LIMIT
    True


More information about the Python-ideas mailing list