[Python-Dev] Opcode cache in ceval loop
Yury Selivanov
yselivanov.ml at gmail.com
Tue Feb 2 14:23:10 EST 2016
On 2016-02-02 1:45 PM, Serhiy Storchaka wrote:
> On 02.02.16 19:45, Yury Selivanov wrote:
>> On 2016-02-02 12:41 PM, Serhiy Storchaka wrote:
>>> On 01.02.16 21:10, Yury Selivanov wrote:
>>>> To measure the max/average memory impact, I tuned my code to optimize
>>>> *every* code object on *first* run. Then I ran the entire Python test
>>>> suite. Python test suite + standard library both contain around 72395
>>>> code objects, which required 20Mb of memory for caches. The test
>>>> process consumed around 400Mb of memory. Thus, the absolute worst
>>>> case
>>>> scenario, the overhead is about 5%.
>>>
>>> Test process consumes such much memory because few tests creates huge
>>> objects. If exclude these tests (note that tests that requires more
>>> than 1Gb are already excluded by default) and tests that creates a
>>> number of threads (threads consume much memory too), the rest of tests
>>> needs less than 100Mb of memory. Absolute required minimum is about
>>> 25Mb. Thus, the absolute worst case scenario, the overhead is about
>>> 100%.
>> Can you give me the exact configuration of tests (command line to run)
>> that would only consume 25mb?
>
> I don't remember what exact tests consume the most of memory, but
> following tests are failed when run with less than 30Mb of memory:
>
> test___all__ test_asynchat test_asyncio test_bz2 test_capi
> test_concurrent_futures test_ctypes test_decimal test_descr
> test_distutils test_docxmlrpc test_eintr test_email test_fork1
> test_fstring test_ftplib test_functools test_gc test_gdb test_hashlib
> test_httplib test_httpservers test_idle test_imaplib test_import
> test_importlib test_io test_itertools test_json test_lib2to3 test_list
> test_logging test_longexp test_lzma test_mmap
> test_multiprocessing_fork test_multiprocessing_forkserver
> test_multiprocessing_main_handling test_multiprocessing_spawn test_os
> test_pickle test_poplib test_pydoc test_queue test_regrtest
> test_resource test_robotparser test_shutil test_smtplib test_socket
> test_sqlite test_ssl test_subprocess test_tarfile test_tcl test_thread
> test_threaded_import test_threadedtempfile test_threading
> test_threading_local test_threadsignals test_tix test_tk test_tools
> test_ttk_guionly test_ttk_textonly test_tuple test_unicode
> test_urllib2_localnet test_wait3 test_wait4 test_xmlrpc test_zipfile
> test_zlib
Alright, I modified the code to optimize ALL code objects, and ran unit
tests with the above tests excluded:
-- Max process mem (ru_maxrss) = 131858432
-- Opcode cache number of objects = 42109
-- Opcode cache total extra mem = 10901106
And asyncio tests:
-- Max process mem (ru_maxrss) = 57081856
-- Opcode cache number of objects = 4656
-- Opcode cache total extra mem = 1766681
So the absolute worst case for a small asyncio program is 3%, for unit
tests (with the above list excluded) - 8%.
I think it'd be very hard to find a real-life program that consists of
only code objects, and nothing else (no data to work with/process, no
objects with dicts, no threads, basically nothing). Because only for
such a program you would have a 100% memory overhead for the bytecode
cache (when all code objects are optimized).
FWIW, here are stats for asyncio with only hot objects being optimized:
-- Max process mem (ru_maxrss) = 54775808
-- Opcode cache number of objects = 121
-- Opcode cache total extra mem = 43521
Yury
More information about the Python-Dev
mailing list