[issue13324] fcntl module doesn't support F_NOCACHE (OS X specific) results in high 'inactive' memory performance issues

[Alex Stewart]

New submission from Alex Stewart <alexs.mac at gmail.com>:

----------------------------
ISSUE DESCRIPTION: 
----------------------------
In 2.6.7, 2.7.2 and also HEAD (r 73301) the fcntl module does not support the F_NOCACHE OSX specific mode.  

The affect of this is that the default behaviour (data caching enabled) is used when parsing files etc.  When data caching is enabled, OSX caches the data parsed from a file by python, keeping it available as 'inactive memory' even when it has been freed by python.  In *theory*, this should be fine as OSX *should* recycle the inactive memory as it is required.  

Unfortunately, at least under OSX 10.6.8 (and it seems 10.7) the system will do almost anything to avoid recycling inactive memory, including swallowing up all 'pure' free memory and then paging manically to the disk.  The net affect of this is significantly degraded system performance.

For most users, operating with relatively small files and a large quantity of RAM this issue is probably not that obvious.  However, for various reasons I'm working with files of 5-125+GB and thus it rapidly becomes a major issue.

----------------------------
FIX
----------------------------
Very simply, all the attached patch does it add support for F_NOCACHE to fcntl just like F_FULLFSYNC (another fcntl OSX specific flag) - it's a trivial change that allows you to control whether OSX uses data caching on a file handle, e.g, the following turns OFF data caching for the specified file:

fcntl.fcntl(theFile.fileno(), fcntl.F_NOCACHE, 1)

With this patch in place the inactive memory on my system stays (low) and flat, without it it rises until it maxes out all available memory and then starts paging.

----------
components: Extension Modules
files: fcntlmodule.patch
keywords: patch
messages: 146846
nosy: Alex.Stewart, ned.deily, ronaldoussoren
priority: normal
severity: normal
status: open
title: fcntl module doesn't support F_NOCACHE (OS X specific) results in high 'inactive' memory performance issues
type: resource usage
versions: Python 2.6, Python 2.7
Added file: http://bugs.python.org/file23592/fcntlmodule.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue13324>
_______________________________________