[Python-ideas] Application awareness of memory storage classes

[R. David Murray]

On Wed, 18 May 2016 16:30:43 +1000, Nick Coghlan <ncoghlan at gmail.com> wrote:
> However, it does suggest "thread local" as an allocator type worth
> considering along with "normal process heap" and "non-volatile
> storage".

Interesting thought.

> That leaves the C++ custom object allocator model, and if I recall
> correctly, allocators there are managed at the *type* level - if you
> want to use a different allocator, you need to declare a new type.

If I understand correctly (my C++ skills are almost non-existent), the
nvml C++ support is using operator overloading on a custom pointer type.
I don't think that's enough by itself, though; the code is still
in progress.

> For Python, I'd suggest looking at going one step further, and
> associating storage management with the *metaclass* - the idea there
> would be to take advantage of the metaclass conflict if someone
> attempts to combine data types with conflicting storage management
> expectations. So you might have "PersistentType", which would itself
> be a normal type instance (stored in RAM), and the classes it created
> would also be stored in RAM, but they'd be configured to handle the
> storage for individual instances differently from normal classes.

I was planning on a base class that implemented checks, but using a
meta class is a good idea.

> I'm not sure if the current descriptor protocol and the GC management
> machinery would give you sufficient access to control all the things
> you care about for this purpose, but it should let you control quite a
> lot of it, and trying it out would give you more info on what's
> currently missing.

I'll give that a whirl and see what happens; thanks for the idea.
But no matter what I do, I'm going to have to write a GC related cleanup
pass when recovering from a crash, and Persistent versions of all the
built-in classes.

--David