[Python-ideas] Application awareness of memory storage classes

[M.-A. Lemburg]

On 24.05.2016 04:35, Nick Coghlan wrote:
> On 24 May 2016 at 05:33, R. David Murray <rdmurray at bitdance.com> wrote:
>> On the other hand, the fact that *all* in-block persistent object state
>> gets restored on block abort, regardless of where the exception occurred,
>> could be somewhat confusing to Python programmers.
> 
> It occurs to me that you may want to talk to Mike Bayer and start
> digging into the way SQL Alchemy session objects work, as what you're
> describing here is starting to sound a *lot* like working with the SQL
> Alchemy ORM:
> 
> - you have stateful objects declared as classes (SQL Alchemy models)
> - you have volatile state in memory (the SQL Alchemy session)
> - you have transactional commits to persistent storage (also the session)
> 
> The difference would be that instead of working with a backing
> relational data store, you're working directly with persistent memory.
> It would also be interesting to see how much of this could be emulated
> with mmap and memoryview, permitting such code to have a slower
> fallback in cases where actual NVRAM wasn't available.

SQLAlchemy relies on underlying Python DB-API modules to
work with the database, so in addition to the above you
also have state in objects of those DB-API modules and
these usually reference objects or buffers in the low-level
database interface libraries to which Python has no direct
access.

As soon as you have memory in use which is not fully managed
by Python, I don't think there's any way to implement
transactions on memory in a meaningful way. The possible side
effect in the unmanaged blocks would render such transactions
meaningless, since a rollback in those would still leave you
with the changes in the unmanaged blocks (other parts of the
system).

Now, back on topic: for writing to NVRAM, having a transaction
mechanism in place does make sense, but it would have to
be clear that only the bits stored in NVRAM are subject
to the transaction.

The reason here being that a failure while writing to NVRAM
could potentially cause your machine to no longer boot.

For volatile RAM, at worst, the process will die, but not have
much effect on other running parts of the system, so there
is less incentive to have transactions (unless, of course,
you are deep into STM and want to work around the GIL :-)).

Given that Armin Rigo has been working on STM for years,
I'd suggest to talk to him about challenges and solutions
for transactions on memory.

My take on all this would be to work with NVRAM as block
rather than single memory cells:

allocate a lock on the NVRAM block
try:
 copy the block into DRAM
 run manipulations in DRAM block
 write back DRAM block
finally:
 release lock on NVRAM block

so instead of worrying about a transaction failing while
manipulating NVRAM, you only make sure that you can lock
the NVRAM block and provide an atomic "block write to NVRAM"
functionality.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, May 24 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...           http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
________________________________________________________________________

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
                      http://www.malemburg.com/