Application awareness of memory storage classes
I'm currently working on a project for Intel involving Python and directly addressable non-volatile memory. See https://nvdimm.wiki.kernel.org/ for links to the Linux features underlying this work, and http://pmem.io/ for the Intel initiated open source project I'm working with whose intent is to bring support for DAX NVRAM to the application programming layer (nvml currently covers C, with C++ support in process, and there are python bindings for the lower level (non-libpmemobj) libraries). tldr: In the future (and the future is now) application programs will want to be aware of what class of memory they are allocating: normal DRAM, persistent RAM, fast DRAM, (etc?). High level languages like Python that manage memory for the application programmer will need to expose a way for programmers to declare which kind of memory backs an object, and define rules for interactions between objects that reside in different memory types. I'm interested in starting a discussion (not necessarily here, though I think that probably makes sense) about the challenges this presents and how Python-the-language and CPython the implementation might solve them. Feel free to tell me this is the wrong forum, and if so we can a new forum to continue the conversation if enough people are interested. Long version: While I mentioned fast DRAM as well as Persistent RAM above, really it is Persistent RAM that provides the most interesting challenges. This is because you actually have to program differently when your objects are backed by persistent memory. Consider the python script ('pop' is short for Persistent Object Pool): source_account = argv[1].lower() target_account = argv[2].lower() delta = float(argv[3]) pop = pypmemobj('/path/to/pool') if not hasattr(pop.ns, 'accounts'): pop.ns.accounts = dict() for acct in (source_account, target_account): if acct not in pop.ns.accounts: pop.ns.accounts[acct] = 0 pop.ns.accounts[source_account] -= delta pop.ns.accounts[target_account] += delta print("{:-10s} {:10.2f} {:-10s}".format( source_account, pop.ns.accounts[source_account], target_account, pop.ns.accounts[target_account] )) (I haven't bothered to test this code, forgive stupid errors.) This is a simple CLI ap that lets you manage a set of bank accounts: > acct deposits checking 10.00 deposits -10.00 savings 10.00 > acct checking savings 5.00 savings 5.00 checking 5.00 Obviously you'd have a lot more code in a real ap. The point here is that we've got *persistent* account objects, with values that are preserved between runs of the program. There's a big problem with this code. What happens if the program crashes or the machine crashes? Specifically, what happens if the machine crashes between the -= and the += lines? You could end up with a debit from one account with no corresponding credit to the other, leaving your accounts in an inconsistent state with no way to recover. The authors of the nvml library documented at the pmem.io site have discovered via theory and practice that what you need for reliable programming with persistent memory is almost exactly same kind of atomicity that is required when doing multi-threaded programming. (I can sense the asyncio programmers groaning...didn't we just solve that problem and now you are bringing it back? As you'll see below if you combine asyncio and persistent memory, providing the atomicity isn't nearly as bad as managing threading locks; although it does take more thought than writing DRAM-backed procedural code, it is relatively straightforward to reason about.) What we need for the above program fragment to work reliably in the face of crashes is for all of the operations that Python programmers think of as atomic (assignment, dictionary update, etc) to be atomic at the persistent memory level, and to surround any code that does multiple-object updates that are interdependent with a 'transaction' guard. Thus we'd rewrite the end of the above program like this: with pop: pop.ns.accounts[source_account] -= delta pop.ns.accounts[target_account] += delta print("{:-10s} {:10.2f} {:-10s}".format( source_account, pop.ns.accounts[source_account], target_account, pop.ns.accounts[target_account] )) If this were in fact a threaded program, we'd be holding a write lock on pop.ns.accounts in the same scope; but in that case (a) we'd also want to include the print statement and (b) we'd need to pay attention to what lock we were holding. The pmem transaction doesn't have to worry about locks, it is guarding *whatever* memory changes are made during the transaction, no mater which object they are made to. This is what I mean by asyncio + pmem transactions being simpler than threading locks. The transaction that is implied by 'with pop' is the heart of what the nvml libpmemobj does: it provides a transaction scope where any registered changes to the persistent memory will be rolled back on abort, or rolled back when the object pool is next opened if the transaction did not complete before the crash. Of course, in Python as it stands today, the above program would not do anything practical, since when the dict that becomes accounts is allocated, it gets allocated in DRAM, not in persistent memory. To use persistent memory, I'm about to start working on an extension module that basically re-implements most of the Python object classes so that they can be stored in persistent memory instead of RAM. Using my proposed API, the above program would actually look like this: source_account = argv[1].lower() target_account = argv[2].lower() delta = float(argv[3]) pop = pypmemobj('/path/to/pool') if not hasattr(pop.ns, 'accounts'): pop.ns.accounts = PersistentDict(pop) for acct in (source_account, target_account): if acct not in pop.ns.accounts: pop.ns.accounts[acct] = 0.0 with pop: pop.ns.accounts[source_account] -= delta pop.ns.accounts[target_account] += delta print("{:-10s} {:10.2f} {:-10s}".format( source_account, pop.ns.accounts[source_account], target_account, pop.ns.accounts[target_account] )) The difference here is creating a PersistentDict object, and telling it what persistent memory to allocate itself in. But what happens when we do: pop.ns.accounts[acct] = 0.0 We don't want to have to write pop.ns.accounts[acct] = PersistentFloat(pop, 0.0) but if we don't, we'd be trying to store a pointer to an float object that lives in normal RAM into our persistent list, and that pointer would be invalid on the next program run. Instead, for immutable objects we can make a copy in persistent ram when the assignment happens. But we have to reject any attempt to assign a mutable object in to a persistent object unless the mutable is itself persistent...we can't simply make a copy of a mutable object and make a persistent version, because of the following Python idiom: pop.ns.accounts = accounts = dict() If we copied the dict into a PersistentDict automatically, pop.ns.accounts and accounts would point to different objects, and the Python programmer's expectations about his program would be violated. So that's a brief description of what I'm planning to try to implement as an extension module (thoughts and feedback welcome). It's pretty clear that having to re-implement every mutable Python C type, as well as immutable collection types (so that we can handle pointers to other persistent objects correctly) will be a pain. What would it take for Python itself to support the notion of objects being backed by different kinds of memory? It seems to me that it would be possible, but that in CPython at least the cost would be a performance penalty for the all-DRAM case. I'm guessing this is not acceptable, but I'd like to present the notion anyway, in the hopes that someone cleverer than me can see a better way :) At the language level, support would mean two things, I think: there would need to be a way to declare which backing store an object belongs to, and there would need to be a notion of a memory hierarchy where, at least in the case of persistent vs non-persistent RAM, an object higher in the hierarchy could not point to an object lower in the hierarchy that was mutable, and that immutables would be copied from lower to higher. (Here I'm notionally defining "lower" as "closer to the CPU", since RAM is 'just there' whereas persistent memory goes through a driver layer before direct access can get exposed, and fast DRAM would be closer yet to the CPU :). In CPython I envision this being implemented by having every object be associated with a 'storage manager', with the DRAM storage manager obviously being the default. I think that which storage manager an object belongs to can be deduced from its address at runtime, although whether that would be better than storing a pointer is an open question. Object method implementations would then need to be expanded as follows: (1) wrapping any operations that comprise an 'atomic' operation with a 'start transaction' and 'end transaction' call on the storage manager. (2) any memory management function (malloc, etc) would be called indirectly through the storage manager, (3) any object pointer to be stored or retrieved would be passed through an appropriate call on the storage manager to give it an opportunity to block it or transform it, and (4) any memory range to be modified would be reported to the memory manager so that it can be registered with the transaction. I *think* that's the minimum set of changes. Clearly, however, making such calls is going to be less efficient in the DRAM case than the current code, even though the RAM implementation would be a noop. Summary: Providing Python language level support for directly addressable persistent RAM requires addressing the issues of how the application indicates when an object is persistent, and managing the interactions between objects that are persistent and those that are not. In addition, operations that a Python programmer expects to be atomic need to be made atomic from the viewpoint of persistent memory by bracketing them with implicit transactions, and a way to declare an explicit transaction needs to be exposed to the application programmer. These are all interesting design challenges, and I may not have managed to identify all the issues involved. Directly addressable persistent memory is going to become much more common, and probably relatively soon. An extension module such as I plan to work on can provide a way for Python to be used in this space, but are there things that we are able and willing to do to support it more directly in the language and, by implication, in the various Python implementations? Are there design considerations for this extension module that would make it easier or harder for more integrated language support to be added later? I realize this could be viewed as early days to be bringing up this subject, since I haven't even started the extension module yet. On the other hand, it seems to me that the community may have architectural insights that could inform the development, and that starting the conversation now without trying to nail anything down could be very helpful for facilitating the long term support of variant memory types in Python.
On 5/16/2016 8:35 PM, R. David Murray wrote:
How is non-volatile NVRAM different from static SRAM?
tldr: In the future (and the future is now) application programs will want to be aware of what class of memory they are allocating:
What I want is to be, if anything, less aware. I remember fiddling with register declarations in C. Then is was discovered that compilers can allocate registers better than move people, so that 'register' is deprecated for most C programmers. I have never had to worry about the L1, L2, L3 on chip caches, though someone has to. I have long thought that I should be able to randomly access data on disk the same way I would access that same data in RAM, and not have to fiddle with seek and so on. Virtual memory is sort of like this, except that it uses the disk as a volatile* cache for RAM objects. (* Volatile in the sense that when the program ends, the disk space is freed for other use, and is inaccessible even if not.) Whereas I am thinking of using RAM as a cache for a persistent disk object. A possible user API (assuming txt stored as a python string with k bytes per char): cons = str(None, path="c:/user/terry/gutenburg/us_constitution.txt") # now to slice like it was in ram preamble = cons[:cons.find(section_marker)] Perhaps you are pointing to being able to make this possible, from the implementer side. The generic interfaces would be bytes(None, path=) (read only) and bytearray(None, path=) (read-write). A list does not seem like a good candidate for static mem, unless insert and delete are suppressed/unused. [snip] If static objects were **always** aligned in 4-byte boundaries, then the lowest 2 bits could be used to indicate memory type. To not slow down programs, this should be supported by the CPU address decoder. Isn't Intel thinking/working on something like this? -- Terry Jan Reedy
On Tue, 17 May 2016 17:07:59 -0400, Terry Reedy <tjreedy@udel.edu> wrote:
NVRAM retains the data even if it loses power. However, the programming issues involved in using direct memory access to battery backed SRAM should be similar, I think.
Yes, and I think for the hypothetical "fast memory" class that's probably what a dynamic language like python would ideally want to do, even if compiled languages didn't. So really I'm talking about NVRAM, the other was just a hypothetical example of another class of memory besides normal DRAM. For RAM of whatever flavor that retains its value between program invocations, the programmer has to be aware that the data is persistent and program accordingly. Indeed, to some degree it does not matter what the persistent store is, the issue is programming for persistence. The thing that NVRAM brings in to the picture is the motivation to do persistence *not* via serialization and deserialization, but via direct random access to memory that retains its state. Adding support for persistence to the language is actually more generically useful than just the NVRAM realm (consider the hoops the ZODB has to go through to support persistence of Python objects, for example). For current persistence schemes the persistence is file-system based, and the time costs of serialization are swamped by the time costs of the file system access (even when the file system is as fast as possible and/or SRAM or NVRAM based). What is different now, and what makes thinking about this now worthwhile, is that the *direct* access (ie: not driver mediated) to the NVRAM memory locations makes the time cost of serialization swamp the time costs of accessing the persistent data.
mmap + memoryview allows you to do this (in python3), does it not?
This is already possible by using the pynvm bindings to nvml and memoryviews, but it would indeed be interesting to provide a more convenient API, and we've discussed this a bit. There wouldn't be much motivation for any changes to python for that level of support, though, since it could be provided by a couple of new bytes-like classes defined in an external module.
A list does not seem like a good candidate for static mem, unless insert and delete are suppressed/unused.
Why not? The point isn't that the memory is *static*, it's that it is *persistent*. So whatever state your objects are in when your program ends, that's the state they are in when you next connect your program to that pool of objects. It's perfectly sensible to want to update a list and have your changes persist, that's why the ZODB, for example, provides a PersistentList class.
That's an interesting thought, thanks :) I'm not clear how the CPU address decoder would support Python in this context, but I'll ask the Intel folks if there's anything relevant. --David
On 5/18/2016 1:11 PM, R. David Murray wrote:
On Tue, 17 May 2016 17:07:59 -0400, Terry Reedy <tjreedy@udel.edu> wrote:
How is non-volatile NVRAM different from static SRAM?
NVRAM retains the data even if it loses power.
OK, I mis-remembered the battery-backup part, and was thinking of flash memory at 'static' rather than 'non-volatile'. A third term really is needed.
Because each is an O(n) operation. I was thinking of non-volatile writes as been relatively slow and in some sense more costly, but maybe that is not necessarily the case for new designs. -- Terry Jan Reedy
Terry Reedy wrote:
Large flash memories are usually organised into blocks, like a disk. Writing a block is relatively slow, so you'd want to buffer up your changes and write as much as you can in one go. For Python objects, that would probably mean keeping a shadow copy in RAM and flushing it to NV storage periodically. Techniques similar to those used in relational databases might be needed to ensure that the contents of the storage remains consistent in the event of failures. -- Greg
In my infinite stupidity I've replied to this mailing group without properly subscribing. Sorry for the duplicates. Full disclosure: I work at Intel on the set of libraries mentioned above. Our project is fully hardware agnostic - any set of NVDIMMs is going to work. This is a new tier of memory between transient RAM and storage SSD's. It combines the features of both, NVDIMMs provide memory that is both byte addressable and persistent across power failures. The speed and other characterstics are obviously vendor dependent. The term I've seen some people use to describe this new memory is "storage class memory" and I think it really captures the intended use case. There are currently few NVDIMM providers - you might recall recent HP announcement about their new non-volatile memory enabled servers. Also a company called Plexistor is providing software-defined persistent memory solutions. As for Intel offering in this space: the 3D XPoint announcement happend few months back and it was widely covered by the media. Here are two nice presentations: http://www.snia.org/sites/default/files/NVM/2016/presentations/RickCoulson_A... http://www.snia.org/sites/default/files/SDC15_presentations/persistant_mem/J... If anyone finds this topic interesting I recommend checking out our website pmem.io and the going through the presentations presented at various SNIA organized NVM Summits: http://www.snia.org/events/non-volatile-memory-nvm-summit Also, the SNIA programming model is something else to look at after some initial reading: http://www.snia.org/tech_activities/standards/curr_standards/npm It's what the NVML is based on. Hope that this clarifies some things. Piotr W dniu czwartek, 19 maja 2016 09:28:07 UTC+2 użytkownik Greg Ewing napisał:
Terry Reedy wrote:
I gather Multics was like that -- the only way to access a file was to map it into memory, like mmap() in unix. I believe that some unix kernels also work that way internally, with the file I/O layer being built on top of the VM layer. I can see some problems with trying to build that sort of thing too deeply into Python, though. One of them is what happens when two processes map in the same file of Python objects. Some kind of super-GIL that works between processes would be needed to manage the reference counts. Another thing is how to specify that particular objects should be allocated in particular memory areas. That could become quite tricky and tedious with object allocation being both very frequent and very implicit. Another other thing is how to handle references that cross different memory areas. Are they allowed? If so, how do you manage them? If not, how do you detect These problems are of course much reduced if you only allow a restricted set of types to live in one of these areas, such as only atomic types, and lists, arrays, etc. of them. But then it feels more like an I/O storage system like pickle or shelve than it does a VM system. -- Greg
On 19 May 2016 at 10:41, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Another consideration is how to deal with processes that crash (or machines that crash). With transient memory, reference counts are always correct whenever the GIL is released...or the process has just segfaulted/kill -9/whatever. If reference counts are stored in NVRAM, then when a process fails while while it holds a reference from its transient memory to the NVRAM stored object, those references will leak. For me at least, I'd want to treat NVRAM as a form of disk, and not try to use transient ram idioms with it - they seem unsafe. -Rob
On 05/16/2016 05:35 PM, R. David Murray wrote:
I'm currently working on a project for Intel involving Python and directly addressable non-volatile memory.
Sounds cool. The first thing it reminded me of was database transactions. What are the expected speed comparisons? I would imagine that persistent RAM is going to be slower. How much would a typical configuration have? Enough for all a program's data? What are the expected use-cases? For example, loop variables might not be a prime target for being stored in persistent RAM. What are the advantages of using this persistent RAM directly as variables vs. keeping the access as it's own step via the driver layer? -- ~Ethan~
On Tue, 17 May 2016 21:39:07 -0700, Ethan Furman <ethan@stoneleaf.us> wrote:
There are related concerns. For nvml, we are concerned about the A and the D of ACID, but not the C or the I. (There was also some discussion of a cloud-enabled ACID system using the equivalent of software transactional memory, but that's not what nvml is focused on.) To be clear about what I mean by Isolation not being involved: because DAX makes VMRAM behave like RAM (except persistent), when you update the data at a memory address in NVRAM, it is immediately visible to all other threads. Consistency in ACID terms is not something nvml is addressing, that's more of an application level property. So the guarantees NVML is providing are only about the Atomicity and Durability.
What are the expected speed comparisons? I would imagine that persistent RAM is going to be slower.
If I understand correctly the RAM itself is not *necessarily* slower. However, providing the crash-proof atomicity requires bookkeeping overhead that will slow down operations that need to be protected by a transaction (which will be lots of things but by no means everything) compared to DRAM access where you don't care about the atomicity in the face of crashes. The goal is obviously to keep the overhead as small as possible :)
How much would a typical configuration have? Enough for all a program's data?
Yes. I'm sure systems will start with "smaller" amounts of DAX NVRAM initially, and more and more will become common as costs drop. But if I understand correctly we're already talking about significant amounts. In fact, I played around with the idea of just pointing python3's memory allocation at nvram and running it, but that does not get you the transaction level support that would allow crash recovery.
What are the expected use-cases? For example, loop variables might not be a prime target for being stored in persistent RAM.
Probably not, but you might be surprised. The main target would of course be the data the program wants to preserve between runs, but one of the examples at pmem.io is a game program whose state is all in nvram. If the machine crashes in the middle of the game, upon restart you pick up exactly where you left off, including all the positioning information you might consider to be part of the "hot loop" part of the program.
What are the advantages of using this persistent RAM directly as variables vs. keeping the access as it's own step via the driver layer?
Speed. --David
That's an excellent observation ;) This is actually what happend when we first wrote that game - there was a NULL-dereference in the game logic and it caused a 'persistent segmentation fault'. It's really important for programmers to step up when it comes to creating software that uses this type of memory directly. Everyone essentially becomes a file system developer, which is not necessarily a good thing. Our hope is that high level languages like python will also help in this regard by making sure that the programmer cannot shoot himself in the foot as easily as it is possible in C. Piotr 2016-05-20 12:30 GMT+02:00 Steven D'Aprano <steve@pearwood.info>:
On 5/20/2016 7:17 AM, Piotr Balcer wrote: [rearranged to undo top posting]
Unless one uses ctypes or buggy external C-coded modules, seg faulting is already harder in Python than C. One possibility to make using persistent memory easier might be a context manager. The __enter__ method would copy a persistent object to a volatile object and return the volatile object. The body of the with statement would manipulate the volatile object. *If there are no exceptions*, the __exit__ method would 'commit' the changes, presumed to be consisting, by copying back to the persistent object. with persistent(locator) as volatile: <manipulate volatile> Ignoring the possibility of a machine crash during the writeback, a restart after an exception or crash during the block would start with the persistent object as it was before the block, which absent bugs would be a consistent state. -- Terry Jan Reedy
On Fri, 20 May 2016 16:06:03 -0400, Terry Reedy <tjreedy@udel.edu> wrote:
Addressing consistency is what libpmemobj does, and what I'm designing the Python layer on top of it to do. But instead of copying the persistent data to volatile data and back again, what we have is a transaction scope inside of which libpmemobj records all object changes. If we never get to the end of the commit phase, then the changelog is used to roll back all of those changes, leaving the persistent objects in the state they were in before the transaction started. This applies across program restarts as well, which takes care of the "ignoring the possibility of a machine crash during writeback" (ie: that possibility is *not* ignored and dealing with it is in fact a primary goal of the design). So, we'll have: with persistent_store: <manipulate persistent objects> and on exit from that code block either *all* of the changes are made and recorded, or *none* of them are, regardless of what happens inside the transaction, even if the machine crashes in the middle of the commit. Likewise, if a Python exception happens, the transaction commit is aborted, and the state of persistent memory is rolled back. I'm not coming up with anything that persistent->volatile->persistent copying would get you that these transactions don't get you, and transactions are more efficient because you don't have to copy the object data around or do the persistent/volatile conversions. There is one difference: in your version the changes to the volatile versions that happened before the python exception would still exist...but I don't see how that would be useful. On the other hand, the fact that *all* in-block persistent object state gets restored on block abort, regardless of where the exception occurred, could be somewhat confusing to Python programmers. --David
On 24 May 2016 at 05:33, R. David Murray <rdmurray@bitdance.com> wrote:
It occurs to me that you may want to talk to Mike Bayer and start digging into the way SQL Alchemy session objects work, as what you're describing here is starting to sound a *lot* like working with the SQL Alchemy ORM: - you have stateful objects declared as classes (SQL Alchemy models) - you have volatile state in memory (the SQL Alchemy session) - you have transactional commits to persistent storage (also the session) The difference would be that instead of working with a backing relational data store, you're working directly with persistent memory. It would also be interesting to see how much of this could be emulated with mmap and memoryview, permitting such code to have a slower fallback in cases where actual NVRAM wasn't available. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 24.05.2016 04:35, Nick Coghlan wrote:
SQLAlchemy relies on underlying Python DB-API modules to work with the database, so in addition to the above you also have state in objects of those DB-API modules and these usually reference objects or buffers in the low-level database interface libraries to which Python has no direct access. As soon as you have memory in use which is not fully managed by Python, I don't think there's any way to implement transactions on memory in a meaningful way. The possible side effect in the unmanaged blocks would render such transactions meaningless, since a rollback in those would still leave you with the changes in the unmanaged blocks (other parts of the system). Now, back on topic: for writing to NVRAM, having a transaction mechanism in place does make sense, but it would have to be clear that only the bits stored in NVRAM are subject to the transaction. The reason here being that a failure while writing to NVRAM could potentially cause your machine to no longer boot. For volatile RAM, at worst, the process will die, but not have much effect on other running parts of the system, so there is less incentive to have transactions (unless, of course, you are deep into STM and want to work around the GIL :-)). Given that Armin Rigo has been working on STM for years, I'd suggest to talk to him about challenges and solutions for transactions on memory. My take on all this would be to work with NVRAM as block rather than single memory cells: allocate a lock on the NVRAM block try: copy the block into DRAM run manipulations in DRAM block write back DRAM block finally: release lock on NVRAM block so instead of worrying about a transaction failing while manipulating NVRAM, you only make sure that you can lock the NVRAM block and provide an atomic "block write to NVRAM" functionality. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, May 24 2016)
::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/
On 24 May 2016 at 17:59, M.-A. Lemburg <mal@egenix.com> wrote:
I'm not talking about using SQL Alchemy itself for this (as you say, the implementation details of the underlying persistence model are all wrong), I'm talking about studying the way SQL Alchemy session specifically, and the way it manages the in-memory state while a transaction is open, and then flushes that state to the database when the transaction is completed. It's a thought prompted by a presentation Mike gave at PyCon US 2013: https://speakerdeck.com/pyconslides/sqlalchemy-session-in-depth-by-mike-baye... Now, David might investigate that, and decide it's more complexity than is needed given persistent RAM, but it's the nearest already-in-existence thing I know to what I believe he's aiming to build, and it also handles the distinction between persistent state (database row for SQL Alchemy, memory allocated in NVRAM for this) and ephemeral state (elements dynamically allocated in normal RAM for both). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 24 May 2016 at 23:26, Nick Coghlan <ncoghlan@gmail.com> wrote:
Hah, now there's an interesting idea: you actually *could* use SQL Alchemy itself with an sqlite in-memory database, but ensure the latter was allocated in NVRAM :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, 24 May 2016 09:59:42 +0200, "M.-A. Lemburg" <mal@egenix.com> wrote:
In this case all the memory will "managed" at the direction of the Python program (and the extension module). The issue is that while we have transactions on the NVRAM objects, the regular python objects don't get their state restored if the transaction block aborts. Which is part of why I was wondering about what it might look like to integrate awareness of storage classes into the language itself.
Yes, exactly.
The reason here being that a failure while writing to NVRAM could potentially cause your machine to no longer boot.
I think you misunderstand. We're not talking about "regular" NVRAM, we're talking about memory banks that are exposed to user space via a DAX driver that uses file system semantics to set up the mapping from user space to the NVRAM, but after that some kernel magic allows the user space program to write directly to the NVRAM. We're not doing this with the NVRAM involved in booting the machine, it is separate dedicated storage.
STM is a different approach, and equally valid, but not the one the underlying library takes.
He's looking at what we might call the reverse of the type of transaction I'm dealing with. An STM transaction makes all changes pending, and throws them away on conflict. Our transaction makes all changes immediately, and *rolls them back* on *failure*. No conflicts are involved, so the things you have to worry about are different from the things you have to worry about in the STM case. I'm sure there are some commonalities, so it may well be worth talking to Armin, since he's thought deeply about this stuff. I'm being handed the transaction machinery by the underlying library, though, so I "only" have to think about how it impacts the Python level :)
Which is the reverse of what the library actually does. It copies the existing data into an NVRAM rollback log, and then makes the changes to the visible memory (that is, the changes are immediately visible to all threads). The rollback log is then used to undo those changes if the transaction fails. And yes, this means that you need locks around your persistent object updates when doing threaded programming, as I mentioned in my original post. I'm personally also interested in the STM-style case, since that allows you to write multi-access, potentially distributed, DB-like applications. However, that's not what this particular project is about. A language that is supporting persistent storage should support both models, I think, because both are useful for different applications. But the primary difference is what happens during a transaction, so at the language syntax level there is probably no difference. I guess that means there are two different classes of persistent memory from the application's point of view, even if they can be backed by the same physical memory: rollback persistent, and STM persistent. --David
On Tue, 24 May 2016 12:35:26 +1000, Nick Coghlan <ncoghlan@gmail.com> wrote:
There are similarities, so yes it will probably be helpful to look over sqlalchemy transactions in some detail. I've certainly been thinking about them while working on the design of the python layer.
The underlying library itself handles the no-NVRAM fallback using regular mmapped files. --David
On Fri, 20 May 2016 13:17:57 +0200, Piotr Balcer <ppbbalcer@gmail.com> wrote:
Also note that that's a limited subset of the possible ways things might crash, even if it will likely be a common one in practice :) In the general case you'll need a way to clear the memory during development, and/or have a "test memory" setup that will be cleared before each test run and/or unit test. You'll also want ways to simulate crashes "at any point" (similar in principle to how Victor's memory tools can be used to simulate Python running out of memory "at any point"). You may need additional tools built into your application to manage the results of a crash. Exactly what tools you will want for updating the persistent state after various kinds of crashes is likely to be application specific, as is the "automatic recovery" code you'll build in to the application. I'm sure that we'll evolve best practices, test libraries, and helpful tools, for various problem domains, just as we have for working with disk-based databases. First we need to be able to write the code, though. --David
On 17 May 2016 at 10:35, R. David Murray <rdmurray@bitdance.com> wrote:
Feel free to tell me this is the wrong forum, and if so we can a new forum to continue the conversation if enough people are interested.
I think it's the right forum, although a SIG may turn out to be appropriate eventually (then again, there are plenty of arcane topics where we've managed to get by without ever creating a dedicated SIG). [snip background info]
The three main multiple-kinds-of-memory systems I'm personally familiar with are heap management in DSP/BIOS (now TI-RTOS), C++ memory allocators, and Rust's tiered memory management model (mainly for thread locals vs global heap objects). The DSP/BIOS model is probably too simplistic to help us out - it's just "allocate this memory on that heap", and then you're own your own at the semantic level. Rust's tiered memory management is really nice, but also completely foreign to the C-style memory management model that underpins CPython. However, it does suggest "thread local" as an allocator type worth considering along with "normal process heap" and "non-volatile storage". That leaves the C++ custom object allocator model, and if I recall correctly, allocators there are managed at the *type* level - if you want to use a different allocator, you need to declare a new type. For Python, I'd suggest looking at going one step further, and associating storage management with the *metaclass* - the idea there would be to take advantage of the metaclass conflict if someone attempts to combine data types with conflicting storage management expectations. So you might have "PersistentType", which would itself be a normal type instance (stored in RAM), and the classes it created would also be stored in RAM, but they'd be configured to handle the storage for individual instances differently from normal classes. I'm not sure if the current descriptor protocol and the GC management machinery would give you sufficient access to control all the things you care about for this purpose, but it should let you control quite a lot of it, and trying it out would give you more info on what's currently missing. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Wed, 18 May 2016 16:30:43 +1000, Nick Coghlan <ncoghlan@gmail.com> wrote:
Interesting thought.
If I understand correctly (my C++ skills are almost non-existent), the nvml C++ support is using operator overloading on a custom pointer type. I don't think that's enough by itself, though; the code is still in progress.
I was planning on a base class that implemented checks, but using a meta class is a good idea.
I'll give that a whirl and see what happens; thanks for the idea. But no matter what I do, I'm going to have to write a GC related cleanup pass when recovering from a crash, and Persistent versions of all the built-in classes. --David
To build on what David has said about the C++ - this in fact exactly what we've done for our language support :) Long story short, for C++ we introduce two new base types: a persistent smart pointer and a persistent memory resident variable. To complement that there's a support for transactions implemented in two ways: by providing a function to execute as transaction or by using RAII. Here's a very simple queue implemented using those features: https://github.com/pmem/nvml/tree/master/src/examples/libpmemobj/cpp But back on topic of custom allocators: we've prototyped a STL compatible allocator that allowed us to use the std::vector, std::list and other STL containers without any modifications by simply providing the allocator template argument. This was possible because the allocator provides the facility to provide a pointer type argument which is persistent_ptr<T> in our case. Were the CPython implemented in C++ it would probably be possible to implement the collections in a way that creating a specialized ones would be very easy: class PersistentDictionary<T> : Dictionary<T, persistent_alloc> But I guess that's out of the question ;) I'm afraid I won't be much of a help for the python specific stuff but I'm happy to answer any questions about persistent memory programming in general. Piotr
On 5/16/2016 8:35 PM, R. David Murray wrote:
How is non-volatile NVRAM different from static SRAM?
tldr: In the future (and the future is now) application programs will want to be aware of what class of memory they are allocating:
What I want is to be, if anything, less aware. I remember fiddling with register declarations in C. Then is was discovered that compilers can allocate registers better than move people, so that 'register' is deprecated for most C programmers. I have never had to worry about the L1, L2, L3 on chip caches, though someone has to. I have long thought that I should be able to randomly access data on disk the same way I would access that same data in RAM, and not have to fiddle with seek and so on. Virtual memory is sort of like this, except that it uses the disk as a volatile* cache for RAM objects. (* Volatile in the sense that when the program ends, the disk space is freed for other use, and is inaccessible even if not.) Whereas I am thinking of using RAM as a cache for a persistent disk object. A possible user API (assuming txt stored as a python string with k bytes per char): cons = str(None, path="c:/user/terry/gutenburg/us_constitution.txt") # now to slice like it was in ram preamble = cons[:cons.find(section_marker)] Perhaps you are pointing to being able to make this possible, from the implementer side. The generic interfaces would be bytes(None, path=) (read only) and bytearray(None, path=) (read-write). A list does not seem like a good candidate for static mem, unless insert and delete are suppressed/unused. [snip] If static objects were **always** aligned in 4-byte boundaries, then the lowest 2 bits could be used to indicate memory type. To not slow down programs, this should be supported by the CPU address decoder. Isn't Intel thinking/working on something like this? -- Terry Jan Reedy
On Tue, 17 May 2016 17:07:59 -0400, Terry Reedy <tjreedy@udel.edu> wrote:
NVRAM retains the data even if it loses power. However, the programming issues involved in using direct memory access to battery backed SRAM should be similar, I think.
Yes, and I think for the hypothetical "fast memory" class that's probably what a dynamic language like python would ideally want to do, even if compiled languages didn't. So really I'm talking about NVRAM, the other was just a hypothetical example of another class of memory besides normal DRAM. For RAM of whatever flavor that retains its value between program invocations, the programmer has to be aware that the data is persistent and program accordingly. Indeed, to some degree it does not matter what the persistent store is, the issue is programming for persistence. The thing that NVRAM brings in to the picture is the motivation to do persistence *not* via serialization and deserialization, but via direct random access to memory that retains its state. Adding support for persistence to the language is actually more generically useful than just the NVRAM realm (consider the hoops the ZODB has to go through to support persistence of Python objects, for example). For current persistence schemes the persistence is file-system based, and the time costs of serialization are swamped by the time costs of the file system access (even when the file system is as fast as possible and/or SRAM or NVRAM based). What is different now, and what makes thinking about this now worthwhile, is that the *direct* access (ie: not driver mediated) to the NVRAM memory locations makes the time cost of serialization swamp the time costs of accessing the persistent data.
mmap + memoryview allows you to do this (in python3), does it not?
This is already possible by using the pynvm bindings to nvml and memoryviews, but it would indeed be interesting to provide a more convenient API, and we've discussed this a bit. There wouldn't be much motivation for any changes to python for that level of support, though, since it could be provided by a couple of new bytes-like classes defined in an external module.
A list does not seem like a good candidate for static mem, unless insert and delete are suppressed/unused.
Why not? The point isn't that the memory is *static*, it's that it is *persistent*. So whatever state your objects are in when your program ends, that's the state they are in when you next connect your program to that pool of objects. It's perfectly sensible to want to update a list and have your changes persist, that's why the ZODB, for example, provides a PersistentList class.
That's an interesting thought, thanks :) I'm not clear how the CPU address decoder would support Python in this context, but I'll ask the Intel folks if there's anything relevant. --David
On 5/18/2016 1:11 PM, R. David Murray wrote:
On Tue, 17 May 2016 17:07:59 -0400, Terry Reedy <tjreedy@udel.edu> wrote:
How is non-volatile NVRAM different from static SRAM?
NVRAM retains the data even if it loses power.
OK, I mis-remembered the battery-backup part, and was thinking of flash memory at 'static' rather than 'non-volatile'. A third term really is needed.
Because each is an O(n) operation. I was thinking of non-volatile writes as been relatively slow and in some sense more costly, but maybe that is not necessarily the case for new designs. -- Terry Jan Reedy
Terry Reedy wrote:
Large flash memories are usually organised into blocks, like a disk. Writing a block is relatively slow, so you'd want to buffer up your changes and write as much as you can in one go. For Python objects, that would probably mean keeping a shadow copy in RAM and flushing it to NV storage periodically. Techniques similar to those used in relational databases might be needed to ensure that the contents of the storage remains consistent in the event of failures. -- Greg
In my infinite stupidity I've replied to this mailing group without properly subscribing. Sorry for the duplicates. Full disclosure: I work at Intel on the set of libraries mentioned above. Our project is fully hardware agnostic - any set of NVDIMMs is going to work. This is a new tier of memory between transient RAM and storage SSD's. It combines the features of both, NVDIMMs provide memory that is both byte addressable and persistent across power failures. The speed and other characterstics are obviously vendor dependent. The term I've seen some people use to describe this new memory is "storage class memory" and I think it really captures the intended use case. There are currently few NVDIMM providers - you might recall recent HP announcement about their new non-volatile memory enabled servers. Also a company called Plexistor is providing software-defined persistent memory solutions. As for Intel offering in this space: the 3D XPoint announcement happend few months back and it was widely covered by the media. Here are two nice presentations: http://www.snia.org/sites/default/files/NVM/2016/presentations/RickCoulson_A... http://www.snia.org/sites/default/files/SDC15_presentations/persistant_mem/J... If anyone finds this topic interesting I recommend checking out our website pmem.io and the going through the presentations presented at various SNIA organized NVM Summits: http://www.snia.org/events/non-volatile-memory-nvm-summit Also, the SNIA programming model is something else to look at after some initial reading: http://www.snia.org/tech_activities/standards/curr_standards/npm It's what the NVML is based on. Hope that this clarifies some things. Piotr W dniu czwartek, 19 maja 2016 09:28:07 UTC+2 użytkownik Greg Ewing napisał:
Terry Reedy wrote:
I gather Multics was like that -- the only way to access a file was to map it into memory, like mmap() in unix. I believe that some unix kernels also work that way internally, with the file I/O layer being built on top of the VM layer. I can see some problems with trying to build that sort of thing too deeply into Python, though. One of them is what happens when two processes map in the same file of Python objects. Some kind of super-GIL that works between processes would be needed to manage the reference counts. Another thing is how to specify that particular objects should be allocated in particular memory areas. That could become quite tricky and tedious with object allocation being both very frequent and very implicit. Another other thing is how to handle references that cross different memory areas. Are they allowed? If so, how do you manage them? If not, how do you detect These problems are of course much reduced if you only allow a restricted set of types to live in one of these areas, such as only atomic types, and lists, arrays, etc. of them. But then it feels more like an I/O storage system like pickle or shelve than it does a VM system. -- Greg
On 19 May 2016 at 10:41, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Another consideration is how to deal with processes that crash (or machines that crash). With transient memory, reference counts are always correct whenever the GIL is released...or the process has just segfaulted/kill -9/whatever. If reference counts are stored in NVRAM, then when a process fails while while it holds a reference from its transient memory to the NVRAM stored object, those references will leak. For me at least, I'd want to treat NVRAM as a form of disk, and not try to use transient ram idioms with it - they seem unsafe. -Rob
On 05/16/2016 05:35 PM, R. David Murray wrote:
I'm currently working on a project for Intel involving Python and directly addressable non-volatile memory.
Sounds cool. The first thing it reminded me of was database transactions. What are the expected speed comparisons? I would imagine that persistent RAM is going to be slower. How much would a typical configuration have? Enough for all a program's data? What are the expected use-cases? For example, loop variables might not be a prime target for being stored in persistent RAM. What are the advantages of using this persistent RAM directly as variables vs. keeping the access as it's own step via the driver layer? -- ~Ethan~
On Tue, 17 May 2016 21:39:07 -0700, Ethan Furman <ethan@stoneleaf.us> wrote:
There are related concerns. For nvml, we are concerned about the A and the D of ACID, but not the C or the I. (There was also some discussion of a cloud-enabled ACID system using the equivalent of software transactional memory, but that's not what nvml is focused on.) To be clear about what I mean by Isolation not being involved: because DAX makes VMRAM behave like RAM (except persistent), when you update the data at a memory address in NVRAM, it is immediately visible to all other threads. Consistency in ACID terms is not something nvml is addressing, that's more of an application level property. So the guarantees NVML is providing are only about the Atomicity and Durability.
What are the expected speed comparisons? I would imagine that persistent RAM is going to be slower.
If I understand correctly the RAM itself is not *necessarily* slower. However, providing the crash-proof atomicity requires bookkeeping overhead that will slow down operations that need to be protected by a transaction (which will be lots of things but by no means everything) compared to DRAM access where you don't care about the atomicity in the face of crashes. The goal is obviously to keep the overhead as small as possible :)
How much would a typical configuration have? Enough for all a program's data?
Yes. I'm sure systems will start with "smaller" amounts of DAX NVRAM initially, and more and more will become common as costs drop. But if I understand correctly we're already talking about significant amounts. In fact, I played around with the idea of just pointing python3's memory allocation at nvram and running it, but that does not get you the transaction level support that would allow crash recovery.
What are the expected use-cases? For example, loop variables might not be a prime target for being stored in persistent RAM.
Probably not, but you might be surprised. The main target would of course be the data the program wants to preserve between runs, but one of the examples at pmem.io is a game program whose state is all in nvram. If the machine crashes in the middle of the game, upon restart you pick up exactly where you left off, including all the positioning information you might consider to be part of the "hot loop" part of the program.
What are the advantages of using this persistent RAM directly as variables vs. keeping the access as it's own step via the driver layer?
Speed. --David
That's an excellent observation ;) This is actually what happend when we first wrote that game - there was a NULL-dereference in the game logic and it caused a 'persistent segmentation fault'. It's really important for programmers to step up when it comes to creating software that uses this type of memory directly. Everyone essentially becomes a file system developer, which is not necessarily a good thing. Our hope is that high level languages like python will also help in this regard by making sure that the programmer cannot shoot himself in the foot as easily as it is possible in C. Piotr 2016-05-20 12:30 GMT+02:00 Steven D'Aprano <steve@pearwood.info>:
On 5/20/2016 7:17 AM, Piotr Balcer wrote: [rearranged to undo top posting]
Unless one uses ctypes or buggy external C-coded modules, seg faulting is already harder in Python than C. One possibility to make using persistent memory easier might be a context manager. The __enter__ method would copy a persistent object to a volatile object and return the volatile object. The body of the with statement would manipulate the volatile object. *If there are no exceptions*, the __exit__ method would 'commit' the changes, presumed to be consisting, by copying back to the persistent object. with persistent(locator) as volatile: <manipulate volatile> Ignoring the possibility of a machine crash during the writeback, a restart after an exception or crash during the block would start with the persistent object as it was before the block, which absent bugs would be a consistent state. -- Terry Jan Reedy
On Fri, 20 May 2016 16:06:03 -0400, Terry Reedy <tjreedy@udel.edu> wrote:
Addressing consistency is what libpmemobj does, and what I'm designing the Python layer on top of it to do. But instead of copying the persistent data to volatile data and back again, what we have is a transaction scope inside of which libpmemobj records all object changes. If we never get to the end of the commit phase, then the changelog is used to roll back all of those changes, leaving the persistent objects in the state they were in before the transaction started. This applies across program restarts as well, which takes care of the "ignoring the possibility of a machine crash during writeback" (ie: that possibility is *not* ignored and dealing with it is in fact a primary goal of the design). So, we'll have: with persistent_store: <manipulate persistent objects> and on exit from that code block either *all* of the changes are made and recorded, or *none* of them are, regardless of what happens inside the transaction, even if the machine crashes in the middle of the commit. Likewise, if a Python exception happens, the transaction commit is aborted, and the state of persistent memory is rolled back. I'm not coming up with anything that persistent->volatile->persistent copying would get you that these transactions don't get you, and transactions are more efficient because you don't have to copy the object data around or do the persistent/volatile conversions. There is one difference: in your version the changes to the volatile versions that happened before the python exception would still exist...but I don't see how that would be useful. On the other hand, the fact that *all* in-block persistent object state gets restored on block abort, regardless of where the exception occurred, could be somewhat confusing to Python programmers. --David
On 24 May 2016 at 05:33, R. David Murray <rdmurray@bitdance.com> wrote:
It occurs to me that you may want to talk to Mike Bayer and start digging into the way SQL Alchemy session objects work, as what you're describing here is starting to sound a *lot* like working with the SQL Alchemy ORM: - you have stateful objects declared as classes (SQL Alchemy models) - you have volatile state in memory (the SQL Alchemy session) - you have transactional commits to persistent storage (also the session) The difference would be that instead of working with a backing relational data store, you're working directly with persistent memory. It would also be interesting to see how much of this could be emulated with mmap and memoryview, permitting such code to have a slower fallback in cases where actual NVRAM wasn't available. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 24.05.2016 04:35, Nick Coghlan wrote:
SQLAlchemy relies on underlying Python DB-API modules to work with the database, so in addition to the above you also have state in objects of those DB-API modules and these usually reference objects or buffers in the low-level database interface libraries to which Python has no direct access. As soon as you have memory in use which is not fully managed by Python, I don't think there's any way to implement transactions on memory in a meaningful way. The possible side effect in the unmanaged blocks would render such transactions meaningless, since a rollback in those would still leave you with the changes in the unmanaged blocks (other parts of the system). Now, back on topic: for writing to NVRAM, having a transaction mechanism in place does make sense, but it would have to be clear that only the bits stored in NVRAM are subject to the transaction. The reason here being that a failure while writing to NVRAM could potentially cause your machine to no longer boot. For volatile RAM, at worst, the process will die, but not have much effect on other running parts of the system, so there is less incentive to have transactions (unless, of course, you are deep into STM and want to work around the GIL :-)). Given that Armin Rigo has been working on STM for years, I'd suggest to talk to him about challenges and solutions for transactions on memory. My take on all this would be to work with NVRAM as block rather than single memory cells: allocate a lock on the NVRAM block try: copy the block into DRAM run manipulations in DRAM block write back DRAM block finally: release lock on NVRAM block so instead of worrying about a transaction failing while manipulating NVRAM, you only make sure that you can lock the NVRAM block and provide an atomic "block write to NVRAM" functionality. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, May 24 2016)
::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/
On 24 May 2016 at 17:59, M.-A. Lemburg <mal@egenix.com> wrote:
I'm not talking about using SQL Alchemy itself for this (as you say, the implementation details of the underlying persistence model are all wrong), I'm talking about studying the way SQL Alchemy session specifically, and the way it manages the in-memory state while a transaction is open, and then flushes that state to the database when the transaction is completed. It's a thought prompted by a presentation Mike gave at PyCon US 2013: https://speakerdeck.com/pyconslides/sqlalchemy-session-in-depth-by-mike-baye... Now, David might investigate that, and decide it's more complexity than is needed given persistent RAM, but it's the nearest already-in-existence thing I know to what I believe he's aiming to build, and it also handles the distinction between persistent state (database row for SQL Alchemy, memory allocated in NVRAM for this) and ephemeral state (elements dynamically allocated in normal RAM for both). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 24 May 2016 at 23:26, Nick Coghlan <ncoghlan@gmail.com> wrote:
Hah, now there's an interesting idea: you actually *could* use SQL Alchemy itself with an sqlite in-memory database, but ensure the latter was allocated in NVRAM :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, 24 May 2016 09:59:42 +0200, "M.-A. Lemburg" <mal@egenix.com> wrote:
In this case all the memory will "managed" at the direction of the Python program (and the extension module). The issue is that while we have transactions on the NVRAM objects, the regular python objects don't get their state restored if the transaction block aborts. Which is part of why I was wondering about what it might look like to integrate awareness of storage classes into the language itself.
Yes, exactly.
The reason here being that a failure while writing to NVRAM could potentially cause your machine to no longer boot.
I think you misunderstand. We're not talking about "regular" NVRAM, we're talking about memory banks that are exposed to user space via a DAX driver that uses file system semantics to set up the mapping from user space to the NVRAM, but after that some kernel magic allows the user space program to write directly to the NVRAM. We're not doing this with the NVRAM involved in booting the machine, it is separate dedicated storage.
STM is a different approach, and equally valid, but not the one the underlying library takes.
He's looking at what we might call the reverse of the type of transaction I'm dealing with. An STM transaction makes all changes pending, and throws them away on conflict. Our transaction makes all changes immediately, and *rolls them back* on *failure*. No conflicts are involved, so the things you have to worry about are different from the things you have to worry about in the STM case. I'm sure there are some commonalities, so it may well be worth talking to Armin, since he's thought deeply about this stuff. I'm being handed the transaction machinery by the underlying library, though, so I "only" have to think about how it impacts the Python level :)
Which is the reverse of what the library actually does. It copies the existing data into an NVRAM rollback log, and then makes the changes to the visible memory (that is, the changes are immediately visible to all threads). The rollback log is then used to undo those changes if the transaction fails. And yes, this means that you need locks around your persistent object updates when doing threaded programming, as I mentioned in my original post. I'm personally also interested in the STM-style case, since that allows you to write multi-access, potentially distributed, DB-like applications. However, that's not what this particular project is about. A language that is supporting persistent storage should support both models, I think, because both are useful for different applications. But the primary difference is what happens during a transaction, so at the language syntax level there is probably no difference. I guess that means there are two different classes of persistent memory from the application's point of view, even if they can be backed by the same physical memory: rollback persistent, and STM persistent. --David
On Tue, 24 May 2016 12:35:26 +1000, Nick Coghlan <ncoghlan@gmail.com> wrote:
There are similarities, so yes it will probably be helpful to look over sqlalchemy transactions in some detail. I've certainly been thinking about them while working on the design of the python layer.
The underlying library itself handles the no-NVRAM fallback using regular mmapped files. --David
On Fri, 20 May 2016 13:17:57 +0200, Piotr Balcer <ppbbalcer@gmail.com> wrote:
Also note that that's a limited subset of the possible ways things might crash, even if it will likely be a common one in practice :) In the general case you'll need a way to clear the memory during development, and/or have a "test memory" setup that will be cleared before each test run and/or unit test. You'll also want ways to simulate crashes "at any point" (similar in principle to how Victor's memory tools can be used to simulate Python running out of memory "at any point"). You may need additional tools built into your application to manage the results of a crash. Exactly what tools you will want for updating the persistent state after various kinds of crashes is likely to be application specific, as is the "automatic recovery" code you'll build in to the application. I'm sure that we'll evolve best practices, test libraries, and helpful tools, for various problem domains, just as we have for working with disk-based databases. First we need to be able to write the code, though. --David
On 17 May 2016 at 10:35, R. David Murray <rdmurray@bitdance.com> wrote:
Feel free to tell me this is the wrong forum, and if so we can a new forum to continue the conversation if enough people are interested.
I think it's the right forum, although a SIG may turn out to be appropriate eventually (then again, there are plenty of arcane topics where we've managed to get by without ever creating a dedicated SIG). [snip background info]
The three main multiple-kinds-of-memory systems I'm personally familiar with are heap management in DSP/BIOS (now TI-RTOS), C++ memory allocators, and Rust's tiered memory management model (mainly for thread locals vs global heap objects). The DSP/BIOS model is probably too simplistic to help us out - it's just "allocate this memory on that heap", and then you're own your own at the semantic level. Rust's tiered memory management is really nice, but also completely foreign to the C-style memory management model that underpins CPython. However, it does suggest "thread local" as an allocator type worth considering along with "normal process heap" and "non-volatile storage". That leaves the C++ custom object allocator model, and if I recall correctly, allocators there are managed at the *type* level - if you want to use a different allocator, you need to declare a new type. For Python, I'd suggest looking at going one step further, and associating storage management with the *metaclass* - the idea there would be to take advantage of the metaclass conflict if someone attempts to combine data types with conflicting storage management expectations. So you might have "PersistentType", which would itself be a normal type instance (stored in RAM), and the classes it created would also be stored in RAM, but they'd be configured to handle the storage for individual instances differently from normal classes. I'm not sure if the current descriptor protocol and the GC management machinery would give you sufficient access to control all the things you care about for this purpose, but it should let you control quite a lot of it, and trying it out would give you more info on what's currently missing. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Wed, 18 May 2016 16:30:43 +1000, Nick Coghlan <ncoghlan@gmail.com> wrote:
Interesting thought.
If I understand correctly (my C++ skills are almost non-existent), the nvml C++ support is using operator overloading on a custom pointer type. I don't think that's enough by itself, though; the code is still in progress.
I was planning on a base class that implemented checks, but using a meta class is a good idea.
I'll give that a whirl and see what happens; thanks for the idea. But no matter what I do, I'm going to have to write a GC related cleanup pass when recovering from a crash, and Persistent versions of all the built-in classes. --David
To build on what David has said about the C++ - this in fact exactly what we've done for our language support :) Long story short, for C++ we introduce two new base types: a persistent smart pointer and a persistent memory resident variable. To complement that there's a support for transactions implemented in two ways: by providing a function to execute as transaction or by using RAII. Here's a very simple queue implemented using those features: https://github.com/pmem/nvml/tree/master/src/examples/libpmemobj/cpp But back on topic of custom allocators: we've prototyped a STL compatible allocator that allowed us to use the std::vector, std::list and other STL containers without any modifications by simply providing the allocator template argument. This was possible because the allocator provides the facility to provide a pointer type argument which is persistent_ptr<T> in our case. Were the CPython implemented in C++ it would probably be possible to implement the collections in a way that creating a specialized ones would be very easy: class PersistentDictionary<T> : Dictionary<T, persistent_alloc> But I guess that's out of the question ;) I'm afraid I won't be much of a help for the python specific stuff but I'm happy to answer any questions about persistent memory programming in general. Piotr
participants (9)
-
Ethan Furman -
Greg Ewing -
M.-A. Lemburg -
Nick Coghlan -
Piotr Balcer -
R. David Murray -
Robert Collins -
Steven D'Aprano -
Terry Reedy