On 24 Jul 2018, at 16:57, Jason R. Coombs wrote:
A less disruptive approach could be to create a new abstraction layer, one in which the persistence backend is still pluggable, but that layer becomes responsible for the entirety of persistence, including file storage, metadata storage, replication, and indexing/searching... and one possible implementation of this backend is the MongoDB backend and another is the SQL/file system/whoosh/python replication functionality based on the existing kit.
I took another look at what we have and I think this is totally feasible. The "KeyFS" + "FileStorage" interface we have internally has very little API and it looks like it could be separated. We would provide a new plugin one layer up from the current "Storage" plugins we have.
I have no experience with MongoDB and only very little with NoSQL. That's another reason I like this less disruptive approach, as it could mature on its own and won't disrupt existing installations while it does.
The biggest architectural deviation I'm proposing here is that the replication functionality be pushed down into the persistence layer rather than operating at the application layer.
That's already the case, but the current Storage plugins are below that layer.
Would the project consider this approach? Are there any use cases I've missed in my consideration that would be adversely affected by such an approach? What would it take for you to be enthusiastic about this change?
I'm all for the less disruptive path, as it would also allow deeper changes in the currently existing backends, like using more relational patterns in the SQL backends and search with postgresql.
Regards, Florian Schulze