On Wed, 22 Nov 2000 00:49:46 -0500 Bill Bumgarner bbum@codefab.com wrote:

I don't think archival should be treated as a database centric operation. The concept of archival falls very naturally into a static hierarchy of collections/directories containing resoruces/files with a bit of additional meta information associated with some resources. This is exactly the kind of information archive that a web server is *designed* to optimally serve. Adding extra layers here or abstracting to a DBI really doesn't buy us much.

While true in general, this ignores the benefits and ease of dynamically generated reports on archives.

Now, perhaps I should talk a little about what I'm spending my copious spare time on lately (other than attempting to chase heisenbugs that crash machines for unknowable reasons).

Among other things I'm attempting to meld the basic concepts of WikiWiki, mail archives, and advogato/slashdot-style commentary systems into a form where the basic tenets of Wiki noun definition are retained such that every individual comment and archive page has a dynamically generated/assigned WikiName and there is a moderately sound karma/trust net running as a meta layer atop this as an abstracted report layer. The fact that further abstractions can be made, and that rule-based reports/abstractions can be created as dynamically filled out WikiPages on the other side of this is just, well, really nice.

I realise that's pretty turgid. Without spending considerable time and verbiage I don't want to spend right now I'm not sure I can do better. If you're interested in details, and I don't mean just casually, please ask away.

But, getting back to the point, to do this properly (ie so that its scalable and reasonably system efficient), especially given the fact that any given data item (be it an archived message, a comment, a report, or whatever) has at least presentation modes each of which has very different surrounding capabilities, the apparancy is that I'm going to have to shove everything into a set of SQL tables.

<shrug>

The current half-implementation is a mix of SQL and static files (mostly wrapped around my current PHPLib template based PHP4+MHonArc based archiving setup).

WebDAV adds the ability to do advanced locking, easy meta information storage, etc... but-- most importantly-- does not take away the very effecient presentation of data naturally present within a filesystem of stuff served by a web server.

I'm actually considering WebDav slightly longer term as a potential Wiki-ish posting method. I'm not at all convinced its a good way to go, but I'm willing to look into it.

As well, a filesystem centric storage/presentation solution-- webdav or raw filesystem-- solves *most* peoples archiving problems *most* of the time.

Very true.

I feel *very strongly* that the archival solution-- whether it be raw messages or decoded messages-- should be centric to storing files in directories and serving files from directories.

In the general case, I agree.

This is *not* to say that the DBI approach isn't the right way to go; if a generic DBI->filesystem, DBI->WebDAV, DBI->DB capable API were put together and was relatively hidden from the user and casual developer, it might be a huge win.

I'd like to see something trivial:

The archiver deconstructs messages into a stasndardised and well documented templatised form on to the file system.

A standard CGI-like toolset is then provided to reconstruct those fragments back into a message (please resend this message to me, I didn't get it the first time), into an HTML page (web archives), etc. This is done by having standard small scripts which do

whatever< by assembling simple call sequences to the toolset.

End users may redefine this by re-writing said scripts, acessing the deconstructions or reconstructions as they wise, to then do anything they want with the resulting archives. This could be SQL inserts, report generation, or black voodoo, it really doesn't matter, and more importantly, Mailman, and Mailman's archiver doesn't need to know about it, only the user does.

The other nice thing is that we don't do it by defining APIs (which then reuires programmers to do useful things with), but by defining small useful atomic tools that may be assembled/ordered in interesting fashions wihtou having to subject the users to Python, or programming much beyond writing shell scripts and editing canned templates.

But an order of magnitude less effecient than downloading the BLOB off of disk via a webserver!

<koh> Greenspun <kof>

Generic access with simple access control is what *most* users/administrators want *most* of the time. More complex/abstracted/portable access is less of a requirement and *a lot* of the people with such requirements also have other issues-- real or imagined-- that dictate that they really just want Mailman to hand the stuff to them as quickly/easily as possible and be done with it.

Precisely.