Python database of plain text editable by notepad or vi

Ethan Furman ethan at
Sat Mar 27 01:59:21 CET 2010

Jon Clements wrote:
> On 26 Mar, 09:49, James Harris <james.harri... at> wrote:
>>On 25 Mar, 22:56, Jon Clements <jon... at> wrote:
>>>On 25 Mar, 22:40, James Harris <james.harri... at> wrote:
>>>>I am looking to store named pieces of text in a form that can be
>>>>edited by a standard editor such as notepad (under Windows) or vi
>>>>(under Unix) and then pulled into Python as needed. The usual record
>>>>locking and transactions of databases are not required.
>>>>Another way to look at it is to treat the separate files as entries in
>>>>a dictionary. The file name would be the key and the lines of the file
>>>>the value.
>>>>Anyone know of a database (with a Python interface) which will allow
>>>>text files to be treated as database fields? If not I can just write
>>>>it but I thought it best to ask if there was an existing solution
>>>I could be missing something here, but aren't you basically just
>>>talking about an OS's filesystem?
>>For storage, yes. The files would be marked-up text stored in the
>>filesystem. The "dbms" (I use the term loosely!) would provide access
>>to them by some full or partial key mechanism yet to be determined.
>>Read-only access would do - at least for now.
>>>glob or listdir somewhere, then create a dict using the file contents
>>>would meet your criteria, with very little lines of code -- but I
>>>would be interested to know what the use-case was for this... Is it
>>>read completely at start up time, or if each file contains a large
>>>amount of lines and aren't fixed width (or has no other indexing
>>>support without maintenance), then is a complete sequential-scan
>>>required each time, or do you just tell the user to not update when
>>>running (unless I s'pose something along the lines of a SIGHUP for
>>>config files is applicable).
>>All good questions. For now, at least, the files can be read-only and
>>I'd want those on disk to be the master copies at all times. If I was
>>writing it myself I'd probably 'cache' some files in memory and stat
>>them before use. If newer I would reread the file.
> It's hard to bore this group :)
>>>Sorry, just don't understand why you'd want this.
>>I tried to avoid boring folks with the details. I'm toying with some
>>ideas for a way to help generate source code (in various languages,
>>not just Python). If it goes ahead the text files would be mainly
>>marked-up code snippets - with or without symbols that need to be
>>Rather than write one single monolithic app I thought to split it into
>>reusable components. One part being data access could perhaps be an
>>existing database (and I'll take a look at jkn's suggestion).
>>Think of the database as similar to an associative array stored on
>>disk. The only difference is I may want to play fast and loose with
>>the keys in some ways - e.g. check for partial key matches or return a
>>list of part-matched keys. The language name could be part of the key
>>but I'd also need to store variants for specific language versions.
>>I'm not sure yet how it will all pan out. As I say, just throwing
>>around some ideas.
> Thanks for the explanation.
> Who admins and, who's editing this data?
> I couldn't 100% guarantee that I could modify a text file and always
> put the right
> delimiter in the right place and remember to escape the relevant chars
> (and I'm
> probably not the 'average' user).
> Any opposition to just putting it in a 'proper' DB, then 'blobbing'
> the values?
> (or just integrate a procedure/script/function whatever your chosen
> RDBMS calls to choose it).
> Or in some systems, 'externally referencing'... loads of DB's have
> free front-ends,
> and there are lots of Python libraries.
> I think perhaps, all I'm saying is, I'd choose a different approach.
> I'd provide a front-end, rather than choose to re-write the wheel over
> DB's.

Just to provide a counter-viewpoint:

The (one) system I have worked with that was like that (program source 
files kept in blobs in a database) I absolutely hated.  I was stuck with 
using  the tools provided by the app (which, amazingly enough, I also 
hated ;), and unable to use my own tools because because *my* source 
files were _not_ saved as *files*.  Okay, venting over.

My point is, if what you are storing is plain ol' source files, 
providing a way to directly access them is a good thing.  If what you 
are storing is a mangled version, the ability to let the user choose any 
editor to use is a good thing.  :)

My $0.02.


More information about the Python-list mailing list