How to store data?

Paul D. Fernhout pdfernhout at kurtz-fernhout.com
Sun Aug 24 04:18:53 CEST 2003


Lars-

I don't want to see you cut on bleeding edge software, but, I just 
released an updated version of the Pointrel Data Repository System
   http://sourceforge.net/projects/pointrel/
which could in theory handle such a task (subject to bugs etc.), and I 
could not resist responding to your post. The Pointrel System can handle 
simple locking and simple transactions (although this has only be tested 
recently under GNU/Linux and Python2.3). As a caveat, this Pointrel 
System database bunny may have long pointy, nasty teeth, etc., and so if 
your application is mission critical you should most likely go with a 
more proven solution -- like a convetional relational database (and 
using SQL to access one is a good skill to learn). And possibly if your 
needs are minimal, pickling your dictionaries or using the 'shelve' 
module might just be good enough for a simple application.

Mainly just for fun, and also as another example for people who are 
interested in playing with the related concepts, rather than to try to 
persuade you specifically to try the Pointrel System (since if you are 
not an experienced programmer you would have no way to judge what you 
were getting into, and so my best advice to you specifically would be to 
go with something proven like pickle, shelve, SQL etc.), here is how you 
might solve the storage issue you outline using the Pointrel System:

==================================================
File: "congress.py"
==================================================

from pointrel20030812 import *

Pointrel_initialize("archive_congress")

# add a first attendant -- uses built in unique ID function
# but could be any unique ID string
# each change will be implicitely a seperate transaction
attendantID = Pointrel_generateUniqueID()
Pointrel_add("congress", attendantID, 'object type', 'user')
Pointrel_add("congress", attendantID, 'name', 'Sir Galahad')
Pointrel_add("congress", attendantID, 'address', 'Camelot')
Pointrel_add("congress", attendantID, 'mail', 'Carrierpigeon at england.com')

# add a second attendant, this time as an atomic transaction
attendantID = Pointrel_generateUniqueID()
Pointrel_startTransaction()
Pointrel_add("congress", attendantID, 'object type', 'user')
Pointrel_add("congress", attendantID, 'name', 'Brian')
Pointrel_add("congress", attendantID, 'address', 'Palestine')
Pointrel_add("congress", attendantID, 'mail', 
'fastridingmessenger at rome.com')
Pointrel_finishTransaction()

# add a third attendant, again as a single transaction
attendantID = Pointrel_generateUniqueID()
Pointrel_startTransaction()
Pointrel_add("congress", attendantID, 'object type', 'user')
Pointrel_add("congress", attendantID, 'name', 'Lars')
Pointrel_add("congress", attendantID, 'address', 'Pythonland')
Pointrel_add("congress", attendantID, 'mail', 'lars at example.com')
Pointrel_finishTransaction()

# update a value, as an implicit transaction
attendantID = Pointrel_lastMatch("congress", WILD, 'name', 'Sir Galahad')
Pointrel_add("congress", attendantID, 'address', "Zoot's castle")

# delete an attendant
attendantID = Pointrel_lastMatch("congress", WILD, 'name', 'Brian')
Pointrel_add("congress", attendantID, 'deleted', 'true')

# get a list of all attendants (deleted or not)
attendants = Pointrel_allMatches("congress", WILD, 'object type', 'user')
print '\nAll attendant IDs (including deleted ones):', attendants

# filter out deleted attendants
currentAttendants = filter(lambda user: Pointrel_lastMatch('congress', 
user, 'deleted', WILD) <> 'true', attendants)
print '\nCurrent attendants:', currentAttendants

print "\nAttendant names:"
for attendant in currentAttendants:
     print Pointrel_lastMatch('congress', attendant, 'name', WILD)

print "\nAttendant csv info"
for attendant in currentAttendants:
     name = Pointrel_lastMatch('congress', attendant, 'name', WILD)
     address = Pointrel_lastMatch('congress', attendant, 'address', WILD)
     email = Pointrel_lastMatch('congress', attendant, 'mail', WILD)
     print '"%s","%s","%s"' %(name, address, email)
     # note proper csv output would require escaping embedded quotes etc.

==================================================
Sample output:
==================================================
creating archive file archive_congress.pointrel_database.poi
archive with only header

All attendant IDs (including deleted ones): 
['unique://freevolution:23:43207645167115827', 
'unique://freevolution:25:37062969616920138', 
'unique://freevolution:26:11213192987278475']

Current attendants: ['unique://freevolution:23:43207645167115827', 
'unique://freevolution:26:11213192987278475']

Attendant names:
Sir Galahad
Lars

Attendant csv info
"Sir Galahad","Zoot's castle","Carrierpigeon at england.com"
"Lars","Pythonland","lars at example.com"

==================================================
Database file created: 'archive_congress.pointrel_database.xml'
(There is also a binary file which is maintained for efficiency).
==================================================

<?xml version="1.0" encoding="utf-8" ?>
<P:archive xmlns:P="http://www.pointrel.org/Pointrel20030812/NS/" 
version="20030812.1" archiveID="unique://freevolution:24:35433055818285208">
	<P:transaction>
		<P:triad
			r="1584L"
			s="congress"
			a="unique://freevolution:23:43207645167115827"
			b="object type"
			c="user"
		/>
	</P:transaction>
	<P:transaction>
		<P:triad
			r="1984L"
			s="congress"
			a="unique://freevolution:23:43207645167115827"
			b="name"
			c="Sir Galahad"
		/>
	</P:transaction>
	<P:transaction>
		<P:triad
			r="2376L"
			s="congress"
			a="unique://freevolution:23:43207645167115827"
			b="address"
			c="Camelot"
		/>
	</P:transaction>
	<P:transaction>
		<P:triad
			r="2792L"
			s="congress"
			a="unique://freevolution:23:43207645167115827"
			b="mail"
			c="Carrierpigeon at england.com"
		/>
	</P:transaction>
	<P:transaction>
		<P:triad
			r="3104L"
			s="congress"
			a="unique://freevolution:25:37062969616920138"
			b="object type"
			c="user"
		/>
		<P:triad
			r="3312L"
			s="congress"
			a="unique://freevolution:25:37062969616920138"
			b="name"
			c="Brian"
		/>
		<P:triad
			r="3528L"
			s="congress"
			a="unique://freevolution:25:37062969616920138"
			b="address"
			c="Palestine"
		/>
		<P:triad
			r="3760L"
			s="congress"
			a="unique://freevolution:25:37062969616920138"
			b="mail"
			c="fastridingmessenger at rome.com"
		/>
	</P:transaction>
	<P:transaction>
		<P:triad
			r="4072L"
			s="congress"
			a="unique://freevolution:26:11213192987278475"
			b="object type"
			c="user"
		/>
		<P:triad
			r="4280L"
			s="congress"
			a="unique://freevolution:26:11213192987278475"
			b="name"
			c="Lars"
		/>
		<P:triad
			r="4496L"
			s="congress"
			a="unique://freevolution:26:11213192987278475"
			b="address"
			c="Pythonland"
		/>
		<P:triad
			r="4712L"
			s="congress"
			a="unique://freevolution:26:11213192987278475"
			b="mail"
			c="lars at example.com"
		/>
	</P:transaction>
	<P:transaction>
		<P:triad
			r="4992L"
			s="congress"
			a="unique://freevolution:23:43207645167115827"
			b="address"
			c="Zoot's castle"
		/>
	</P:transaction>
	<P:transaction>
		<P:triad
			r="5384L"
			s="congress"
			a="unique://freevolution:25:37062969616920138"
			b="deleted"
			c="true"
		/>
	</P:transaction>
</P:archive>

[And, as an example of bleeding edge, from testing this, I found a bug 
from a last minute change I made to the unique ID system which had been 
working well for a long time; it turns out that because of an error in 
how I open a support file when it does not exists, the file 
"a_pointrel_uniqueIDCounter.txt" now needs to exist in the local 
directory with a number (e.g. '10') in it to get sequence numbers in 
addition to a random ID component in unique IDs. The system still works 
without this file though -- the unique IDs just aren't as collision 
resistant. Something for an incremental release ASAP though.]

The one plug I'd put for the Pointrel System is that the simple (yet, I 
hope, elegant) data storage approach is designed to be expandable with 
changing needs over time, and usually data storage needs do change, and 
in unexpected ways. Whether the Pointrel System can really deliver on 
that promise remains to be seen. Again, MySql etc. have a proven track 
record handling the sorts of issues you have outlined so far, and are 
probably your best choice.

Anyway, if all this sounds like gibberish, feel free to ignore it! :-)

Best of luck with your project.

--Paul Fernhout

Lars Behrens wrote:
> Hi there!
> 
> For a web project I need a little expert help. I don't have written much 
> code yet, just been fiddling around a bit, testing and planning.
> 
> The web site will have a submission page for attendants of a congress. 
> In a form the user will submit name, mailadress etc. and I plan to store 
> the data in a list of dictionaries:
> 
> People = [{'name' : 'Sir Galahad', 'address' : 'Camelot', 'mail' : 
> 'Carrierpigeon at england.com'}]
> 
> #new attendant:
> People.append({'name' : 'Brian', 'address' : 'Palestine', 'mail' : 
> 'fastridingmessenger at rome.com'})
> 
> The 'lines' can be accessed by their index.
> 
> Now my question: Where do you suggest to store the data? I know that 
> something sql-like would be quite useful but I don't have any knowledge 
> of sql. So I thought about putting the data either into a comma 
> separated file or pickling it.
> 
> I should mention that some other people on different operation systems 
> should be able to import the file (Excel, Open Office) so I tend towards 
> the csv file.
> 
> What would be better?
> 
> And another problem: How to lock the file when several users try to 
> write to it?
> 
> As I'm no very experienced programmer I would appreciate your help,
> thanks in advance
> 
> Cheerz Lars



-----= Posted via Newsfeeds.Com, Uncensored Usenet News =-----
http://www.newsfeeds.com - The #1 Newsgroup Service in the World!
-----==  Over 100,000 Newsgroups - 19 Different Servers! =-----




More information about the Python-list mailing list