How to store data?
Paul D. Fernhout
pdfernhout at kurtz-fernhout.com
Sat Aug 23 22:18:53 EDT 2003
Lars-
I don't want to see you cut on bleeding edge software, but, I just
released an updated version of the Pointrel Data Repository System
http://sourceforge.net/projects/pointrel/
which could in theory handle such a task (subject to bugs etc.), and I
could not resist responding to your post. The Pointrel System can handle
simple locking and simple transactions (although this has only be tested
recently under GNU/Linux and Python2.3). As a caveat, this Pointrel
System database bunny may have long pointy, nasty teeth, etc., and so if
your application is mission critical you should most likely go with a
more proven solution -- like a convetional relational database (and
using SQL to access one is a good skill to learn). And possibly if your
needs are minimal, pickling your dictionaries or using the 'shelve'
module might just be good enough for a simple application.
Mainly just for fun, and also as another example for people who are
interested in playing with the related concepts, rather than to try to
persuade you specifically to try the Pointrel System (since if you are
not an experienced programmer you would have no way to judge what you
were getting into, and so my best advice to you specifically would be to
go with something proven like pickle, shelve, SQL etc.), here is how you
might solve the storage issue you outline using the Pointrel System:
==================================================
File: "congress.py"
==================================================
from pointrel20030812 import *
Pointrel_initialize("archive_congress")
# add a first attendant -- uses built in unique ID function
# but could be any unique ID string
# each change will be implicitely a seperate transaction
attendantID = Pointrel_generateUniqueID()
Pointrel_add("congress", attendantID, 'object type', 'user')
Pointrel_add("congress", attendantID, 'name', 'Sir Galahad')
Pointrel_add("congress", attendantID, 'address', 'Camelot')
Pointrel_add("congress", attendantID, 'mail', 'Carrierpigeon at england.com')
# add a second attendant, this time as an atomic transaction
attendantID = Pointrel_generateUniqueID()
Pointrel_startTransaction()
Pointrel_add("congress", attendantID, 'object type', 'user')
Pointrel_add("congress", attendantID, 'name', 'Brian')
Pointrel_add("congress", attendantID, 'address', 'Palestine')
Pointrel_add("congress", attendantID, 'mail',
'fastridingmessenger at rome.com')
Pointrel_finishTransaction()
# add a third attendant, again as a single transaction
attendantID = Pointrel_generateUniqueID()
Pointrel_startTransaction()
Pointrel_add("congress", attendantID, 'object type', 'user')
Pointrel_add("congress", attendantID, 'name', 'Lars')
Pointrel_add("congress", attendantID, 'address', 'Pythonland')
Pointrel_add("congress", attendantID, 'mail', 'lars at example.com')
Pointrel_finishTransaction()
# update a value, as an implicit transaction
attendantID = Pointrel_lastMatch("congress", WILD, 'name', 'Sir Galahad')
Pointrel_add("congress", attendantID, 'address', "Zoot's castle")
# delete an attendant
attendantID = Pointrel_lastMatch("congress", WILD, 'name', 'Brian')
Pointrel_add("congress", attendantID, 'deleted', 'true')
# get a list of all attendants (deleted or not)
attendants = Pointrel_allMatches("congress", WILD, 'object type', 'user')
print '\nAll attendant IDs (including deleted ones):', attendants
# filter out deleted attendants
currentAttendants = filter(lambda user: Pointrel_lastMatch('congress',
user, 'deleted', WILD) <> 'true', attendants)
print '\nCurrent attendants:', currentAttendants
print "\nAttendant names:"
for attendant in currentAttendants:
print Pointrel_lastMatch('congress', attendant, 'name', WILD)
print "\nAttendant csv info"
for attendant in currentAttendants:
name = Pointrel_lastMatch('congress', attendant, 'name', WILD)
address = Pointrel_lastMatch('congress', attendant, 'address', WILD)
email = Pointrel_lastMatch('congress', attendant, 'mail', WILD)
print '"%s","%s","%s"' %(name, address, email)
# note proper csv output would require escaping embedded quotes etc.
==================================================
Sample output:
==================================================
creating archive file archive_congress.pointrel_database.poi
archive with only header
All attendant IDs (including deleted ones):
['unique://freevolution:23:43207645167115827',
'unique://freevolution:25:37062969616920138',
'unique://freevolution:26:11213192987278475']
Current attendants: ['unique://freevolution:23:43207645167115827',
'unique://freevolution:26:11213192987278475']
Attendant names:
Sir Galahad
Lars
Attendant csv info
"Sir Galahad","Zoot's castle","Carrierpigeon at england.com"
"Lars","Pythonland","lars at example.com"
==================================================
Database file created: 'archive_congress.pointrel_database.xml'
(There is also a binary file which is maintained for efficiency).
==================================================
<?xml version="1.0" encoding="utf-8" ?>
<P:archive xmlns:P="http://www.pointrel.org/Pointrel20030812/NS/"
version="20030812.1" archiveID="unique://freevolution:24:35433055818285208">
<P:transaction>
<P:triad
r="1584L"
s="congress"
a="unique://freevolution:23:43207645167115827"
b="object type"
c="user"
/>
</P:transaction>
<P:transaction>
<P:triad
r="1984L"
s="congress"
a="unique://freevolution:23:43207645167115827"
b="name"
c="Sir Galahad"
/>
</P:transaction>
<P:transaction>
<P:triad
r="2376L"
s="congress"
a="unique://freevolution:23:43207645167115827"
b="address"
c="Camelot"
/>
</P:transaction>
<P:transaction>
<P:triad
r="2792L"
s="congress"
a="unique://freevolution:23:43207645167115827"
b="mail"
c="Carrierpigeon at england.com"
/>
</P:transaction>
<P:transaction>
<P:triad
r="3104L"
s="congress"
a="unique://freevolution:25:37062969616920138"
b="object type"
c="user"
/>
<P:triad
r="3312L"
s="congress"
a="unique://freevolution:25:37062969616920138"
b="name"
c="Brian"
/>
<P:triad
r="3528L"
s="congress"
a="unique://freevolution:25:37062969616920138"
b="address"
c="Palestine"
/>
<P:triad
r="3760L"
s="congress"
a="unique://freevolution:25:37062969616920138"
b="mail"
c="fastridingmessenger at rome.com"
/>
</P:transaction>
<P:transaction>
<P:triad
r="4072L"
s="congress"
a="unique://freevolution:26:11213192987278475"
b="object type"
c="user"
/>
<P:triad
r="4280L"
s="congress"
a="unique://freevolution:26:11213192987278475"
b="name"
c="Lars"
/>
<P:triad
r="4496L"
s="congress"
a="unique://freevolution:26:11213192987278475"
b="address"
c="Pythonland"
/>
<P:triad
r="4712L"
s="congress"
a="unique://freevolution:26:11213192987278475"
b="mail"
c="lars at example.com"
/>
</P:transaction>
<P:transaction>
<P:triad
r="4992L"
s="congress"
a="unique://freevolution:23:43207645167115827"
b="address"
c="Zoot's castle"
/>
</P:transaction>
<P:transaction>
<P:triad
r="5384L"
s="congress"
a="unique://freevolution:25:37062969616920138"
b="deleted"
c="true"
/>
</P:transaction>
</P:archive>
[And, as an example of bleeding edge, from testing this, I found a bug
from a last minute change I made to the unique ID system which had been
working well for a long time; it turns out that because of an error in
how I open a support file when it does not exists, the file
"a_pointrel_uniqueIDCounter.txt" now needs to exist in the local
directory with a number (e.g. '10') in it to get sequence numbers in
addition to a random ID component in unique IDs. The system still works
without this file though -- the unique IDs just aren't as collision
resistant. Something for an incremental release ASAP though.]
The one plug I'd put for the Pointrel System is that the simple (yet, I
hope, elegant) data storage approach is designed to be expandable with
changing needs over time, and usually data storage needs do change, and
in unexpected ways. Whether the Pointrel System can really deliver on
that promise remains to be seen. Again, MySql etc. have a proven track
record handling the sorts of issues you have outlined so far, and are
probably your best choice.
Anyway, if all this sounds like gibberish, feel free to ignore it! :-)
Best of luck with your project.
--Paul Fernhout
Lars Behrens wrote:
> Hi there!
>
> For a web project I need a little expert help. I don't have written much
> code yet, just been fiddling around a bit, testing and planning.
>
> The web site will have a submission page for attendants of a congress.
> In a form the user will submit name, mailadress etc. and I plan to store
> the data in a list of dictionaries:
>
> People = [{'name' : 'Sir Galahad', 'address' : 'Camelot', 'mail' :
> 'Carrierpigeon at england.com'}]
>
> #new attendant:
> People.append({'name' : 'Brian', 'address' : 'Palestine', 'mail' :
> 'fastridingmessenger at rome.com'})
>
> The 'lines' can be accessed by their index.
>
> Now my question: Where do you suggest to store the data? I know that
> something sql-like would be quite useful but I don't have any knowledge
> of sql. So I thought about putting the data either into a comma
> separated file or pickling it.
>
> I should mention that some other people on different operation systems
> should be able to import the file (Excel, Open Office) so I tend towards
> the csv file.
>
> What would be better?
>
> And another problem: How to lock the file when several users try to
> write to it?
>
> As I'm no very experienced programmer I would appreciate your help,
> thanks in advance
>
> Cheerz Lars
-----= Posted via Newsfeeds.Com, Uncensored Usenet News =-----
http://www.newsfeeds.com - The #1 Newsgroup Service in the World!
-----== Over 100,000 Newsgroups - 19 Different Servers! =-----
More information about the Python-list
mailing list