From heston.james at coldbeans.co.uk Fri Jul 4 12:34:07 2008 From: heston.james at coldbeans.co.uk (Heston James - Cold Beans) Date: Fri, 4 Jul 2008 11:34:07 +0100 Subject: [DB-SIG] Exeception Handling. Message-ID: <00c701c8ddc1$7d4fd670$77ef8350$@james@coldbeans.co.uk> Good afternoon Chaps, I'm looking for your advice on the best way to handle exceptions on a query and commit. Within my application I have a universal database connection which gets injected into objects which require database access, I then use this connection to create cursors, run queries and commit the changes to the database. Does that sounds like a good approach? Take a look at this query below: # Create the Cursor for the database. cursor = self.datasource.cursor() # Run the insert query to create the entry. cursor.execute(""" INSERT INTO message ( message_id, name, checksum, created, modified ) VALUES ( %s, %s, %s, now(), now() )""", (self.message_id, self.name, self.checksum)) # Commit the change to the database. self.datasource.commit() # Close the cursor for the database. cursor.close() Now to add some exception handling to that query, do I simply wrap the whole code block up in a try/except, and handle any exceptions? or should I be doing a nested try/except which only runs the commit method is the execute runs without exception, and likewise only closes the cursor if it opens successfully? I'd really appreciate your advice on this, I'm relatively new to the world of db implementation using the dbapi and want to make this as water tight as possible. Many thanks, Heston James. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Fri Jul 4 13:29:37 2008 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 04 Jul 2008 13:29:37 +0200 Subject: [DB-SIG] Exeception Handling. In-Reply-To: <00c701c8ddc1$7d4fd670$77ef8350$@james@coldbeans.co.uk> References: <00c701c8ddc1$7d4fd670$77ef8350$@james@coldbeans.co.uk> Message-ID: <486E09A1.1010600@egenix.com> On 2008-07-04 12:34, Heston James - Cold Beans wrote: > Good afternoon Chaps, > > > > I'm looking for your advice on the best way to handle exceptions on a query > and commit. Within my application I have a universal database connection > which gets injected into objects which require database access, I then use > this connection to create cursors, run queries and commit the changes to the > database. Does that sounds like a good approach? Yes. > Take a look at this query below: > > > > # Create the Cursor for the database. > > cursor = self.datasource.cursor() > > > > # Run the insert query to create the entry. > > cursor.execute(""" > > INSERT INTO message ( > > message_id, > > name, > > checksum, > > created, > > modified > > ) > > VALUES ( > > %s, > > %s, > > %s, > > now(), > > now() > > )""", (self.message_id, self.name, > self.checksum)) > > > > # Commit the change to the database. > > self.datasource.commit() > > > > # Close the cursor for the database. > > cursor.close() > > Now to add some exception handling to that query, do I simply wrap the whole > code block up in a try/except, and handle any exceptions? or should I be > doing a nested try/except which only runs the commit method is the execute > runs without exception, and likewise only closes the cursor if it opens > successfully? I'd put the whole transaction code into a try-except: try: # Start of transaction ... do stuff ... except Exception: self.datasource.rollback() raise else: self.datasource.commit() This assures that a successful execution of your code results in the transaction to be committed. In case something goes wrong, the transaction is rolled back and the error reraised so that you can process it at some higher level. > I'd really appreciate your advice on this, I'm relatively new to the world > of db implementation using the dbapi and want to make this as water tight as > possible. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 04 2008) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2008-07-07: EuroPython 2008, Vilnius, Lithuania 2 days to go :::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 From heston.james at coldbeans.co.uk Fri Jul 4 14:03:55 2008 From: heston.james at coldbeans.co.uk (Heston James - Cold Beans) Date: Fri, 4 Jul 2008 13:03:55 +0100 Subject: [DB-SIG] Exeception Handling. In-Reply-To: <486E09A1.1010600@egenix.com> References: <00c701c8ddc1$7d4fd670$77ef8350$@james@coldbeans.co.uk> <486E09A1.1010600@egenix.com> Message-ID: <00e401c8ddce$08618770$19249650$@james@coldbeans.co.uk> > I'd put the whole transaction code into a try-except: > > try: > # Start of transaction > ... do stuff ... > except Exception: > self.datasource.rollback() > raise > else: > self.datasource.commit() > > This assures that a successful execution of your code results in > the transaction to be committed. In case something goes wrong, > the transaction is rolled back and the error reraised so that you > can process it at some higher level. > > I'd really appreciate your advice on this, I'm relatively new to the world > of db implementation using the dbapi and want to make this as water tight as > possible. Hi Marc-Andre, Thank you for your advice, that's really very helpful. I'd like to extend this question a little if you don't mind. Let's say for instance that the persistence of an object isn't just in the database but also on the file system. And for instance this is done like so: # Open the file object in write binary mode. message = open("/files/%s/%s" % (self.message_id, self.name), "wb") # Write the binary to the file. message.write(base64.decodestring(self.binary_data)) # Close the file object. message.close() Now, within my object let's say that I have a method such as save() which I need to save the object to the database (using the code we've already discussed) and also run that file save code to write it to the file system. How would you transaction ALL of this? So if either the file save or the database persist fails then it rolls them both back? So I don't end up with a file without a DB entry or a DB entry without a file? I'd be interested to know A) how would you arrange a try/except block to handle this, and B) would you put this method in the bean or abstract it out into a service layer and have two methods in the bean, one for the file save and one for the database save and wrap those in a transaction at a higher level? Many thanks Heston From mal at egenix.com Fri Jul 4 14:22:38 2008 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 04 Jul 2008 14:22:38 +0200 Subject: [DB-SIG] Exeception Handling. In-Reply-To: <00e401c8ddce$08618770$19249650$@james@coldbeans.co.uk> References: <00c701c8ddc1$7d4fd670$77ef8350$@james@coldbeans.co.uk> <486E09A1.1010600@egenix.com> <00e401c8ddce$08618770$19249650$@james@coldbeans.co.uk> Message-ID: <486E160E.5080709@egenix.com> On 2008-07-04 14:03, Heston James - Cold Beans wrote: >> I'd put the whole transaction code into a try-except: >> >> try: >> # Start of transaction >> ... do stuff ... >> except Exception: >> self.datasource.rollback() >> raise >> else: >> self.datasource.commit() >> >> This assures that a successful execution of your code results in >> the transaction to be committed. In case something goes wrong, >> the transaction is rolled back and the error reraised so that you >> can process it at some higher level. >> >> I'd really appreciate your advice on this, I'm relatively new to the world >> of db implementation using the dbapi and want to make this as water tight > as >> possible. > > Hi Marc-Andre, > > Thank you for your advice, that's really very helpful. I'd like to extend > this question a little if you don't mind. Let's say for instance that the > persistence of an object isn't just in the database but also on the file > system. And for instance this is done like so: > > # Open the file object in write binary mode. > message = open("/files/%s/%s" % (self.message_id, self.name), > "wb") > > # Write the binary to the file. > message.write(base64.decodestring(self.binary_data)) > > # Close the file object. > message.close() > > Now, within my object let's say that I have a method such as save() which I > need to save the object to the database (using the code we've already > discussed) and also run that file save code to write it to the file system. > > How would you transaction ALL of this? So if either the file save or the > database persist fails then it rolls them both back? So I don't end up with > a file without a DB entry or a DB entry without a file? If you want to do this right, you need a transaction manager and then have to use two-phase commits to commit the work on all resources. Depending on the needs and requirements, this can be anything from hard to undoable (not all resources support two-phase commit). > I'd be interested to know A) how would you arrange a try/except block to > handle this, and B) would you put this method in the bean or abstract it out > into a service layer and have two methods in the bean, one for the file save > and one for the database save and wrap those in a transaction at a higher > level? I usually write manager classes for such things which buffer the output and only write to the file system if .commit() is called. A .rollback() would just clear the buffers. If done right, you can also add two-phase commit logic to such a manager class. BTW: The DB-API has recently received an update which defines a two-phase extension for those backends which support this. Have a look at PEP 249 near the end of the page. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 04 2008) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2008-07-07: EuroPython 2008, Vilnius, Lithuania 2 days to go :::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 From mal at egenix.com Fri Jul 4 14:34:08 2008 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 04 Jul 2008 14:34:08 +0200 Subject: [DB-SIG] Exeception Handling. In-Reply-To: <486E160E.5080709@egenix.com> References: <00c701c8ddc1$7d4fd670$77ef8350$@james@coldbeans.co.uk> <486E09A1.1010600@egenix.com> <00e401c8ddce$08618770$19249650$@james@coldbeans.co.uk> <486E160E.5080709@egenix.com> Message-ID: <486E18C0.6000704@egenix.com> On 2008-07-04 14:22, M.-A. Lemburg wrote: > On 2008-07-04 14:03, Heston James - Cold Beans wrote: >>> I'd put the whole transaction code into a try-except: >>> >>> try: >>> # Start of transaction >>> ... do stuff ... >>> except Exception: >>> self.datasource.rollback() >>> raise >>> else: >>> self.datasource.commit() >>> >>> This assures that a successful execution of your code results in >>> the transaction to be committed. In case something goes wrong, >>> the transaction is rolled back and the error reraised so that you >>> can process it at some higher level. >>> >>> I'd really appreciate your advice on this, I'm relatively new to the >>> world >>> of db implementation using the dbapi and want to make this as water >>> tight >> as >>> possible. >> >> Hi Marc-Andre, >> >> Thank you for your advice, that's really very helpful. I'd like to extend >> this question a little if you don't mind. Let's say for instance that the >> persistence of an object isn't just in the database but also on the file >> system. And for instance this is done like so: >> >> # Open the file object in write binary mode. >> message = open("/files/%s/%s" % (self.message_id, self.name), >> "wb") >> >> # Write the binary to the file. >> message.write(base64.decodestring(self.binary_data)) >> >> # Close the file object. >> message.close() >> >> Now, within my object let's say that I have a method such as save() >> which I >> need to save the object to the database (using the code we've already >> discussed) and also run that file save code to write it to the file >> system. >> >> How would you transaction ALL of this? So if either the file save or the >> database persist fails then it rolls them both back? So I don't end up >> with >> a file without a DB entry or a DB entry without a file? > > If you want to do this right, you need a transaction manager and > then have to use two-phase commits to commit the work on all > resources. Depending on the needs and requirements, this can be > anything from hard to undoable (not all resources support two-phase > commit). > >> I'd be interested to know A) how would you arrange a try/except block to >> handle this, and B) would you put this method in the bean or abstract >> it out >> into a service layer and have two methods in the bean, one for the >> file save >> and one for the database save and wrap those in a transaction at a higher >> level? > > I usually write manager classes for such things which buffer the output > and only write to the file system if .commit() is called. A .rollback() > would just clear the buffers. > > If done right, you can also add two-phase commit logic to such a manager > class. > > BTW: The DB-API has recently received an update which defines a > two-phase extension for those backends which support this. Have > a look at PEP 249 near the end of the page. FWIW: I'll be giving a talk on large-scale application design at EuroPython 2008 next week which touches some of these design principles. Here's an older version: http://www.egenix.com/library/presentations/#DevelopingLargeScaleApplicationsInPython -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 04 2008) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2008-07-07: EuroPython 2008, Vilnius, Lithuania 2 days to go :::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 From a.gulati at tsc.nsw.edu.au Tue Jul 8 01:43:01 2008 From: a.gulati at tsc.nsw.edu.au (Gulati, Anil) Date: Tue, 8 Jul 2008 09:43:01 +1000 Subject: [DB-SIG] FW: pep 249 (ImportError python libifsql.so) Message-ID: <6B8B15FBF0734B41A599C5BA824B58310158B1B5@piter.tsc.nsw.edu.au> Issue resolved! LD_LIBRARY_PATH seems to need to be set in /etc/profile. I was only setting it in the shell that I was running the test from. Fixing that immediately changed the error message. Then I just needed to find out about SQLHOSTS file and the service entry in /etc/services and the INFORMIXSERVER variable. It seems hard to find this information in the IBM Informix documentation. Now that all these are set up I am now retrieving data from Informix on Red Hat Linux, and running Python scripts to do so! Thanks very much for your feedback and for being available. Anil Gulati ________________________________ From: Gulati, Anil Sent: Monday, 7 July 2008 12:18 PM To: 'Carsten Haese' Subject: RE: FW: pep 249 (ImportError python libifsql.so) Thanks for your rapid reply, Carsten... That's very nice of you to offer help on these basic installation problems for the IBM installation. All the files are owned by Informix user in Informix group. All files are either rwxr-xr-x or rw-r--r--, directories with x of course. The .so files have execute but the .a files don't. Everything is world readable all the way up to / so I don't see a problem there. Yes, the LD_LIBRARY_PATH was all lower case! My (unmentionable) email client likes to munge stuff for me when I'm not expecting it. This is a totally new box and this is the first install of IBM Informix, so there is no contamination anywhere. I have just re-installed from the same download taking careful note of the options and logging and got the same result. I also did a rm -rf on /usr/local/informix prior to re-installing, which seems to be the IBM recommended uninstall operation. I am thinking of * checking if there is a different version I should be installing instead * re-downloading the installer anyway and trying again fresh * if there is a different environment variable that needs setting with the library paths * trying a static library install (.a only) Anil ________________________________ From: Carsten Haese [mailto:carsten.haese at gmail.com] Sent: Monday, 7 July 2008 11:28 AM To: Gulati, Anil Subject: Re: FW: pep 249 (ImportError python libifsql.so) On Sun, Jul 6, 2008 at 8:58 PM, Gulati, Anil wrote: Ah, just noticed the maintainer addresses in the README... Also found I get the same error when running Informix ESQL/C Client demo In /usr/local/informix/demo/esqlc $ make demo1 $ ./demo1 error while loading shared libraries: libifsql.so: cannot open shared object file: No such file or directory Sorry, it seems this has nothing to do with the Python layer. The Informix client itself is not finding it's own libraries in /usr/local/Informix/lib/esql I tried using export LD_LIBRARY_PATH=/usr/local/informix/lib:/usr/local/Informix/lib/esql But to no avail. The error message means either that the dynamic linker can't *find* the library or that it can't *read* the library. That in turn is usually caused by an incorrect or corrupted installation of the Client SDK, or by an incorrect setting of the library path. If the above is *exactly* what you set LD_LIBRARY_PATH to, that may be your problem right there, since Unix-like OSes are case-sensitive, and you have informix spelled with a lowercase "i" in one case and with an uppercase "I" in the other. If this is just a transcription error and you actually did set LD_LIBRARY_PATH correctly, the next thing I'd check is whether the libraries are where you say they are and whether their permissions are set to rwxr-xr-x, and whether the permissions of the parent directories are sufficiently open. It's also possible that you accidentally corrupted your installation, for example by installing a 32bit CSDK on top of it. If all else fails, try to carefully reinstall the 64bit CSDK in /usr/local/informix. Hope this helps, Carsten. -------------- next part -------------- An HTML attachment was scrubbed... URL: From heston.james at coldbeans.co.uk Thu Jul 10 14:19:02 2008 From: heston.james at coldbeans.co.uk (Heston James - Cold Beans) Date: Thu, 10 Jul 2008 13:19:02 +0100 Subject: [DB-SIG] Saving composed objects. Message-ID: <00e801c8e287$23a9c4d0$6afd4e70$@james@coldbeans.co.uk> Morning Chaps, I'm looking for your thoughts and advice on saving composed objects to the database using the db API. Where do you place your persistence code? For arguments sake let's look at a simple foo/bar example. We have too classes 'foo' and 'bar', and a 'foo' can have many 'bars' within it. How would you handle reading a 'foo' record from the database and saving it back? Making sure that when you read it, you get a list of 'bar' objects inside it, and when saving it saves its relationships and all contained 'bar' objects back to the database? Would you place the queries which save the 'foo' and its 'bars' inside the foo object? Or abstract it out and have 'bar' know how to save itself? If 'bar' does know how to save itself, do we call the save from within 'foo' or abstract that call out to a save_foo() method in a service of something? Having the service loop over the bars and save them? I'm really feeling a little confused about this stuff, saving composed objects to the database is new to me. Thanks guys, Heston -------------- next part -------------- An HTML attachment was scrubbed... URL: From fog at initd.org Thu Jul 10 19:41:40 2008 From: fog at initd.org (Federico Di Gregorio) Date: Thu, 10 Jul 2008 19:41:40 +0200 Subject: [DB-SIG] Saving composed objects. Message-ID: <20080710174140.GA14624@initd.org> On Thu, Jul 10, 2008 at 01:19:02PM +0100, Heston James - Cold Beans wrote: > I'm looking for your thoughts and advice on saving composed objects to the > database using the db API. Where do you place your persistence code? For > arguments sake let's look at a simple foo/bar example. Use one of the many persitence layers available in Python. Just search on f r e s h m e a t for them. ;) federico From alex at moreati.org.uk Thu Jul 10 20:32:50 2008 From: alex at moreati.org.uk (Alex Willmer) Date: Thu, 10 Jul 2008 19:32:50 +0100 Subject: [DB-SIG] Saving composed objects. In-Reply-To: <20080710174140.GA14624@initd.org> References: <20080710174140.GA14624@initd.org> Message-ID: <1215714770.19003.6.camel@martha> On Thu, 2008-07-10 at 19:41 +0200, Federico Di Gregorio wrote: > On Thu, Jul 10, 2008 at 01:19:02PM +0100, Heston James - Cold Beans > wrote: > > I'm looking for your thoughts and advice on saving composed objects > to the > > database using the db API. Where do you place your persistence code? > For > > arguments sake let's look at a simple foo/bar example. > > Use one of the many persitence layers available in Python. > Just search on f r e s h m e a t for them. ;) Also take a look here: http://wiki.python.org/moin/HigherLevelDatabaseProgramming Object Relational Mapping is the term for persisting class instances in a database. The current celebrities are SQLObject and SQLAlchemy. Axiom asynchronous, being closely aligned with the Twisted suite. Alex From info at egenix.com Mon Jul 21 15:26:49 2008 From: info at egenix.com (eGenix Team: M.-A. Lemburg) Date: Mon, 21 Jul 2008 15:26:49 +0200 Subject: [DB-SIG] ANN: eGenix EuroPython 2008 Presentations & Videos: Python DB-API Talk Message-ID: <48848E99.2010004@egenix.com> ________________________________________________________________________ eGenix EuroPython 2008 Presentations & Videos ________________________________________________________________________ We have uploaded our EuroPython 2008 presentations to our website. Learn the concepts behind the Python DB-API and how to design large-scale applications. This announcement is also available on our web-site for online reading: http://www.egenix.com/company/news/EuroPython-2008-Presentations.html ________________________________________________________________________ INTRODUCTION The EuroPython Conference is the one of the premier conferences for Python & Zope users and developers. This year it was being held from the 7th to 9th July in Vilnius, Lithuania. eGenix was one of the founding members of the EuroPython conference team and played a major role in organizing the first EuroPython conference in the year 2002. Since then we have attended every EuroPython conference to meet up face-to-face with the people from the Python & Zope communities and have given regular talks at these conferences. ________________________________________________________________________ TALKS AT EUROPYTHON 2008 We gave the following two talks at the conference. The presentations are available for viewing and download from our Presentations and Talks section: http://www.egenix.com/library/presentations/ As special feature, we have added talk videos in addition to providing the slide PDFs. You can view the talks online if you have the Adobe Flash Player 8 or later installed. * Using the Python Database API The Python Database API (DB-API) is a specification of a module interface that allows interfacing from Python to a relational database. The talk gives a high-level introduction to the concepts used in the Python DB-API and relational databases in general. Connection, cursors and transactions are discussed, and their use in Python database applications is explained. * Designing Large-Scale Applications in Python Python is widely and somewhat inaccurately referred to as a scripting language. While Python is an ideal platform for small scripting tasks, it does in fact cover all the concepts needed for large scale object oriented application development. However, complex applications bring different challenges. This talk draws on eGenix' many years experience with large scale application development using Python as central implementation language and provides a cookbook approach to many of the the problems you face when designing and organizing complex application frameworks. Enjoy, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 21 2008) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ :::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611