From mark at microenh.com Wed Sep 3 20:32:36 2008 From: mark at microenh.com (Mark Erbaugh) Date: Wed, 03 Sep 2008 14:32:36 -0400 Subject: [CentralOH] Pickling data into a database Message-ID: <1220466757.14287.13.camel@P1900> I'm working on a Python database application where the type and number of fields that need to be saved vary from row to row. It is an application that allows the user to configure my client's products to their requirements. Depending on the specifics of the application, the required data changes. In the past, being an old database designer, I just designed a database with enough columns to hold all the possible parameters. However, the other day I had a "brainstorm". Much of this data does not need to be queried from the database. I could streamline my database design and store this variable data in Pickle'd format in a text field. The data that does need to be queried would be kept in database fields of the appropriate type. One of the outputs of this application is a performance curve that needs to be saved. In the current application, I create a one to many child table that has the X and Y coordinates of the curve (one pair per row). I could easily Pickle a sequence of tuples. There are usually only a dozen or fewer points on the curve (I think the worst case is about 40). Is this a good approach? What are the potential pitfalls? One pitfall that just came to mind is that this limits the ability of non-Python applications to process the data. Maybe an alternative would be to convert the data to XML instead of Pickling it? Thanks, Mark From pete at osc.edu Wed Sep 3 20:57:02 2008 From: pete at osc.edu (Pete Carswell) Date: Wed, 03 Sep 2008 14:57:02 -0400 Subject: [CentralOH] using python in web applications... In-Reply-To: <293817.77933.qm@web80014.mail.sp1.yahoo.com> References: <1217991094.23565.6.camel@P1900> <293817.77933.qm@web80014.mail.sp1.yahoo.com> Message-ID: <48BEA5C0.A14A.0000.0@osc.edu> I was interested in any references about use of python applications as a web portal interface. A quick google search turned up http://www.pythonthreads.com/news/latest/python-web-application-using-google-app-engine.html but I would be interested in other references. Web portals are, admittedly, not my best subject. Thanks. pete From brian.costlow at gmail.com Wed Sep 3 21:37:26 2008 From: brian.costlow at gmail.com (Brian Costlow) Date: Wed, 3 Sep 2008 15:37:26 -0400 Subject: [CentralOH] Pickling data into a database In-Reply-To: <1220466757.14287.13.camel@P1900> References: <1220466757.14287.13.camel@P1900> Message-ID: <89d8b1b00809031237p5affeec1j445095c1e20383e1@mail.gmail.com> On Wed, Sep 3, 2008 at 2:32 PM, Mark Erbaugh wrote: > I'm working on a Python database application where the type and number > of fields that need to be saved vary from row to row. SNIP > However, the other day I had a "brainstorm". Much of this data does not > need to be queried from the database. I could streamline my database > design and store this variable data in Pickle'd format in a text field.0). > > Is this a good approach? What are the potential pitfalls? > > One pitfall that just came to mind is that this limits the ability of > non-Python applications to process the data. Maybe an alternative would > be to convert the data to XML instead of Pickling it? > I think you hit one of the biggies there. However, if all the data is as simple as the curve pairs, XML may be overkill. You could, for instance, represent the pairs as a simple space delimited string. On the other hand, if the data is going to be around for a decade, XML gives you a way to add some semantic info so that some theoretical future developer using the next big language knows what the data means. You need to think through these kind of trade-offs. Also, in my experience, sooner or later someone will want to search this data, no matter how unlikely it seems now. I built an app that ingests XML docs, parsed out the stuff the customer was interested in, and inserted it into a postgres db. After playing around with speed of retrieving the entire XML file (these were multi MB files) from db fields or disk files, I wrote the XML itself to disk and indexed the location in the db. Now of course the end users want to search on things in the XML, but not db fields that they told me earlier they'd never search. So I'm trying to decide whether to move the XML back to the db and implement text search, or if we can do something via Xquery (which I know very little about). So, my caveats: don't do something python centric for data going into the db that might get used by non python apps. Be careful about making a 'this won't ever be searched' decision. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.costlow at gmail.com Wed Sep 3 21:51:16 2008 From: brian.costlow at gmail.com (Brian Costlow) Date: Wed, 3 Sep 2008 15:51:16 -0400 Subject: [CentralOH] using python in web applications... In-Reply-To: <48BEA5C0.A14A.0000.0@osc.edu> References: <1217991094.23565.6.camel@P1900> <293817.77933.qm@web80014.mail.sp1.yahoo.com> <48BEA5C0.A14A.0000.0@osc.edu> Message-ID: <89d8b1b00809031251g53936c79ha81e035ed28376e5@mail.gmail.com> Hi Pete, Web portal can be a somewhat ambiguous term. To some, it means Yahoo and similar sites that provide a single point of entry to lots of online services. (Or corporate or government sites that serve similar purpose). To others, it means content management apps like Drupal or Mambo. Yet again, especially in the Java enterprisey world it can mean a product that is a combination content manager and application framework that lets you use shared authentication and access control for web apps that plug into the portal. If you are looking for content managers, Plone is the granddaddy of python content managers. www.plone.org If you are looking for frameworks to build a portal of your own, try zope at zope.org, django at www.djangoproject.com or turbogears, at turbogears.org. There are also some up and coming content managers being built on django or tg, you can find links off of those sites. Good luck. Brian On Wed, Sep 3, 2008 at 2:57 PM, Pete Carswell wrote: > I was interested in any references about use of python applications as a > web portal interface. A quick google search turned up > > > http://www.pythonthreads.com/news/latest/python-web-application-using-google-app-engine.html > > but I would be interested in other references. Web portals are, admittedly, > not my best subject. Thanks. > > pete > > > _______________________________________________ > CentralOH mailing list > CentralOH at python.org > http://mail.python.org/mailman/listinfo/centraloh > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pete at osc.edu Wed Sep 3 22:35:45 2008 From: pete at osc.edu (Pete Carswell) Date: Wed, 03 Sep 2008 16:35:45 -0400 Subject: [CentralOH] using python in web applications... In-Reply-To: <89d8b1b00809031251g53936c79ha81e035ed28376e5@mail.gmail.com> References: <1217991094.23565.6.camel@P1900> <293817.77933.qm@web80014.mail.sp1.yahoo.com> <48BEA5C0.A14A.0000.0@osc.edu> <89d8b1b00809031251g53936c79ha81e035ed28376e5@mail.gmail.com> Message-ID: <48BEBCE3.A14A.0000.0@osc.edu> Thanks, Brian. I thought it might be helpful to include a segment of email I had sent to the Paraview forum. Initially I was inquiring about embedding Paraview in a web portal. What we want is greater control with the sci viz processes in generating viewable images from the CFD parameters. Python is also an API for both VTK and Paraview. Here is that email: "I would like to clarify how we would like to use paraview in this fashion, in the hope that it might generate some response. Currently, there is a CFD course that is taught online. Parameters to a particular problem are entered in a web page and submitted for a run on our HPC cluster. Images of the solution are generated and supplied to the student through the web interface. The batch script runs a processed script generated from Paraview 2.6. We currently also have Paraview 3 installed on the cluster. We would like to have more flexibility in processing the images after the solution is submitted, i.e. switching the modes of the data, vector or scalar data, and panning and zooming on the images, and, subsequently, handled by the students in the web application. I hope this describes our intentions better. Would appreciate any feedback." I hope this might give someone some ideas. Thanks. pete >>> "Brian Costlow" 9/3/2008 3:51 PM >>> Hi Pete, Web portal can be a somewhat ambiguous term. To some, it means Yahoo and similar sites that provide a single point of entry to lots of online services. (Or corporate or government sites that serve similar purpose). To others, it means content management apps like Drupal or Mambo. Yet again, especially in the Java enterprisey world it can mean a product that is a combination content manager and application framework that lets you use shared authentication and access control for web apps that plug into the portal. If you are looking for content managers, Plone is the granddaddy of python content managers. www.plone.org If you are looking for frameworks to build a portal of your own, try zope at zope.org, django at www.djangoproject.com or turbogears, at turbogears.org. There are also some up and coming content managers being built on django or tg, you can find links off of those sites. Good luck. Brian On Wed, Sep 3, 2008 at 2:57 PM, Pete Carswell wrote: > I was interested in any references about use of python applications as a > web portal interface. A quick google search turned up > > > http://www.pythonthreads.com/news/latest/python-web-application-using-google-app-engine.html > > but I would be interested in other references. Web portals are, admittedly, > not my best subject. Thanks. > > pete > > > _______________________________________________ > CentralOH mailing list > CentralOH at python.org > http://mail.python.org/mailman/listinfo/centraloh > From nludban at osc.edu Wed Sep 3 22:56:03 2008 From: nludban at osc.edu (Neil Ludban) Date: Wed, 3 Sep 2008 16:56:03 -0400 Subject: [CentralOH] Pickling data into a database In-Reply-To: <1220466757.14287.13.camel@P1900> References: <1220466757.14287.13.camel@P1900> Message-ID: <20080903165603.f14088ec.nludban@osc.edu> On Wed, 03 Sep 2008 14:32:36 -0400 Mark Erbaugh wrote: ... > One of the outputs of this application is a performance curve that needs > to be saved. In the current application, I create a one to many child > table that has the X and Y coordinates of the curve (one pair per row). > I could easily Pickle a sequence of tuples. There are usually only a > dozen or fewer points on the curve (I think the worst case is about 40). > > Is this a good approach? What are the potential pitfalls? The serialized data is also difficult to look at for debugging, repr() and eval() are simpler if you only need to support Python. Look at JSON and YAML for support in other languages. > One pitfall that just came to mind is that this limits the ability of > non-Python applications to process the data. Maybe an alternative would > be to convert the data to XML instead of Pickling it? Overkill. Simple text file formats like .ini or .csv can do a lot, and the Python standard library supports them both. From brian.costlow at gmail.com Thu Sep 4 16:18:50 2008 From: brian.costlow at gmail.com (Brian Costlow) Date: Thu, 4 Sep 2008 10:18:50 -0400 Subject: [CentralOH] using python in web applications... In-Reply-To: <48BEBCE3.A14A.0000.0@osc.edu> References: <1217991094.23565.6.camel@P1900> <293817.77933.qm@web80014.mail.sp1.yahoo.com> <48BEA5C0.A14A.0000.0@osc.edu> <89d8b1b00809031251g53936c79ha81e035ed28376e5@mail.gmail.com> <48BEBCE3.A14A.0000.0@osc.edu> Message-ID: <89d8b1b00809040718p647b3f57i73c0c575b9603216@mail.gmail.com> Hi Pete, Okay, I did a bit of digging because I'm not familiar with Paraview and VTK. What you seem to want, is a web application (i.e. the interface is, or runs in, a browser) that has more of the interactivity available in the GUI of the Paraview desktop application. I don't know if the external API to Paraview or VTK is extensive enough to do this, but assuming that it is, I would envision it works something like this. Current app: User submits parameters in a web form, that connects with Paraview, which renders an image(s). That gets dispayed to users in browser. (Is this real time, or does the user submit the request, you do some batch process to generate the render, and then the students get the images later off the web?) Desired version: User submits parameters in a web form, that connects with Paraview, which renders images and or animations and displays back to user in near real time. That gets displayed with controls to pan/zoom etc. If user requests a view or modification that can't be shown using the current render, browser requests additional render data from Paraview/VTK. I don't think there's a silver bullet here for you. Certainly parts of TurboGears would help in developing the part of the web application that takes data from the user and calls the Paraview/VTK api, and also help in simplifying generation of the web pages themselves. But I think the bigger issue is what rendered image/movie/animation formats can Paraview return that the browser can also display, and how will you develop an interactive GUI in the browser. I suspect the gui programming in JavaScript, ActionScript, or whatever you decide to do it in may end up being more complex than the python part. On Wed, Sep 3, 2008 at 4:35 PM, Pete Carswell wrote: > Thanks, Brian. > > I thought it might be helpful to include a segment of email I had sent to > the Paraview forum. Initially I was inquiring about embedding Paraview in a > web portal. What we want is greater control with the sci viz processes in > generating viewable images from the CFD parameters. Python is also an API > for both VTK and Paraview. Here is that email: > > "I would like to clarify how we would like to use paraview in this fashion, > in the hope that it might generate some response. Currently, there is a CFD > course that is taught online. Parameters to a particular problem are entered > in a web page and submitted for a run on our HPC cluster. Images of the > solution are generated and supplied to the student through the web > interface. The batch script runs a processed script generated from Paraview > 2.6. We currently also have Paraview 3 installed on the cluster. > > We would like to have more flexibility in processing the images after the > solution is submitted, i.e. switching the modes of the data, vector or > scalar data, and panning and zooming on the images, and, subsequently, > handled by the students in the web application. > > I hope this describes our intentions better. Would appreciate any > feedback." > > I hope this might give someone some ideas. Thanks. > > pete -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark at microenh.com Tue Sep 9 02:27:32 2008 From: mark at microenh.com (Mark Erbaugh) Date: Mon, 08 Sep 2008 20:27:32 -0400 Subject: [CentralOH] WebPy and Cookies Message-ID: <1220920052.18920.13.camel@P1900> This is probably due to my lack of understanding. I'm using WebPy to create a Python based web application. According to "Javascript: The Definitive Guide" by Flannagan: "[cookies] are still uploaded to the web server in the request for any web page with which they are associated." In the client application Javascript, a cookie is created. The main use of this cookie is to coordinate information between related HTML pages. This cookie is associated with the root web page of the application and is available to all the HTML pages. It seems to be properly working between the HTML pages. My understanding of the above statement is that the cookie is also sent to the server, which I could actually use. In the application, the user fills out several pages of information. I'm using the cookie so that the user can navigate among these pages with the back and forward arrows in the browser. When they leave the last page, this information needs to be sent to the server for processing. WebPy has a function to retrieve the cookies passed to the CGI app. Basically, that just extracts the HTTP_COOKIE environment variable. In my app, that is always coming back empty. Another thing I though I could use the cookie for is to make sure that the user has arrived at this page after going through the initial logon page. I know it's not rock-solid security, but when the user visits the logon page, they enter their name and password. That is then validated by the server which re-directs them to the application main menu. That main menu page creates the cookie including their user id. I was thinking that if the server ever received a request for a page inside the application and there was no cookie (or the cookie didn't have a user id), the server could redirect them back to the logon screen. That would prevent someone bypassing the logon by entering the page address directly. From gacsinger at gmail.com Tue Sep 9 02:40:31 2008 From: gacsinger at gmail.com (Greg Singer) Date: Mon, 8 Sep 2008 20:40:31 -0400 Subject: [CentralOH] WebPy and Cookies In-Reply-To: <1220920052.18920.13.camel@P1900> References: <1220920052.18920.13.camel@P1900> Message-ID: On Mon, Sep 8, 2008 at 8:27 PM, Mark Erbaugh wrote: > This is probably due to my lack of understanding. I'm using WebPy to > create a Python based web application. Perhaps it's overkill, but I recommend that you use flup to handle your cookies. Just insert the following into your code.py: from flup.middleware.session import DiskSessionStore, SessionMiddleware, MemorySessionStore #Choose memorysession or disksession as appropriate def session_mw(app): sessionStore=DiskSessionStore(storeDir="/tmp/sessions/", timeout=50) return SessionMiddleware(sessionStore, app) class index: def GET(self): session = web.ctx.environ['com.saddi.service.session'].session #other code deleted if __name__=='__main__': render=web.template.render('templates/',cache=False) web.run(urls,globals(),session_mw) Note the code in the index class. This is how you retrieve the session, which is accessed just like a dictionary. - Greg From brian.costlow at gmail.com Tue Sep 9 14:55:42 2008 From: brian.costlow at gmail.com (Brian Costlow) Date: Tue, 9 Sep 2008 08:55:42 -0400 Subject: [CentralOH] WebPy and Cookies In-Reply-To: <1220920052.18920.13.camel@P1900> References: <1220920052.18920.13.camel@P1900> Message-ID: <89d8b1b00809090555s58a769dcp3d862e59cebac949@mail.gmail.com> On Mon, Sep 8, 2008 at 8:27 PM, Mark Erbaugh wrote: > This is probably due to my lack of understanding. I'm using WebPy to > create a Python based web application. > > According to "Javascript: The Definitive Guide" by Flannagan: > > "[cookies] are still uploaded to the web server in the request for any > web page with which they are associated." > > WebPy has a function to retrieve the cookies passed to the CGI app. > Basically, that just extracts the HTTP_COOKIE environment variable. In > my app, that is always coming back empty. Whenever I see this kind of issue, it almost always ends up being some kind of mismatch between the site address in the cookie, and what the server thinks this address should be. First make sure you are not getting some kind of subtle WebPy bug (it happens with the best of frameworks) by setting a cookie on the server side and reading it back. Next set a cookie with a different name, but otherwise identical settings from the server, and on that page, in JavaScript, and examine them both from the client side (i.e. FireBug or similar) to see if they are otherwise identical. > > > Another thing I though I could use the cookie for is to make sure that > the user has arrived at this page after going through the initial logon > page. I know it's not rock-solid security, but when the user visits the > logon page, they enter their name and password. That is then validated > by the server which re-directs them to the application main menu. That > main menu page creates the cookie including their user id. I was > thinking that if the server ever received a request for a page inside > the application and there was no cookie (or the cookie didn't have a > user id), the server could redirect them back to the logon screen. That > would prevent someone bypassing the logon by entering the page address > directly. > I don't know the security requirements of your site, but the first thing I always tell people is unless the entire session is ssl encrypted, you are subject to a possible man in the middle attack. The larger issue with just using the user id in plaintext in a cookie, is once a successful man in the middle occurs (or someone snoops cookies on the users computer) they can fake the cookie and compromise your site permanently. You really should hash the cookie using a salt that changes. One way is to compute a hash on every page request, keep a copy of the hash in local session store, and return to the browser in a cookie. On the next request, you can make sure they match, then change the hash. However because you must keep state on the server, this will present issues when it comes to horizontal scaling. Another way is to add a fixed salt and the id to a changing value such as usertime and hash them (I've seen some folks advocating hashing that again -- if there are any crypto guys weigh in on whether this helps or introduces patterns). Put the time and the salt in the cookie. Then you can reauthenticate based on the hash matching and change the hash by updating the timestamp. There are other variations on this, such as always rounding usertime down to the nearest hour so you don't need it in plaintext in the cookie, that have their own pros and cons. All of these methods are still susceptible to man in the middle, but at least a compromise isn't permanent. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.costlow at gmail.com Tue Sep 9 16:41:58 2008 From: brian.costlow at gmail.com (Brian Costlow) Date: Tue, 9 Sep 2008 10:41:58 -0400 Subject: [CentralOH] WebPy and Cookies In-Reply-To: <89d8b1b00809090555s58a769dcp3d862e59cebac949@mail.gmail.com> References: <1220920052.18920.13.camel@P1900> <89d8b1b00809090555s58a769dcp3d862e59cebac949@mail.gmail.com> Message-ID: <89d8b1b00809090741i1abcb439o859788f5e2c15155@mail.gmail.com> Correction, this sentence in my last post to this thread: "Put the time and the salt in the cookie." Should have said: *Put the time and the hash in the cookie. * -------------- next part -------------- An HTML attachment was scrubbed... URL: