using sqlite for storing submissions
Hi all, [idea] Right now we are using Hg for storing submissions inside static root..Is not totally good on a cloud platform with limited resources..its not fast too (takes lot of server load) I rather would like to use SQLite separately for storing submissions (I agree that using Postgre SQL for it is expensive on cloud).. What do you guys say? we dont have to provide all revisions as a Hg repo when downloading.. people can employ some tools to compare if they want... looking for your comments cheers surya
On 15.07.2013 17:39, Surya Kasturi wrote:
Hi all,
[idea]
Right now we are using Hg for storing submissions inside static root..Is not totally good on a cloud platform with limited resources..its not fast too (takes lot of server load)
I rather would like to use SQLite separately for storing submissions (I agree that using Postgre SQL for it is expensive on cloud)..
What do you guys say?
we dont have to provide all revisions as a Hg repo when downloading.. people can employ some tools to compare if they want...
I disagree. Using SQLite in a production server environment causes nothing but pain (at least to my experience). I would assume using some DVCS for storing revisions should be the most natural way to do so, and I would hope that there's some webapp libraries availble to show diff's etc, though I don't have any direct experience with this. Oh, and I personally don't see the need to have spyce running on a "cloud platform" (depending on what you mean by that, of course). If you have one single server, then you don't need all the distributed-whatsoever and can concentrate on having everything on one machine. Just my 2ct. -- Andreas.
On Mon, Jul 15, 2013 at 5:46 PM, Andreas Hilboll <lists@hilboll.de> wrote:
On 15.07.2013 17:39, Surya Kasturi wrote:
Hi all,
[idea]
Right now we are using Hg for storing submissions inside static root..Is not totally good on a cloud platform with limited resources..its not fast too (takes lot of server load)
I rather would like to use SQLite separately for storing submissions (I agree that using Postgre SQL for it is expensive on cloud)..
What do you guys say?
we dont have to provide all revisions as a Hg repo when downloading.. people can employ some tools to compare if they want...
I disagree. Using SQLite in a production server environment causes nothing but pain (at least to my experience). I would assume using some DVCS for storing revisions should be the most natural way to do so, and I would hope that there's some webapp libraries availble to show diff's etc, though I don't have any direct experience with this.
Showing diff is one idea I had too but we can figure out something within database too (I guess... dont know for sure). If we are not a busy site.. with very huge traffic and only using SQLite partly, it should work nicely http://stackoverflow.com/questions/913067/sqlite-as-a-production-database-fo...
Oh, and I personally don't see the need to have spyce running on a "cloud platform" (depending on what you mean by that, of course). If you have one single server, then you don't need all the distributed-whatsoever and can concentrate on having everything on one machine.
I didnt get you
Just my 2ct.
-- Andreas.
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-dev
On Mon, Jul 15, 2013 at 6:06 PM, Surya Kasturi <suryak@ieee.org> wrote:
On Mon, Jul 15, 2013 at 5:46 PM, Andreas Hilboll <lists@hilboll.de> wrote:
On 15.07.2013 17:39, Surya Kasturi wrote:
Hi all,
[idea]
Right now we are using Hg for storing submissions inside static root..Is not totally good on a cloud platform with limited resources..its not fast too (takes lot of server load)
I rather would like to use SQLite separately for storing submissions (I agree that using Postgre SQL for it is expensive on cloud)..
What do you guys say?
we dont have to provide all revisions as a Hg repo when downloading.. people can employ some tools to compare if they want...
I disagree. Using SQLite in a production server environment causes nothing but pain (at least to my experience). I would assume using some DVCS for storing revisions should be the most natural way to do so, and I would hope that there's some webapp libraries availble to show diff's etc, though I don't have any direct experience with this.
Showing diff is one idea I had too but we can figure out something within database too (I guess... dont know for sure). If we are not a busy site.. with very huge traffic and only using SQLite partly, it should work nicely
http://stackoverflow.com/questions/913067/sqlite-as-a-production-database-fo...
Oh, and I personally don't see the need to have spyce running on a "cloud platform" (depending on what you mean by that, of course). If you have one single server, then you don't need all the distributed-whatsoever and can concentrate on having everything on one machine.
I didnt get you
Just my 2ct.
-- Andreas.
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-dev
Truly, I barely have experience with databases, queries etc.. but I guess, its not bad experimenting things.. couchDB does some fantastic stuff.. May be there are other technologies we can use.. (searching!!) I personally don't feel right using repos in static data.. just feel that its very complex that way..
On 15.07.2013 18:06, Surya Kasturi wrote:
On Mon, Jul 15, 2013 at 5:46 PM, Andreas Hilboll <lists@hilboll.de <mailto:lists@hilboll.de>> wrote:
On 15.07.2013 17:39, Surya Kasturi wrote: > Hi all, > > [idea] > > Right now we are using Hg for storing submissions inside static root..Is > not totally good on a cloud platform with limited resources..its not > fast too (takes lot of server load) > > I rather would like to use SQLite separately for storing submissions (I > agree that using Postgre SQL for it is expensive on cloud).. > > What do you guys say? > > > we dont have to provide all revisions as a Hg repo when downloading.. > people can employ some tools to compare if they want...
I disagree. Using SQLite in a production server environment causes nothing but pain (at least to my experience). I would assume using some DVCS for storing revisions should be the most natural way to do so, and I would hope that there's some webapp libraries availble to show diff's etc, though I don't have any direct experience with this.
Showing diff is one idea I had too but we can figure out something within database too (I guess... dont know for sure). If we are not a busy site.. with very huge traffic and only using SQLite partly, it should work nicely
http://stackoverflow.com/questions/913067/sqlite-as-a-production-database-fo...
From my experience, it doesn't, even on low-traffic sites. SQLite doesn't handle concurrency, so every so often you get some "database locked" errors. SQLite is good for development stage, but for production sites, don't use it.
BTW, I don't see a problem using PostgreSQL as a database server. Especially for low traffic sites, the performance should be fine.
Oh, and I personally don't see the need to have spyce running on a "cloud platform" (depending on what you mean by that, of course). If you have one single server, then you don't need all the distributed-whatsoever and can concentrate on having everything on one machine.
I didnt get you
Never mind. Just a pun about this "cloud hype" terminology. Cheers, Andreas.
Just my 2ct.
-- Andreas.
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@scipy.org <mailto:SciPy-Dev@scipy.org> http://mail.scipy.org/mailman/listinfo/scipy-dev
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-dev
On Mon, Jul 15, 2013 at 6:15 PM, Andreas Hilboll <lists@hilboll.de> wrote:
On 15.07.2013 18:06, Surya Kasturi wrote:
On Mon, Jul 15, 2013 at 5:46 PM, Andreas Hilboll <lists@hilboll.de <mailto:lists@hilboll.de>> wrote:
On 15.07.2013 17:39, Surya Kasturi wrote: > Hi all, > > [idea] > > Right now we are using Hg for storing submissions inside static root..Is > not totally good on a cloud platform with limited resources..its
not
> fast too (takes lot of server load) > > I rather would like to use SQLite separately for storing submissions (I > agree that using Postgre SQL for it is expensive on cloud).. > > What do you guys say? > > > we dont have to provide all revisions as a Hg repo when
downloading..
> people can employ some tools to compare if they want...
I disagree. Using SQLite in a production server environment causes nothing but pain (at least to my experience). I would assume using
some
DVCS for storing revisions should be the most natural way to do so,
and
I would hope that there's some webapp libraries availble to show
diff's
etc, though I don't have any direct experience with this.
Showing diff is one idea I had too but we can figure out something within database too (I guess... dont know for sure). If we are not a busy site.. with very huge traffic and only using SQLite partly, it should work nicely
http://stackoverflow.com/questions/913067/sqlite-as-a-production-database-fo...
From my experience, it doesn't, even on low-traffic sites. SQLite doesn't handle concurrency, so every so often you get some "database locked" errors. SQLite is good for development stage, but for production sites, don't use it.
BTW, I don't see a problem using PostgreSQL as a database server. Especially for low traffic sites, the performance should be fine.
Oh, and I personally don't see the need to have spyce running on a "cloud platform" (depending on what you mean by that, of course). If
you
have one single server, then you don't need all the distributed-whatsoever and can concentrate on having everything on
one
machine.
I didnt get you
Never mind. Just a pun about this "cloud hype" terminology.
Cheers, Andreas.
Just my 2ct.
-- Andreas.
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@scipy.org <mailto:SciPy-Dev@scipy.org> http://mail.scipy.org/mailman/listinfo/scipy-dev
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@scipy.org http://mail.scipy.org/mailman/listinfo/scipy-dev
I am not worried about current PostgreSQL.. I am thinking to do something with static submissions data (as Hg repos) on filesystem..
On 15.07.2013 18:16, Surya Kasturi wrote:
On Mon, Jul 15, 2013 at 6:15 PM, Andreas Hilboll <lists@hilboll.de <mailto:lists@hilboll.de>> wrote:
On 15.07.2013 18:06, Surya Kasturi wrote: > > > > On Mon, Jul 15, 2013 at 5:46 PM, Andreas Hilboll <lists@hilboll.de <mailto:lists@hilboll.de> > <mailto:lists@hilboll.de <mailto:lists@hilboll.de>>> wrote: > > On 15.07.2013 17:39, Surya Kasturi wrote: > > Hi all, > > > > [idea] > > > > Right now we are using Hg for storing submissions inside static > root..Is > > not totally good on a cloud platform with limited resources..its not > > fast too (takes lot of server load) > > > > I rather would like to use SQLite separately for storing > submissions (I > > agree that using Postgre SQL for it is expensive on cloud).. > > > > What do you guys say? > > > > > > we dont have to provide all revisions as a Hg repo when downloading.. > > people can employ some tools to compare if they want... > > I disagree. Using SQLite in a production server environment causes > nothing but pain (at least to my experience). I would assume using some > DVCS for storing revisions should be the most natural way to do so, and > I would hope that there's some webapp libraries availble to show diff's > etc, though I don't have any direct experience with this. > > > Showing diff is one idea I had too but we can figure out something > within database too (I guess... dont know for sure). > If we are not a busy site.. with very huge traffic and only using SQLite > partly, it should work nicely > > http://stackoverflow.com/questions/913067/sqlite-as-a-production-database-fo...
From my experience, it doesn't, even on low-traffic sites. SQLite doesn't handle concurrency, so every so often you get some "database locked" errors. SQLite is good for development stage, but for production sites, don't use it.
BTW, I don't see a problem using PostgreSQL as a database server. Especially for low traffic sites, the performance should be fine.
> > > Oh, and I personally don't see the need to have spyce running on a > "cloud platform" (depending on what you mean by that, of course). If you > have one single server, then you don't need all the > distributed-whatsoever and can concentrate on having everything on one > machine. > > I didnt get you
Never mind. Just a pun about this "cloud hype" terminology.
Cheers, Andreas.
> > > Just my 2ct. > > -- Andreas. > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev@scipy.org <mailto:SciPy-Dev@scipy.org> <mailto:SciPy-Dev@scipy.org <mailto:SciPy-Dev@scipy.org>> > http://mail.scipy.org/mailman/listinfo/scipy-dev > > > > > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev@scipy.org <mailto:SciPy-Dev@scipy.org> > http://mail.scipy.org/mailman/listinfo/scipy-dev >
I am not worried about current PostgreSQL.. I am thinking to do something with static submissions data (as Hg repos) on filesystem..
Well, in your original mail, you said
I rather would like to use SQLite separately for storing submissions (I agree that using Postgre SQL for it is expensive on cloud)..
But also for file-system interactions with hg or git I wouldn't be worried with a low-traffic site. After all, gitorious, bitbucket, github, and the like are all based on DVCS which store their data in the filesystem. I doubt that github have their repositories in some SQL database (but might be mistaken here) ... -- Andreas.
15.07.2013 19:21, Andreas Hilboll kirjoitti: [clip]
Well, in your original mail, you said
I rather would like to use SQLite separately for storing submissions (I agree that using Postgre SQL for it is expensive on cloud)..
Well, here "Sqlite" should be interpreted as "SQL database". It'll go through an ORM, so the backend can be Postgres or whatever.
But also for file-system interactions with hg or git I wouldn't be worried with a low-traffic site. After all, gitorious, bitbucket, github, and the like are all based on DVCS which store their data in the filesystem. I doubt that github have their repositories in some SQL database (but might be mistaken here) ...
A problem may be the spawning of hg processes. It needs to load a whole Python interpreter to memory. However, if hg is only invoked on upload time, it probably doesn't matter. I don't remember how it works currently. I'd expect that direct file access is not a bottleneck. So before trying to change this, I'd recommend measuring that this is actually a problem. -- Pauli Virtanen
participants (3)
-
Andreas Hilboll -
Pauli Virtanen -
Surya Kasturi