git lfs for the answer-store directory
Hi folks, We're seeing a bit of churn in the answer-store directory. I'd like to use git-lfs to manage this so that we can keep going without bloating the repo *too* much. We're getting a bit of churn right now and every time it adds a couple megs to the repo -- the latest one I'm looking at adds 15 megs, and I'm reluctant to do that. https://git-lfs.github.com/ https://github.com/git-lfs/git-lfs/wiki/Tutorial This would keep our repo size down but also not disrupt our "answer-store" directory contents. Any objections to trying this out? -Matt
I say go for it! On Mon, Apr 13, 2020 at 10:01 PM Matthew Turk <matthewturk@gmail.com> wrote:
Hi folks,
We're seeing a bit of churn in the answer-store directory. I'd like to use git-lfs to manage this so that we can keep going without bloating the repo *too* much. We're getting a bit of churn right now and every time it adds a couple megs to the repo -- the latest one I'm looking at adds 15 megs, and I'm reluctant to do that.
https://git-lfs.github.com/ https://github.com/git-lfs/git-lfs/wiki/Tutorial
This would keep our repo size down but also not disrupt our "answer-store" directory contents.
Any objections to trying this out?
-Matt _______________________________________________ yt-dev mailing list -- yt-dev@python.org To unsubscribe send an email to yt-dev-leave@python.org
I've issued a PR that turns this on, but does *not* yet migrate any existing items to LFS. https://github.com/yt-project/yt/pull/2540 So what would happen is, I think, as we continue to merge and update the answers, they would move there over time. On Tue, Apr 14, 2020 at 3:51 AM Britton Smith <brittonsmith@gmail.com> wrote:
I say go for it!
On Mon, Apr 13, 2020 at 10:01 PM Matthew Turk <matthewturk@gmail.com> wrote:
Hi folks,
We're seeing a bit of churn in the answer-store directory. I'd like to use git-lfs to manage this so that we can keep going without bloating the repo *too* much. We're getting a bit of churn right now and every time it adds a couple megs to the repo -- the latest one I'm looking at adds 15 megs, and I'm reluctant to do that.
https://git-lfs.github.com/ https://github.com/git-lfs/git-lfs/wiki/Tutorial
This would keep our repo size down but also not disrupt our "answer-store" directory contents.
Any objections to trying this out?
-Matt _______________________________________________ yt-dev mailing list -- yt-dev@python.org To unsubscribe send an email to yt-dev-leave@python.org
_______________________________________________ yt-dev mailing list -- yt-dev@python.org To unsubscribe send an email to yt-dev-leave@python.org
Hey! sounds like a good idea! I think I saw a couple of other projects doing that. LSST (or whatever their current name is :-) ) being one of them. Cheers, Kacper On 4/13/20 4:00 PM, Matthew Turk wrote:
Hi folks,
We're seeing a bit of churn in the answer-store directory. I'd like to use git-lfs to manage this so that we can keep going without bloating the repo *too* much. We're getting a bit of churn right now and every time it adds a couple megs to the repo -- the latest one I'm looking at adds 15 megs, and I'm reluctant to do that.
https://git-lfs.github.com/ https://github.com/git-lfs/git-lfs/wiki/Tutorial
This would keep our repo size down but also not disrupt our "answer-store" directory contents.
Any objections to trying this out?
-Matt _______________________________________________ yt-dev mailing list -- yt-dev@python.org To unsubscribe send an email to yt-dev-leave@python.org
Hi all, I've been digging in, and it looks like it's possible I completely misread it, and that this would -- even with our small repository -- immediately blow up our bandwidth quota. It seems that the "large files" bandwidth is monitored separately, and that we'd be counted against it on every clone on an LFS-enabled system. So I'm still digging in to understand and see if there's an alternate, but right now I'm having some serious second thoughts. -Matt On Tue, Apr 14, 2020 at 8:02 AM Kacper Kowalik <xarthisius.kk@gmail.com> wrote:
Hey! sounds like a good idea! I think I saw a couple of other projects doing that. LSST (or whatever their current name is :-) ) being one of them. Cheers, Kacper
On 4/13/20 4:00 PM, Matthew Turk wrote:
Hi folks,
We're seeing a bit of churn in the answer-store directory. I'd like to use git-lfs to manage this so that we can keep going without bloating the repo *too* much. We're getting a bit of churn right now and every time it adds a couple megs to the repo -- the latest one I'm looking at adds 15 megs, and I'm reluctant to do that.
https://git-lfs.github.com/ https://github.com/git-lfs/git-lfs/wiki/Tutorial
This would keep our repo size down but also not disrupt our "answer-store" directory contents.
Any objections to trying this out?
-Matt _______________________________________________ yt-dev mailing list -- yt-dev@python.org To unsubscribe send an email to yt-dev-leave@python.org
_______________________________________________ yt-dev mailing list -- yt-dev@python.org To unsubscribe send an email to yt-dev-leave@python.org
Hi all, So this is my fault originally, I should have probably thought through the repo size issue more carefully when we originally did this work with Abhishek a few years ago. Would having a separate repository that exists only to store the answer files be sufficient? The files are small enough and changes infrequent enough you likely won’t need to use lfs at all and offloading the issue away from the main yt repo avoids most of the pain people experience in practice (although unfortunately won’t help much with size of the existing repo). Nathan On Tue, Apr 14, 2020 at 1:26 PM Matthew Turk <matthewturk@gmail.com> wrote:
Hi all,
I've been digging in, and it looks like it's possible I completely misread it, and that this would -- even with our small repository -- immediately blow up our bandwidth quota. It seems that the "large files" bandwidth is monitored separately, and that we'd be counted against it on every clone on an LFS-enabled system. So I'm still digging in to understand and see if there's an alternate, but right now I'm having some serious second thoughts.
-Matt
On Tue, Apr 14, 2020 at 8:02 AM Kacper Kowalik <xarthisius.kk@gmail.com> wrote:
Hey! sounds like a good idea! I think I saw a couple of other projects doing that. LSST (or whatever their current name is :-) ) being one of them. Cheers, Kacper
On 4/13/20 4:00 PM, Matthew Turk wrote:
Hi folks,
We're seeing a bit of churn in the answer-store directory. I'd like to use git-lfs to manage this so that we can keep going without bloating the repo *too* much. We're getting a bit of churn right now and every time it adds a couple megs to the repo -- the latest one I'm looking at adds 15 megs, and I'm reluctant to do that.
https://git-lfs.github.com/ https://github.com/git-lfs/git-lfs/wiki/Tutorial
This would keep our repo size down but also not disrupt our "answer-store" directory contents.
Any objections to trying this out?
-Matt _______________________________________________ yt-dev mailing list -- yt-dev@python.org To unsubscribe send an email to yt-dev-leave@python.org
_______________________________________________ yt-dev mailing list -- yt-dev@python.org To unsubscribe send an email to yt-dev-leave@python.org
_______________________________________________ yt-dev mailing list -- yt-dev@python.org To unsubscribe send an email to yt-dev-leave@python.org
Hi Nathan, I think you're right, a separate repo (maybe even subrepo) would be nice. I'll explore that and see if it is as nice an experience as using the answer-store directory, which is pretty nice. On Tue, Apr 14, 2020 at 2:58 PM Nathan <nathan.goldbaum@gmail.com> wrote:
Hi all,
So this is my fault originally, I should have probably thought through the repo size issue more carefully when we originally did this work with Abhishek a few years ago.
Would having a separate repository that exists only to store the answer files be sufficient? The files are small enough and changes infrequent enough you likely won’t need to use lfs at all and offloading the issue away from the main yt repo avoids most of the pain people experience in practice (although unfortunately won’t help much with size of the existing repo).
Nathan
On Tue, Apr 14, 2020 at 1:26 PM Matthew Turk <matthewturk@gmail.com> wrote:
Hi all,
I've been digging in, and it looks like it's possible I completely misread it, and that this would -- even with our small repository -- immediately blow up our bandwidth quota. It seems that the "large files" bandwidth is monitored separately, and that we'd be counted against it on every clone on an LFS-enabled system. So I'm still digging in to understand and see if there's an alternate, but right now I'm having some serious second thoughts.
-Matt
On Tue, Apr 14, 2020 at 8:02 AM Kacper Kowalik <xarthisius.kk@gmail.com> wrote:
Hey! sounds like a good idea! I think I saw a couple of other projects doing that. LSST (or whatever their current name is :-) ) being one of them. Cheers, Kacper
On 4/13/20 4:00 PM, Matthew Turk wrote:
Hi folks,
We're seeing a bit of churn in the answer-store directory. I'd like to use git-lfs to manage this so that we can keep going without bloating the repo *too* much. We're getting a bit of churn right now and every time it adds a couple megs to the repo -- the latest one I'm looking at adds 15 megs, and I'm reluctant to do that.
https://git-lfs.github.com/ https://github.com/git-lfs/git-lfs/wiki/Tutorial
This would keep our repo size down but also not disrupt our "answer-store" directory contents.
Any objections to trying this out?
-Matt _______________________________________________ yt-dev mailing list -- yt-dev@python.org To unsubscribe send an email to yt-dev-leave@python.org
_______________________________________________ yt-dev mailing list -- yt-dev@python.org To unsubscribe send an email to yt-dev-leave@python.org
_______________________________________________ yt-dev mailing list -- yt-dev@python.org To unsubscribe send an email to yt-dev-leave@python.org
_______________________________________________ yt-dev mailing list -- yt-dev@python.org To unsubscribe send an email to yt-dev-leave@python.org
participants (4)
-
Britton Smith -
Kacper Kowalik -
Matthew Turk -
Nathan