Quick setup of VMs (virtual machines) for issue testing
Can someone recommend a quick and secure (= reliable image sources) way of downloading and running ready-to-use VM images to test issues reported on different Linux distros and version? I am currently using Gnome Boxes on Ubuntu 20.04 which must be installed via snap (the apt version is to old and has many broken download links to no longer existing VM Images)...
An important point! I'm currently using VirtualBox as well. It's not exactly lightweight, and comes with its own kernel modules for some functionality. But it has nice comfort features (like host-guest shared clipboard) and good snapshot management (saving several different states of the machine, both when running or powered down). Cheers, Michael On 09.09.2022 10:34, Hakan Bayındır wrote:
Good topic. I also thought about that but I don't have much experience with that. But I know that Hakan has much experience with it. Currently I have some VMs in VirtualBox. I install them manually not using an image. I snapshot them in a fresh state and roll them back to that snapshot state. I would like to have Back In Time on a lot of different VMs just making a snapshot from A to B. And then a testing script does a simple "diff A B" and give a green or red light. But that "light" for each machine need to be collected on a central monitor or log file. I don't want to look into each machine myself. I am not sure if and how this could be solved. But I also heard Vagrant in the context of such cases. But currently I have enough to do to get the usual unittest running and understanding the existing testing code. :D
Just as a heads-up, I’ll be building a couple of Vagrant machines for BiT testing, and will try to write some documentation about it. We should keep this documentation visible and usable by everyone, so how should I provide it? A markdown file in the repo? A wiki change? First an RFC for the mailing list? I’ll be making Vagrant files available in my Vagrant Machines repository[0] first, and will announce here. I’m planning to start with Fedora 36 and Ubuntu 22.04. Any other requests/preferences? Please let me know any specific requests or anything, Will let you know as I progress further, Cheers, Hakan
On 2022-09-11 14:57 Hakan Bayındır <hakan@bayindir.org> wrote:
Just as a heads-up, I’ll be building a couple of Vagrant machines for BiT testing, and will try to write some documentation about it.
Awesome.
First an RFC on the list. But finally a simple markdown file.
I need to reflect the different rsync versions; One with Version 3.2.4 (or higher) and one with 3.2.3 or lower. In Debian this would be Debian 10 (current "oldstable") and a Debian 12 (not stable released yet but known as "testing") would be great. But I think this could be done with two Ubuntu versions also.
Hi, Hakan!
That's great, thanks a lot for the effort! Let's discuss it on the mailing list first, then document in the repo when we have things (mostly ;)) working.
Taking a look at: https://github.com/bit-team/backintime/labels/Distro-Specific I would say that we can also make good use of Manjaro and CentOS for testing. In further Issues I haven't sorted out yet, there are some lurking problems with Wayland, so we'd best get a distro with that, too. Thanks! Michael
Hello all, I've created two Debian VMs for testing: - Debian Buster (Stable) with RSync 3.2.3 [0] - Debian Bookworm (Testing) RSync 3.2.5 [1] The VMs use official Debian images, update themselves to latest state, and install XFCE4 with Back in Time from repositories. The username and password of the systems are bit/bit. To automatically install virtualbox guest additions, please install vagrant-vbguest via "vagrant plugin install vagrant-vbguest". The systems already have semi-up to date guest utilities out of the box though. Also, Debian testing has an incompatibility with the plugin, so the Vagrantfile for Testing disables auto installation by default. To run these VMs, please put the relevant Vagrantfiles to their own folder and run "vagrant up". The rest should be automagic. I did not detail installation of Vagrant and Virtualbox, since I assume both should be in repositories of major distributions already. Next steps: 1. Write proper documentation. 2. Make systems login automatically (needs some script grease, had no time). 3. Add other distributions (CentOS, Fedora, Manjaro, Ubuntu). Please don't hesitate to contact for any questions, comments, suggestions, etc. Hope this helps, Cheers, H. [0]: https://git.sr.ht/~bayindirh/vagrant-machines/tree/master/item/debian11-back... [1]: https://git.sr.ht/~bayindirh/vagrant-machines/tree/master/item/debian12-back... On 11.09.2022 20:43, Michael Büker wrote:
Dear Hakan, awesome! ;) Am 12.09.2022 09:48 schrieb Hakan Bayındır:
run "vagrant up". The rest should be automagic.
Boah! Heaven! :D One question here: Does vagrant run the VMs all at the same time or one after the other? When having two or more VMs on my machine everything freeze. It doesn't matter for me if the testing will run multiple days on my slow home machine. ;)
Please don't hesitate to contact for any questions, comments, suggestions, etc.
I suspect that so far I have not fully grasped the purpose and use case for Vagrant. Therefore, I probably wish for some features that do not exist. I assume that Vagrant can configured that way that it start a test script automatically. But the key question for me here is where do the results go? Maybe there are 10 VMs with each 3 or 4 bigger tests. This maybe run 2 or 3 days on my home machine. Can I see a central log file or something like that at the end of that time period? I imagine something like a colored output of pytest or Travis. But as I said maybe I am to naive here. ;) Kind Christian
Dear Christian, and all, Vagrant is just a small layer on Virtualbox. The VMs are different files, so you can run either or both at the same time. Each of the machines have a single CPU with 2GB of RAM, so it wouldn't bog your system down. If you need more resources, you can tune via the Vagrantfile itself, but please tell me too, so I can update the files in the repository. Oh, I also forgot that you need to restart the machines once after everything is installed. "vagrant reload" should do this gracefully. Vagrant kindly reminds that after a successful provisioning though. We can create another VM which downloads the latest version from GitHub automatically and run the tests at first start or set itself up, so all tests can be run with a single command. These kinds of scenarios are what Vagrant is built for. Even, we can setup such a VM which can be our semi-official development environment, so contributors don't spend a day to be able to start contributing Back in Time. A single Vagrantfile can contain multiple VMs, too. After deciding the architecture, we can "summon" a virtual cluster to do the all the testing, however it would be heavy for some machines. Maybe we can utilize containers there (again, inside a VM), but we need to talk about it. I'll try to do my best to provide this infrastructure to you and the project, so we can move forward with less friction and greater momentum, while being sustainable with minimum effort. Please don't hesitate to contact for any questions, Cheers, Hakan On 12.09.2022 11:49, c.buhtz@posteo.jp wrote:
On Mon, 2022-09-12 at 08:49 +0000, c.buhtz@posteo.jp wrote:
I suspect that so far I have not fully grasped the purpose and use case for Vagrant
I will use it "just" for - issue testing (reproducing and fixing) that mentions a special Linux distribution and version - PR testing if it adds a feature that I do not have (or is too slow to boot/login) on my local dev computer (eg. different desktop environments like KDE and Gnome) - maybe manual testing of GUI and unit tests before releases
I personally would not go that far to setup a build matrix infrastructure like Travis because it is very tricky and time consuming to automate CI/CD. That is why we have TravisCI activated at Github
Good evening Am 12.09.2022 18:15 schrieb J. A.:
For me all this points are for a VM but for Vagrant because all that points are "manual" tasks.
My need is not about CI/CD (deployment, building). It is just automated testing in different environments. I just want to prevent myself from booting, test-script-starting, log-file-collecting for each of 10 or more VMs. Kind Christian
I just want to prevent myself from booting, test-script-starting, log-file-collecting for each of 10 or more VMs.
A have just tested the synced_folder feature of Vagrant to share files between host and guests. Just add this line to the vagrant config: # Share an additional folder to the guest VM. The first argument is # the path on the host to the actual folder. The second argument is # the path on the guest to mount the folder. And the optional third # argument is a set of non-required options. # # By default owner and group are set to the ssh user (vagrant?). Others are RO... # To grant another user access rights add the user to the vagrant group # (or change the mount options to allwe READ WRITE EXECUTE for everyone) config.vm.synced_folder "../shared_data", "/vagrant_shared_data", mount_options: ["dmode=777,fmode=777"] Then create a "shared" folder at the same folder level as your vagrant box folders (in order to not loose shared files if you delete or move a vagrant box folder). Then you can access the shared folder from within the VM via "/vagrant_shared_data".
On Sat, 2022-09-17 at 17:33 -0500, gsker wrote:
Would it be reasonable to ask you to repost the vagrant file URLs? I keep having to dig for them when testing your new updates.
The vagrant files were designed and provided by Hakan, not me, but he posted these links: https://git.sr.ht/~bayindirh/vagrant-machines/tree/master/item/debian11-back... https://git.sr.ht/~bayindirh/vagrant-machines/tree/master/item/debian12-back...
On Mon, 2022-09-12 at 10:48 +0300, Hakan Bayındır wrote:
Thanks a lot, saves much time. I am just running vagrant...
The VMs use official Debian images, update themselves to latest state, and install XFCE4 with Back in Time from repositories.
XFCE4 is gtk-based (like Gnome). Would it be possible to also include "plain vanilla" Gnome + KDE (since KDE used Qt and I need that to cross-test the GUI - eg. icons, system tray, notifications). X11 + wayland would also be good for Gnome - but let's I don't want to burn your time (I have a working machine for this this ATM). I would prefer to have at least 4 GB of RAM preconfigured due to the desktops and the need to start an IDE for debugging (but I also have luxurious 64 GB of RAM ;-)
3. Add other distributions (CentOS, Fedora, Manjaro, Ubuntu)
I have currently no overview over the OS distros and versions with the most issue reports but Fedora (by Red Hat) and possibly Arch BTW: German's famous ct computer magazin did publish a Linux Distro comparisson picture here which I like a quick overview over main distro differenences: ct.de/linuxplan
Hello, Thanks for the feedback. I will add different VMs with different DEs (adding many DEs to the same system is not a realistic scenario in real world). I have chosen XFCE4 & 2GB of RAM because I didn't want to bring people's machines down. Maybe for IDE, I can export a folder, so the IDE can live on the host machine while the source code is shared, what would you think about that? Of course I can (and will need to) increase the default RAM on some VMs, but, I just want to maximize flexibility. I may add Fedora relatively quickly since it's popular, easy to manage and is a good representation of RPM based world. I'll rather add other distributions as the consensus builds around them. I'll also start to write the documentation, but it might take a bit longer since I'm a bit busier than ordinary for the next month or so. However, if this effort accelerates the project in any way, I'll consider this as a win. Please don't hesitate to contact for any questions & feedback, Cheers, Hakan On 12.09.2022 16:08, J. A. wrote:
Moin! On 12.09.2022 15:08, J. A. wrote:
I'm trying to keep the https://github.com/bit-team/backintime/labels/Distro-Specific Label on GitHub up to date with distro-specific (and desktop-environment-specific) Issues, so you can take a look there to get some idea. I would also include X11/Wayland-specific Issues here, but I haven't gotten around to sorting them out yet.
How sweet :) Cheers! Michael
@Hakan On Mon, 2022-09-12 at 10:48 +0300, Hakan Bayındır wrote:
To run these VMs, please put the relevant Vagrantfiles to their own folder and run "vagrant up". The rest should be automagic.
Do you have hint for me? My installation on Ubuntu 20.04.5 LTS as host with VirtualBox 6.1 and vagrant 2.2.6 hangs here despite the VM is running and waiting at the login screen (see screen shot with VM and console). I have already tried to manually "cable connect" the network adapter 1 and 2, install vagrant-vbguest plugin and edit the Vagrant file like described here etc.: https://stackoverflow.com/questions/38463579/vagrant-hangs-at-ssh-auth-metho... BTW: How do I continue a broken installation, just via "vagrant up"? Thanks a lot :-) Jürgen config.vm.synced_folder '/host/path', '/guest/path', SharedFoldersEnableSymlinksCreate: false ==> default: Clearing any previously set network interfaces... ==> default: Preparing network interfaces based on configuration... default: Adapter 1: nat default: Adapter 2: hostonly ==> default: Forwarding ports... default: 22 (guest) => 2222 (host) (adapter 1) ==> default: Running 'pre-boot' VM customizations... ==> default: Booting VM... ==> default: Waiting for machine to boot. This may take a few minutes... default: SSH address: 127.0.0.1:2222 default: SSH username: vagrant default: SSH auth method: private key default: Warning: Authentication failure. Retrying... default: Warning: Authentication failure. Retrying... default: Warning: Authentication failure. Retrying... ... x times ... Timed out while waiting for the machine to boot. This means that Vagrant was unable to communicate with the guest machine within the configured ("config.vm.boot_timeout" value) time period. If you look above, you should be able to see the error(s) that Vagrant had when attempting to connect to the machine. These errors are usually good hints as to what may be wrong. If you're using a custom box, make sure that networking is properly working and you're able to connect to the machine. It is a common problem that networking isn't setup properly in these boxes. Verify that authentication configurations are also setup properly, as well. If the box appears to be booting properly, you may want to increase the timeout ("config.vm.boot_timeout") value. ~/vagrant >
Hmm, strange. I never experienced this. Let me research the issue and come back to you. This is completely unexpected
To narrow down the problem I have created a new simple box with "vagrant init" + only config.vm.box = "generic/ubuntu2204" And had exactly the same problems (with and without the *guest* plugin). So the problem must be with Ubuntu 20.04 (or Debian upstream) or the packaged vagrant version. I logged in as root and checked if the public ssh key was installed in ~/.ssh/ and it was. I also checked if the PPKs match (host == VM): cat .ssh/authorized_keys ssh-keygen -y -f ~/.vagrant.d/insecure_private_key Manually logging in via private key (without a password) ssh -i ~/.vagrant.d/insecure_private_key ssh://vagrant@127.0.0.1:2222 does work if I delete and re-accept the fingerprint of the "unknown" server... Good thing is: - I have learned the very basics of vagrant (thanks for your initiative!) - I have now two more different VMs that I can start at least manually without vagrant - I know now to to setup a new VM quickly by copying and modifying a Vagrantfile For the next VM I will download and install the latest vagrant version manually, so don't spend too much time to investigate this... BTW: I think the Vagrantfile for Debian 11 is not up-to-date (creation of the bit user is missing eg.) - forgot to commit/push? but copying the script from Debian 12 is straight-forward
SSH authentication issue SOLVED for my Ubuntu 20.04 host! default: SSH address: 127.0.0.1:2222 default: SSH username: vagrant default: SSH auth method: private key default: Warning: Authentication failure. Retrying... I have updated the Ubuntu-packaged vagrant version (2.2.6) via download from https://www.vagrantup.com/downloads to 2.3.0 and re-installed the vbguest plugin via vagrant plugin install vagrant-vbguest and the installation went straight through. Just configured RAM and 128 MB video memory (for higher resolutions)... On Mon, 2022-09-12 at 23:25 +0200, J. A. wrote:
Dear Hakan! Wonderful work! Thank you so much! I've used VirtualBox for many years, but this is the first time I've seen vagrant, and it's amazing :) I managed to set up the Debian 12 VM very quickly, log in and launch backintime. I think this will be a great help in maintaining backintime, especially for reproducing bugs and testing funtionality in different distros, desktop environments and X11/Wayland. Christian is mainly interested in running automated tests in all these environments -- and from what I've seen, that should be well possible with some SSH magic. Since my main machine has 32G of RAM, I'd also vote for 4G per VM :) I'm looking forward to having a way to use these VMs to checkout the git main branch (or some other branch) for testing :) We'd have to think about avoiding collisions between the repository-installed stock backintime and our git version, but you probably already have a good idea how to do that. So thank you very much so far, I'm very happy with the way this is going :) Cheers, Michael On 12.09.2022 09:48, Hakan Bayındır wrote:
Dear Christian and all, Thanks for your kind words. I'm glad that the VMs helped to accelerate the process. First of all, I have updated the VMs. The changelog is as follows: - [debian11] Fix: User creation script is added (I pushed the wrong file, sorry). - [debian11] Fix: Post-up message refers to correct Debian version. - [debian11/12] Feat: RAM is bumped to 4096MB. - [debian11/12] Feat: VRAM is bumped to 128MB. - [README.md] Doc: Reflect memory changes. I'm planning to create two more Debian VMs which automatically checks-out the latest code and installs any tooling necessary to develop and test Back in Time. These will be "-dev" machines. These "-dev" machines will need some scripting and setup but nothing impossible. Did something like that before. Just needs time. Then, we can add more distributions to "-lab" family to test with issues and other problems. So, my current roadmap is as follows: 1. Create "-dev" machines which gets the latest code from BiT repositories, and required tooling to work on the code. 2. Write proper documentation. 3. Make systems login automatically (needs some script grease, had no time). 4. Add other distributions (CentOS, Fedora, Manjaro, Ubuntu). Hope this plan helps, Please don't hesitate to contact for any questions and feedback, Cheers, Hakan On 12.09.2022 23:58, Michael Büker wrote:
Hi, Hakan! On 13.09.2022 09:50, Hakan Bayındır wrote:
Please don't hesitate to contact for any questions and feedback,
Maybe you can help me understand one thing: Vagrant seems to work in "environments", where each environment has its own folder that includes a "Vagrantfile", and has an automatically created ".vagrant" subfolder (similar to ".git", I guess). Each "environment" can apparently contain many "providers"(??). How do these "environments" and "providers" relate to your Vagrantfiles? Is each VM its own environment in its own folder? Or should they be combined somehow (now or in the future)? I've tried to Google some documentation on this, but I don't understand too much of what I find ;) Thanks :) Michael
Hello Michael, Yes, you’re right. Vagrant machines works in their own environments. Which is the folder where the Vagrantfile resides. Besides “.vagrant” folder, you can use the Vagrantfile to copy files from that folder during provision, or create folders which bridge the VM to the host for file sharing. While you may possibly reach outside this folder via Vagrantfile, storing everything in that folder allows zipping the folder, giving to someone else and saying, “just vagrant up”. Providers is something different. Vagrant is not limited to VirtualBox. Vagrant can work with VirtualBox, VMWare, libvirt, docker and many more. So your Vagrantfile can configure a Docker container instead of a VirtualBox VM, for example. The file will be different, however. Moreover, a single Vagrantfile can contain more than one virtual machine. I have worked with two Virtualbox VMs while working on something, but didn’t explore the concept further. However, for our case, I’ll rather create different machines, because bringing up a cluster of machines will not be practical and feasible for everyone. Hope this information sheds some more information on capabilities of Vagrant. Please don't hesitate to contact for any questions and feedback, As always, Cheers, Hakan
An important point! I'm currently using VirtualBox as well. It's not exactly lightweight, and comes with its own kernel modules for some functionality. But it has nice comfort features (like host-guest shared clipboard) and good snapshot management (saving several different states of the machine, both when running or powered down). Cheers, Michael On 09.09.2022 10:34, Hakan Bayındır wrote:
Good topic. I also thought about that but I don't have much experience with that. But I know that Hakan has much experience with it. Currently I have some VMs in VirtualBox. I install them manually not using an image. I snapshot them in a fresh state and roll them back to that snapshot state. I would like to have Back In Time on a lot of different VMs just making a snapshot from A to B. And then a testing script does a simple "diff A B" and give a green or red light. But that "light" for each machine need to be collected on a central monitor or log file. I don't want to look into each machine myself. I am not sure if and how this could be solved. But I also heard Vagrant in the context of such cases. But currently I have enough to do to get the usual unittest running and understanding the existing testing code. :D
Just as a heads-up, I’ll be building a couple of Vagrant machines for BiT testing, and will try to write some documentation about it. We should keep this documentation visible and usable by everyone, so how should I provide it? A markdown file in the repo? A wiki change? First an RFC for the mailing list? I’ll be making Vagrant files available in my Vagrant Machines repository[0] first, and will announce here. I’m planning to start with Fedora 36 and Ubuntu 22.04. Any other requests/preferences? Please let me know any specific requests or anything, Will let you know as I progress further, Cheers, Hakan
On 2022-09-11 14:57 Hakan Bayındır <hakan@bayindir.org> wrote:
Just as a heads-up, I’ll be building a couple of Vagrant machines for BiT testing, and will try to write some documentation about it.
Awesome.
First an RFC on the list. But finally a simple markdown file.
I need to reflect the different rsync versions; One with Version 3.2.4 (or higher) and one with 3.2.3 or lower. In Debian this would be Debian 10 (current "oldstable") and a Debian 12 (not stable released yet but known as "testing") would be great. But I think this could be done with two Ubuntu versions also.
Hi, Hakan!
That's great, thanks a lot for the effort! Let's discuss it on the mailing list first, then document in the repo when we have things (mostly ;)) working.
Taking a look at: https://github.com/bit-team/backintime/labels/Distro-Specific I would say that we can also make good use of Manjaro and CentOS for testing. In further Issues I haven't sorted out yet, there are some lurking problems with Wayland, so we'd best get a distro with that, too. Thanks! Michael
Hello all, I've created two Debian VMs for testing: - Debian Buster (Stable) with RSync 3.2.3 [0] - Debian Bookworm (Testing) RSync 3.2.5 [1] The VMs use official Debian images, update themselves to latest state, and install XFCE4 with Back in Time from repositories. The username and password of the systems are bit/bit. To automatically install virtualbox guest additions, please install vagrant-vbguest via "vagrant plugin install vagrant-vbguest". The systems already have semi-up to date guest utilities out of the box though. Also, Debian testing has an incompatibility with the plugin, so the Vagrantfile for Testing disables auto installation by default. To run these VMs, please put the relevant Vagrantfiles to their own folder and run "vagrant up". The rest should be automagic. I did not detail installation of Vagrant and Virtualbox, since I assume both should be in repositories of major distributions already. Next steps: 1. Write proper documentation. 2. Make systems login automatically (needs some script grease, had no time). 3. Add other distributions (CentOS, Fedora, Manjaro, Ubuntu). Please don't hesitate to contact for any questions, comments, suggestions, etc. Hope this helps, Cheers, H. [0]: https://git.sr.ht/~bayindirh/vagrant-machines/tree/master/item/debian11-back... [1]: https://git.sr.ht/~bayindirh/vagrant-machines/tree/master/item/debian12-back... On 11.09.2022 20:43, Michael Büker wrote:
Dear Hakan, awesome! ;) Am 12.09.2022 09:48 schrieb Hakan Bayındır:
run "vagrant up". The rest should be automagic.
Boah! Heaven! :D One question here: Does vagrant run the VMs all at the same time or one after the other? When having two or more VMs on my machine everything freeze. It doesn't matter for me if the testing will run multiple days on my slow home machine. ;)
Please don't hesitate to contact for any questions, comments, suggestions, etc.
I suspect that so far I have not fully grasped the purpose and use case for Vagrant. Therefore, I probably wish for some features that do not exist. I assume that Vagrant can configured that way that it start a test script automatically. But the key question for me here is where do the results go? Maybe there are 10 VMs with each 3 or 4 bigger tests. This maybe run 2 or 3 days on my home machine. Can I see a central log file or something like that at the end of that time period? I imagine something like a colored output of pytest or Travis. But as I said maybe I am to naive here. ;) Kind Christian
Dear Christian, and all, Vagrant is just a small layer on Virtualbox. The VMs are different files, so you can run either or both at the same time. Each of the machines have a single CPU with 2GB of RAM, so it wouldn't bog your system down. If you need more resources, you can tune via the Vagrantfile itself, but please tell me too, so I can update the files in the repository. Oh, I also forgot that you need to restart the machines once after everything is installed. "vagrant reload" should do this gracefully. Vagrant kindly reminds that after a successful provisioning though. We can create another VM which downloads the latest version from GitHub automatically and run the tests at first start or set itself up, so all tests can be run with a single command. These kinds of scenarios are what Vagrant is built for. Even, we can setup such a VM which can be our semi-official development environment, so contributors don't spend a day to be able to start contributing Back in Time. A single Vagrantfile can contain multiple VMs, too. After deciding the architecture, we can "summon" a virtual cluster to do the all the testing, however it would be heavy for some machines. Maybe we can utilize containers there (again, inside a VM), but we need to talk about it. I'll try to do my best to provide this infrastructure to you and the project, so we can move forward with less friction and greater momentum, while being sustainable with minimum effort. Please don't hesitate to contact for any questions, Cheers, Hakan On 12.09.2022 11:49, c.buhtz@posteo.jp wrote:
On Mon, 2022-09-12 at 08:49 +0000, c.buhtz@posteo.jp wrote:
I suspect that so far I have not fully grasped the purpose and use case for Vagrant
I will use it "just" for - issue testing (reproducing and fixing) that mentions a special Linux distribution and version - PR testing if it adds a feature that I do not have (or is too slow to boot/login) on my local dev computer (eg. different desktop environments like KDE and Gnome) - maybe manual testing of GUI and unit tests before releases
I personally would not go that far to setup a build matrix infrastructure like Travis because it is very tricky and time consuming to automate CI/CD. That is why we have TravisCI activated at Github
Good evening Am 12.09.2022 18:15 schrieb J. A.:
For me all this points are for a VM but for Vagrant because all that points are "manual" tasks.
My need is not about CI/CD (deployment, building). It is just automated testing in different environments. I just want to prevent myself from booting, test-script-starting, log-file-collecting for each of 10 or more VMs. Kind Christian
I just want to prevent myself from booting, test-script-starting, log-file-collecting for each of 10 or more VMs.
A have just tested the synced_folder feature of Vagrant to share files between host and guests. Just add this line to the vagrant config: # Share an additional folder to the guest VM. The first argument is # the path on the host to the actual folder. The second argument is # the path on the guest to mount the folder. And the optional third # argument is a set of non-required options. # # By default owner and group are set to the ssh user (vagrant?). Others are RO... # To grant another user access rights add the user to the vagrant group # (or change the mount options to allwe READ WRITE EXECUTE for everyone) config.vm.synced_folder "../shared_data", "/vagrant_shared_data", mount_options: ["dmode=777,fmode=777"] Then create a "shared" folder at the same folder level as your vagrant box folders (in order to not loose shared files if you delete or move a vagrant box folder). Then you can access the shared folder from within the VM via "/vagrant_shared_data".
On Sat, 2022-09-17 at 17:33 -0500, gsker wrote:
Would it be reasonable to ask you to repost the vagrant file URLs? I keep having to dig for them when testing your new updates.
The vagrant files were designed and provided by Hakan, not me, but he posted these links: https://git.sr.ht/~bayindirh/vagrant-machines/tree/master/item/debian11-back... https://git.sr.ht/~bayindirh/vagrant-machines/tree/master/item/debian12-back...
On Mon, 2022-09-12 at 10:48 +0300, Hakan Bayındır wrote:
Thanks a lot, saves much time. I am just running vagrant...
The VMs use official Debian images, update themselves to latest state, and install XFCE4 with Back in Time from repositories.
XFCE4 is gtk-based (like Gnome). Would it be possible to also include "plain vanilla" Gnome + KDE (since KDE used Qt and I need that to cross-test the GUI - eg. icons, system tray, notifications). X11 + wayland would also be good for Gnome - but let's I don't want to burn your time (I have a working machine for this this ATM). I would prefer to have at least 4 GB of RAM preconfigured due to the desktops and the need to start an IDE for debugging (but I also have luxurious 64 GB of RAM ;-)
3. Add other distributions (CentOS, Fedora, Manjaro, Ubuntu)
I have currently no overview over the OS distros and versions with the most issue reports but Fedora (by Red Hat) and possibly Arch BTW: German's famous ct computer magazin did publish a Linux Distro comparisson picture here which I like a quick overview over main distro differenences: ct.de/linuxplan
Hello, Thanks for the feedback. I will add different VMs with different DEs (adding many DEs to the same system is not a realistic scenario in real world). I have chosen XFCE4 & 2GB of RAM because I didn't want to bring people's machines down. Maybe for IDE, I can export a folder, so the IDE can live on the host machine while the source code is shared, what would you think about that? Of course I can (and will need to) increase the default RAM on some VMs, but, I just want to maximize flexibility. I may add Fedora relatively quickly since it's popular, easy to manage and is a good representation of RPM based world. I'll rather add other distributions as the consensus builds around them. I'll also start to write the documentation, but it might take a bit longer since I'm a bit busier than ordinary for the next month or so. However, if this effort accelerates the project in any way, I'll consider this as a win. Please don't hesitate to contact for any questions & feedback, Cheers, Hakan On 12.09.2022 16:08, J. A. wrote:
Moin! On 12.09.2022 15:08, J. A. wrote:
I'm trying to keep the https://github.com/bit-team/backintime/labels/Distro-Specific Label on GitHub up to date with distro-specific (and desktop-environment-specific) Issues, so you can take a look there to get some idea. I would also include X11/Wayland-specific Issues here, but I haven't gotten around to sorting them out yet.
How sweet :) Cheers! Michael
@Hakan On Mon, 2022-09-12 at 10:48 +0300, Hakan Bayındır wrote:
To run these VMs, please put the relevant Vagrantfiles to their own folder and run "vagrant up". The rest should be automagic.
Do you have hint for me? My installation on Ubuntu 20.04.5 LTS as host with VirtualBox 6.1 and vagrant 2.2.6 hangs here despite the VM is running and waiting at the login screen (see screen shot with VM and console). I have already tried to manually "cable connect" the network adapter 1 and 2, install vagrant-vbguest plugin and edit the Vagrant file like described here etc.: https://stackoverflow.com/questions/38463579/vagrant-hangs-at-ssh-auth-metho... BTW: How do I continue a broken installation, just via "vagrant up"? Thanks a lot :-) Jürgen config.vm.synced_folder '/host/path', '/guest/path', SharedFoldersEnableSymlinksCreate: false ==> default: Clearing any previously set network interfaces... ==> default: Preparing network interfaces based on configuration... default: Adapter 1: nat default: Adapter 2: hostonly ==> default: Forwarding ports... default: 22 (guest) => 2222 (host) (adapter 1) ==> default: Running 'pre-boot' VM customizations... ==> default: Booting VM... ==> default: Waiting for machine to boot. This may take a few minutes... default: SSH address: 127.0.0.1:2222 default: SSH username: vagrant default: SSH auth method: private key default: Warning: Authentication failure. Retrying... default: Warning: Authentication failure. Retrying... default: Warning: Authentication failure. Retrying... ... x times ... Timed out while waiting for the machine to boot. This means that Vagrant was unable to communicate with the guest machine within the configured ("config.vm.boot_timeout" value) time period. If you look above, you should be able to see the error(s) that Vagrant had when attempting to connect to the machine. These errors are usually good hints as to what may be wrong. If you're using a custom box, make sure that networking is properly working and you're able to connect to the machine. It is a common problem that networking isn't setup properly in these boxes. Verify that authentication configurations are also setup properly, as well. If the box appears to be booting properly, you may want to increase the timeout ("config.vm.boot_timeout") value. ~/vagrant >
Hmm, strange. I never experienced this. Let me research the issue and come back to you. This is completely unexpected
To narrow down the problem I have created a new simple box with "vagrant init" + only config.vm.box = "generic/ubuntu2204" And had exactly the same problems (with and without the *guest* plugin). So the problem must be with Ubuntu 20.04 (or Debian upstream) or the packaged vagrant version. I logged in as root and checked if the public ssh key was installed in ~/.ssh/ and it was. I also checked if the PPKs match (host == VM): cat .ssh/authorized_keys ssh-keygen -y -f ~/.vagrant.d/insecure_private_key Manually logging in via private key (without a password) ssh -i ~/.vagrant.d/insecure_private_key ssh://vagrant@127.0.0.1:2222 does work if I delete and re-accept the fingerprint of the "unknown" server... Good thing is: - I have learned the very basics of vagrant (thanks for your initiative!) - I have now two more different VMs that I can start at least manually without vagrant - I know now to to setup a new VM quickly by copying and modifying a Vagrantfile For the next VM I will download and install the latest vagrant version manually, so don't spend too much time to investigate this... BTW: I think the Vagrantfile for Debian 11 is not up-to-date (creation of the bit user is missing eg.) - forgot to commit/push? but copying the script from Debian 12 is straight-forward
SSH authentication issue SOLVED for my Ubuntu 20.04 host! default: SSH address: 127.0.0.1:2222 default: SSH username: vagrant default: SSH auth method: private key default: Warning: Authentication failure. Retrying... I have updated the Ubuntu-packaged vagrant version (2.2.6) via download from https://www.vagrantup.com/downloads to 2.3.0 and re-installed the vbguest plugin via vagrant plugin install vagrant-vbguest and the installation went straight through. Just configured RAM and 128 MB video memory (for higher resolutions)... On Mon, 2022-09-12 at 23:25 +0200, J. A. wrote:
Dear Hakan! Wonderful work! Thank you so much! I've used VirtualBox for many years, but this is the first time I've seen vagrant, and it's amazing :) I managed to set up the Debian 12 VM very quickly, log in and launch backintime. I think this will be a great help in maintaining backintime, especially for reproducing bugs and testing funtionality in different distros, desktop environments and X11/Wayland. Christian is mainly interested in running automated tests in all these environments -- and from what I've seen, that should be well possible with some SSH magic. Since my main machine has 32G of RAM, I'd also vote for 4G per VM :) I'm looking forward to having a way to use these VMs to checkout the git main branch (or some other branch) for testing :) We'd have to think about avoiding collisions between the repository-installed stock backintime and our git version, but you probably already have a good idea how to do that. So thank you very much so far, I'm very happy with the way this is going :) Cheers, Michael On 12.09.2022 09:48, Hakan Bayındır wrote:
Dear Christian and all, Thanks for your kind words. I'm glad that the VMs helped to accelerate the process. First of all, I have updated the VMs. The changelog is as follows: - [debian11] Fix: User creation script is added (I pushed the wrong file, sorry). - [debian11] Fix: Post-up message refers to correct Debian version. - [debian11/12] Feat: RAM is bumped to 4096MB. - [debian11/12] Feat: VRAM is bumped to 128MB. - [README.md] Doc: Reflect memory changes. I'm planning to create two more Debian VMs which automatically checks-out the latest code and installs any tooling necessary to develop and test Back in Time. These will be "-dev" machines. These "-dev" machines will need some scripting and setup but nothing impossible. Did something like that before. Just needs time. Then, we can add more distributions to "-lab" family to test with issues and other problems. So, my current roadmap is as follows: 1. Create "-dev" machines which gets the latest code from BiT repositories, and required tooling to work on the code. 2. Write proper documentation. 3. Make systems login automatically (needs some script grease, had no time). 4. Add other distributions (CentOS, Fedora, Manjaro, Ubuntu). Hope this plan helps, Please don't hesitate to contact for any questions and feedback, Cheers, Hakan On 12.09.2022 23:58, Michael Büker wrote:
Hi, Hakan! On 13.09.2022 09:50, Hakan Bayındır wrote:
Please don't hesitate to contact for any questions and feedback,
Maybe you can help me understand one thing: Vagrant seems to work in "environments", where each environment has its own folder that includes a "Vagrantfile", and has an automatically created ".vagrant" subfolder (similar to ".git", I guess). Each "environment" can apparently contain many "providers"(??). How do these "environments" and "providers" relate to your Vagrantfiles? Is each VM its own environment in its own folder? Or should they be combined somehow (now or in the future)? I've tried to Google some documentation on this, but I don't understand too much of what I find ;) Thanks :) Michael
Hello Michael, Yes, you’re right. Vagrant machines works in their own environments. Which is the folder where the Vagrantfile resides. Besides “.vagrant” folder, you can use the Vagrantfile to copy files from that folder during provision, or create folders which bridge the VM to the host for file sharing. While you may possibly reach outside this folder via Vagrantfile, storing everything in that folder allows zipping the folder, giving to someone else and saying, “just vagrant up”. Providers is something different. Vagrant is not limited to VirtualBox. Vagrant can work with VirtualBox, VMWare, libvirt, docker and many more. So your Vagrantfile can configure a Docker container instead of a VirtualBox VM, for example. The file will be different, however. Moreover, a single Vagrantfile can contain more than one virtual machine. I have worked with two Virtualbox VMs while working on something, but didn’t explore the concept further. However, for our case, I’ll rather create different machines, because bringing up a cluster of machines will not be practical and feasible for everyone. Hope this information sheds some more information on capabilities of Vagrant. Please don't hesitate to contact for any questions and feedback, As always, Cheers, Hakan
participants (5)
-
c.buhtz＠posteo.jp -
gsker -
Hakan Bayındır -
J. A. -
Michael Büker