Mailman 3 July 2021 - fastlmm-user

New FaST-LMM 0.6.0: Now works with any Python distribution
by Carl K July 21, 2021

July 21, 2021

Greetings, We're happy to announce a new version of FaST-LMM. Version 0.6.0 is easier to install and works with any Python distribution, not just Miniconda. To install: pip uninstall fastlmm (if previously install) pip install fastlmm You can use any Python distribution that supports NumPy. For best performance, be sure your Python distribution includes a fast version of NumPy. We still use Anaconda's Miniconda. As before, FaST-LMM runs under Python 3.7, 3.8, or 3.9 and runs on x64 versions of Linux, Windows, and MacOS. (We may now be able to support ARM in the future. Let us know if there is interest.) Yours, Carl Carl Kadie, Ph.D. FaST-LMM & PySnpTools Team<https://fastlmm.github.io/> (Microsoft Research, retired) https://www.linkedin.com/in/carlk/ Join the FaST-LMM user discussion and announcement list via email<mailto:fastlmm-user-join@python.org?subject=Subscribe> (or use web sign up<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.pyth…>)

1 0

New version of FaST-LMM
by Carl K July 15, 2021

July 15, 2021

Greetings, I'm happy to announce a new version of FaST-LMM with new features for epistasis. Recall that FaST-LMM allows you to test for epistatic interactions between pairs of SNPs. New: * The output now includes β, the linear coefficient of the epistatic interaction, and its variance. Specifically, the epistatic interaction is the product of standardized and is a fixed effect. Thanks to Ava Khamseh of the University of University of Edinburgh for her derivation of the formula for the variance<https://github.com/fastlmm/FaST-LMM/raw/epibeta/doc/notes/FaST_LMM_variance…> and for requesting this feature. Thanks to FaST-LMM team members Christian Widmer and David Heckerman for their work on this feature. * User-supplied eigenvalue decomposition for epistasis - If you know the eigenvalue decomposition of realized relationship matrix (RRM), the epistasis function can use that information to run much faster. For examples of the new features see: fastlmm2021.ipynb#epistasis<https://nbviewer.jupyter.org/github/fastlmm/FaST-LMM/blob/master/doc/ipynb/…> To install the new version: * pip uninstall fastlmm · conda install "mkl==2019.4" "scipy" "numpy" · pip install --no-build-isolation fastlmm Yours, Carl Carl Kadie, Ph.D. FaST-LMM & PySnpTools Team<https://fastlmm.github.io/> (Microsoft Research, retired) https://www.linkedin.com/in/carlk/ Join the FaST-LMM user discussion and announcement list via email<mailto:fastlmm-user-join@python.org?subject=Subscribe> (or use web sign up<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.pyth…>)

1 0

New User-Requested PySnpTools Features
by Carl KADIE July 3, 2021

July 3, 2021

Greetings, I'm happy to announce PySnpTools version 0.5.3. The new features were all at user request. The features are available to any program that uses PySnpTools, including FaST-LMM. The new features are: * The Bed reader can now handle non-numeric chromosome names, for example, "X" and "Y". Likewise, the Bed writer can handle non-numeric chromosomes names. * The Bed reader can now use *.fam and *.bim files with non-standard names. * The SnpMemMap writer can now get data from any SnpReader, not just the in-memory SnpData. This, for example, allows even low-memory machines to create terabyte-sized memory-mapped files from *.bed files. * The Dat reader can skip lines at the top of the file. * For details, see the PySnpTools documentation<https://fastlmm.github.io/PySnpTools/#snpreader-bed>. * (Bonus: Bed-Reader which used to have an undocumented function for low-memory computers to create SNP-similarity matrices from memory-mapped files, now has an undocumented function to create kinship matrices from memory-mapped files. Let me know if you want details.) You can get the new features with: pip uninstall pysnptools pip install pysnptools Thanks for the great suggestions! Yours, Carl and the FaST-LMM team

1 0

I've turned on moderation of the FaST-LMM User mailing list.
by Carl KADIE July 3, 2021

July 3, 2021

Greetings, Sorry for the three recent spam emails to fastlmm-user. I've turned on moderation for this mailing list so there shouldn't be any more. Yours, Carl Carl Kadie, Ph.D. FaST-LMM & PySnpTools Team<https://fastlmm.github.io/> (Microsoft Research, retired) https://www.linkedin.com/in/carlk/ Join the FaST-LMM user discussion and announcement list via email<mailto:fastlmm-user-join@python.org?subject=Subscribe> (or use web sign up<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.pyth…>)

1 0

Release of PySnpTools 0.4.19
by Carl KADIE July 3, 2021

July 3, 2021

Greetings, We're released a new version of PySnpTools (0.4.19). Here are the new features: * Bug fixed in the "Bgen" reader<https://fastlmm.github.io/PySnpTools/#module-pysnptools.distreader>. It has now has been tested on files as large 487,400 individuals x 4,840,000 SNPs (the size of the UK Biobank imputed genotype data). It should work with even larger files. * New option when reading from "Bed"<https://fastlmm.github.io/PySnpTools/#snpreader-bed> files directly into "int8" arrays. This saves 3x memory and time compared to reading into a "float32" array and converting. * Carl Carl Kadie, Ph.D. FaST-LMM & PySnpTools Team (Microsoft Research, retired) https://fastlmm.github.io/ Join the FaST-LMM user discussion and announcement list via email<mailto:fastlmm-user-join@python.org?subject=Subscribe> (or use web sign up<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.pyth…>)

2 1

New releases of FaST-LMM and PySnpTools
by Carl KADIE July 3, 2021

July 3, 2021

I’m happy to announce a new releases of FaST-LMM<https://pypi.org/project/fastlmm/> and PySnpTools<https://pypi.org/project/pysnptools/>. (This release been my “work” since I retired last summer.) The new releases updates both packages to work with the newest version of Pandas, Numpy, and Scikit-learn. The new FaST-LMM release includes single_snp_scale, which allows FaST-LMM to use a cluster and scale to 1 million individuals. See Kadie and Heckerman, bioRxiv 2018<https://www.biorxiv.org/content/10.1101/154682v2> for background. Similar tools would require 100,000 computers to scale this much, but FaST-LMM needs “only” a cluster of 100 computers. (The code can run on any cluster but to run on a particular cluster we must create a module detailing how to automate batch jobs and move files.) The new PySnpTools release adds support for cluster-sized data. Including: * snpreader.SnpGen<https://fastlmm.github.io/PySnpTools/#pysnptools.snpreader.SnpGen>: Generate synthetic SNP data on the fly. * snpreader.SnpMemMap<https://fastlmm.github.io/PySnpTools/#pysnptools.snpreader.SnpMemMap>: Support larger in-memory data via on-disk memory mapping. * snpreader.DistributedBed<https://fastlmm.github.io/PySnpTools/#pysnptools.snpreader.DistributedBed>: Split Bed<https://fastlmm.github.io/PySnpTools/#pysnptools.snpreader.Bed>-like data into multiple files for more efficient cluster use * util.mapreduce1<https://fastlmm.github.io/PySnpTools/#module-pysnptools.util.mapreduce1>: Run loops in parallel on multiple processes, threads, or clusters * util.filecache<https://fastlmm.github.io/PySnpTools/#module-pysnptools.util.filecache>: Automatically copy files to and from any remote storage. FaST-LMM and PySnpTools were originally developed and open sourced at Microsoft Research. Active development has now based at https://fastlmm.github.io/. Roadmap: I plan to continue working on FaST-LMM and PySnpTools. We’d like to run a giant job on real, rather than synthetic, data. We like to compare it other fast methods that we suspect sacrifice accuracy. I’d like to port it from Python 2 to Python 3. (More todo’s: analyze multiple traits in one run, analyze pairs of DNA locations using the single-DNA-location tools, …) Contacts: Email the developers at fastlmm-dev(a)python.org<mailto:fastlmm-dev@python.org>. Join<mailto:fastlmm-user-join@python.org?subject=Subscribe> the user discussion and announcement list (or use web sign up<https://mail.python.org/mailman3/lists/fastlmm-user.python.org>). Yours, Carl Carl Kadie, Ph.D. FaST-LMM Team

2 1

More multiple phenotype support in FaST-LMM
by Carl KADIE July 2, 2021

July 2, 2021

Greetings, I'm happy to announce support for multiple phenotypes by FaST-LMM's single_snp function. Running with more phenotypes can generate more data, so we've also added new features related to caching and filtering out unneeded output lines. A new notebook<https://nbviewer.jupyter.org/github/fastlmm/FaST-LMM/blob/master/doc/ipynb/…> demonstrates all the new features. A university user requested these feature and then used them to do a multi-day run with 16,000 phenotypes. That run found a couple areas with promising associations. To install FaST-LMM, start with Miniconda or Anaconda. (If already installed, un-install it with "pip uninstall fastlmm".) Then ... conda install "mkl==2019.4" "scipy" "numpy" pip install --no-build-isolation fastlmm * Carl

5 4