Mailman 3 September 2020 - fastlmm-user

Release of PySnpTools 0.4.19
by Carl KADIE July 3, 2021

July 3, 2021

Greetings, We're released a new version of PySnpTools (0.4.19). Here are the new features: * Bug fixed in the "Bgen" reader<https://fastlmm.github.io/PySnpTools/#module-pysnptools.distreader>. It has now has been tested on files as large 487,400 individuals x 4,840,000 SNPs (the size of the UK Biobank imputed genotype data). It should work with even larger files. * New option when reading from "Bed"<https://fastlmm.github.io/PySnpTools/#snpreader-bed> files directly into "int8" arrays. This saves 3x memory and time compared to reading into a "float32" array and converting. * Carl Carl Kadie, Ph.D. FaST-LMM & PySnpTools Team (Microsoft Research, retired) https://fastlmm.github.io/ Join the FaST-LMM user discussion and announcement list via email<mailto:fastlmm-user-join@python.org?subject=Subscribe> (or use web sign up<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.pyth…>)

2 1

New releases of FaST-LMM and PySnpTools
by Carl KADIE July 2, 2021

July 2, 2021

I’m happy to announce a new releases of FaST-LMM<https://pypi.org/project/fastlmm/> and PySnpTools<https://pypi.org/project/pysnptools/>. (This release been my “work” since I retired last summer.) The new releases updates both packages to work with the newest version of Pandas, Numpy, and Scikit-learn. The new FaST-LMM release includes single_snp_scale, which allows FaST-LMM to use a cluster and scale to 1 million individuals. See Kadie and Heckerman, bioRxiv 2018<https://www.biorxiv.org/content/10.1101/154682v2> for background. Similar tools would require 100,000 computers to scale this much, but FaST-LMM needs “only” a cluster of 100 computers. (The code can run on any cluster but to run on a particular cluster we must create a module detailing how to automate batch jobs and move files.) The new PySnpTools release adds support for cluster-sized data. Including: * snpreader.SnpGen<https://fastlmm.github.io/PySnpTools/#pysnptools.snpreader.SnpGen>: Generate synthetic SNP data on the fly. * snpreader.SnpMemMap<https://fastlmm.github.io/PySnpTools/#pysnptools.snpreader.SnpMemMap>: Support larger in-memory data via on-disk memory mapping. * snpreader.DistributedBed<https://fastlmm.github.io/PySnpTools/#pysnptools.snpreader.DistributedBed>: Split Bed<https://fastlmm.github.io/PySnpTools/#pysnptools.snpreader.Bed>-like data into multiple files for more efficient cluster use * util.mapreduce1<https://fastlmm.github.io/PySnpTools/#module-pysnptools.util.mapreduce1>: Run loops in parallel on multiple processes, threads, or clusters * util.filecache<https://fastlmm.github.io/PySnpTools/#module-pysnptools.util.filecache>: Automatically copy files to and from any remote storage. FaST-LMM and PySnpTools were originally developed and open sourced at Microsoft Research. Active development has now based at https://fastlmm.github.io/. Roadmap: I plan to continue working on FaST-LMM and PySnpTools. We’d like to run a giant job on real, rather than synthetic, data. We like to compare it other fast methods that we suspect sacrifice accuracy. I’d like to port it from Python 2 to Python 3. (More todo’s: analyze multiple traits in one run, analyze pairs of DNA locations using the single-DNA-location tools, …) Contacts: Email the developers at fastlmm-dev(a)python.org<mailto:fastlmm-dev@python.org>. Join<mailto:fastlmm-user-join@python.org?subject=Subscribe> the user discussion and announcement list (or use web sign up<https://mail.python.org/mailman3/lists/fastlmm-user.python.org>). Yours, Carl Carl Kadie, Ph.D. FaST-LMM Team

2 1

New FaST-LMM Version: 4.0.11
by Carl KADIE Sept. 23, 2020

Sept. 23, 2020

Greetings, I'm happy to announce a new version of FaST-LMM. Install from https://pypi.org/project/fastlmm/. What's New: * Support for Python 3.8 The new version allows FaST-LMM to work with Anaconda's recent Python 3.8 release. FaST-LMM now runs on Python 3.7 and 3.8, on Linux, Windows, and Mac. * The single_snp function now supports per chromosome precomputed similarity matrices and covariates. https://fastlmm.github.io/FaST-LMM/#single-snp * Via the new PySnpTools, up to 5 times faster loading of PLINK Bed and of BGEN files. * Carl Carl Kadie, Ph.D. FaST-LMM & PySnpTools Team (Microsoft Research, retired) Join the FaST-LMM user discussion and announcement list via email<mailto:fastlmm-user-join@python.org?subject=Subscribe> (or use web sign up<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.pyth…>)

2 1

PySnpTools 4.0.22
by Carl KADIE Sept. 17, 2020

Sept. 17, 2020

Greetings, I'm happy to announce the release of PySnpTools 4.0.22. Upgrade with: pip install --upgrade pysnptools This version offers new multithreaded reading for both the PLINK BED format and the BGEN format. On my 6 processor machine, on UK Biobank-like data, it now reads 111 giant SNPs/sec & 162M probabilities/second (was 30 SNPs and 46M probs). * Carl

1 0

'bed-reader': A New Standalone PLINK 1.9 .bed Reader & Writer
by Carl KADIE Sept. 8, 2020

Sept. 8, 2020

Greetings, I'm happy to announce a new standalone reader and writer for the PLINK .bed format. See https://pypi.org/project/bed-reader/. Example: >>> from bed_reader import open_bed, sample_file >>> >>> file_name = sample_file("small.bed") >>> bed = open_bed(file_name) >>> val = bed.read() >>> print(val) [[ 1. 0. nan 0.] [ 2. 0. nan 2.] [ 0. 1. 2. 0.]] >>> del bed Compared to the PySnpTools's Bed reader, the new reader/writer: * Reads on multiple-threads, up to 5 times faster on my machine. * Can read data as int8's, saving memory * Can read any fields in the *.fam and *.bim files. * Can read genotype data without needing to read *.fam or *.bim files at all. * Has fewer dependencies, so faster to install. The new bed-reader, like PySnpTools, can read slices of individuals (samples) and/or SNPs (variants). Unlike PySnpTools, bed-reader can't read or manipulate other file (and in-memory) formats. (The next version of PySnpTools will, under the covers, use bed-reader as its engine for reading and writing .bed files.) Thanks to Nicolo Fusi, who wrote the original .bed C++ code used by FaST-LMM, then PySnpTools, and now bed-reader. Thanks to users John Gregg and the SGKit project (especially Eric Czech and Jeff Hammerbacher) for their requests and suggestions. Thanks to Danilo Horta, whose bgen-reader project, served as a model for updating bed-reader's code. * Carl Carl Kadie, Ph.D. FaST-LMM & PySnpTools Team (Microsoft Research, retired) Join the FaST-LMM user discussion and announcement list via email<mailto:fastlmm-user-join@python.org?subject=Subscribe> (or use web sign up<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.pyth…>)

1 0