Stefanie, * ? If the bed file contains 10.000 samples and I pass SnpReader.iid with 300, only those 300 will be used to build the model right? That would be super convenient... Yes, that is correct. The single_snp will use only individuals that are common to the test snps, K0 (of given), K1 (if given), phenotype, and covar (if given). It will also reorder the individuals if needed so they are in the same order from all the inputs. Happy to hear things are working well. * Carl From: Stefanie Lück <luecks@gmail.com> Sent: Monday, May 10, 2021 10:32 PM To: Carl KADIE <carlk@msn.com> Cc: fastlmm-user@python.org Subject: Re: [fastlmm-user] Kinship Importance: High Hi, Thanks a lot Car, for this detailed information! I missed the ids in my .npz file, so that's the problem. I have one more question for the bed format: from pysnptools.snpreader import Bed
bedfile = example_file("tests/datasets/distributed_bed_test1_X.*","*.bed")
If I understand correctly, when passing SnpReader.iid information, fast-lmm will use only those sample IDs for the run, is this correct? If the bed file contains 10.000 samples and I pass SnpReader.iid with 300, only those 300 will be used to build the model right? That would be super convenient... Thanks again for maintaining fast-lmm! I set it up already with a small flask server and everythings works great so far :) Cheers Stefanie Am Mo., 10. Mai 2021 um 17:46 Uhr schrieb Carl KADIE <carlk@msn.com<mailto:carlk@msn.com>>: Stefanie, Good questions. * 1) Which format do I have to provide if I use the npz format? My npz matrix throws the error below. Here is how to create an in-memory kernel from a numpy array: Example
from pysnptools.kernelreader import KernelData kerneldata = KernelData(iid=[['fam0','iid0'],['fam0','iid1']], val=[[1.,.5],[.5,1.]]) print((kerneldata.val[0,1], kerneldata.iid_count)) (0.5, 2)
This kerneldata object can be passed in to single_snp. You can also write that kerneldata object to *.kernel.npz format like this: from pysnptools.kernelreader import KernelNpz KernelNpz.write("tempdir/toydata.kernel.npz",kerneldata) (Aside: *.kernel.npz is a version of *.npz that includes not only the kernel itself, but also information about the individual id's and optimal metadata) After that you could pass KernelNpz("tempdir/toydata.kernel.npz") to single_snp For more details, follow up with me or see https://fastlmm.github.io/PySnpTools/#module-pysnptools.kernelreader<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ffastlmm.github.io%2FPySnpTools%2F%23module-pysnptools.kernelreader&data=04%7C01%7C%7C06c03a0482f24738bce808d9143e2b4c%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637563079507758021%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=AZ4Qq0XTpu02nMM6T7a85rt%2BF217hobaG3fZZMUzf%2Fc%3D&reserved=0> * 2) If I don't provide K0, how is the similarity matrix calculated and is it possible to store the matrix for other runs? If K0 is not provided, it is computed from the test SNP data. (Aside: when leave-one-chromsome-out is applicable, there is actually one K0 for each chromosome left out). The SNP data is standardized (made to have mean 0 and std dev 1 and missing filled with 0) to Gx and then K0 is Gx.T.dot(Gx) where T is numpy's transpose and dot is (for some reason) numpy's matrix multiply. Let me know if this is clear. I'm happy to help with Python format conversion problems if you let me know about the format you're starting with. Thanks again for using FaST-LMM! * Carl From: Stefanie Lück <luecks@gmail.com<mailto:luecks@gmail.com>> Sent: Monday, May 10, 2021 12:25 AM To: fastlmm-user@python.org<mailto:fastlmm-user@python.org> Subject: [fastlmm-user] Kinship Importance: High Hi again, I have two questions using the similarity matrix in single_snp: 1) Which format do I have to provide if I use the npz format? My npz matrix throws the error below. 2) If I don't provide K0, how is the similarity matrix calculated and is it possible to store the matrix for other runs? Thanks a lot Stefanie Error: Traceback (most recent call last): File "C:/Users//PycharmProjects/BCC_Experiments/lmm/lmm01.py", line 27, in <module> results_df = single_snp(bed_fn, pheno_fn, K0=k) File "C:\Users\\AppData\Local\Continuum\anaconda3\envs\gwas_flow\lib\site-packages\fastlmm\association\single_snp.py", line 246, in single_snp runner = runner) File "C:\Users\\AppData\Local\Continuum\anaconda3\envs\gwas_flow\lib\site-packages\pysnptools\util\mapreduce1\mapreduce.py", line 202, in map_reduce result = runner.run(dist) File "C:\Users\\AppData\Local\Continuum\anaconda3\envs\gwas_flow\lib\site-packages\pysnptools\util\mapreduce1\runner\local.py", line 48, in run result = _run_all_in_memory(distributable) File "C:\Users\\AppData\Local\Continuum\anaconda3\envs\gwas_flow\lib\site-packages\pysnptools\util\mapreduce1\runner\__init__.py", line 30, in _run_all_in_memory return work.reduce(result_sequence) File "C:\Users\\AppData\Local\Continuum\anaconda3\envs\gwas_flow\lib\site-packages\pysnptools\util\mapreduce1\mapreduce.py", line 77, in reduce return self.reducer(output_seq) File "C:\Users\\AppData\Local\Continuum\anaconda3\envs\gwas_flow\lib\site-packages\fastlmm\association\single_snp.py", line 228, in reducer_closure frame = pd.concat(frame_sequence) File "C:\Users\\AppData\Local\Continuum\anaconda3\envs\gwas_flow\lib\site-packages\pandas\core\reshape\concat.py", line 295, in concat sort=sort, File "C:\Users\\AppData\Local\Continuum\anaconda3\envs\gwas_flow\lib\site-packages\pandas\core\reshape\concat.py", line 339, in __init__ objs = list(objs) File "C:\Users\\AppData\Local\Continuum\anaconda3\envs\gwas_flow\lib\site-packages\pysnptools\util\mapreduce1\runner\__init__.py", line 14, in work_sequence_to_result_sequence result = work() File "C:\Users\\AppData\Local\Continuum\anaconda3\envs\gwas_flow\lib\site-packages\pysnptools\util\mapreduce1\mapreduce.py", line 65, in <lambda> yield lambda i=i, input_arg=input_arg: self.dowork(i, input_arg) # the 'i=i',etc is need to get around a strangeness in Python File "C:\Users\\AppData\Local\Continuum\anaconda3\envs\gwas_flow\lib\site-packages\pysnptools\util\mapreduce1\mapreduce.py", line 92, in dowork result = _run_all_in_memory(work) File "C:\Users\\AppData\Local\Continuum\anaconda3\envs\gwas_flow\lib\site-packages\pysnptools\util\mapreduce1\runner\__init__.py", line 25, in _run_all_in_memory return work() File "C:\Users\\AppData\Local\Continuum\anaconda3\envs\gwas_flow\lib\site-packages\pysnptools\util\mapreduce1\mapreduce.py", line 91, in <lambda> work = lambda : self.mapper(input_arg) File "C:\Users\\AppData\Local\Continuum\anaconda3\envs\gwas_flow\lib\site-packages\fastlmm\association\single_snp.py", line 211, in nested_closure K0_chrom = _K_per_chrom(K0 or G0 or test_snps, chrom, test_snps.iid) File "C:\Users\\AppData\Local\Continuum\anaconda3\envs\gwas_flow\lib\site-packages\fastlmm\association\single_snp.py", line 301, in _K_per_chrom return SnpKernel(K_all.snpreader[:,K_all.pos[:,0] != chrom],K_all.standardizer) File "C:\Users\\AppData\Local\Continuum\anaconda3\envs\gwas_flow\lib\site-packages\pysnptools\kernelreader\snpkernel.py", line 150, in pos return self.snpreader.pos File "C:\Users\\AppData\Local\Continuum\anaconda3\envs\gwas_flow\lib\site-packages\pysnptools\snpreader\snpreader.py", line 404, in pos return self.col_property File "C:\Users\\AppData\Local\Continuum\anaconda3\envs\gwas_flow\lib\site-packages\pysnptools\pstreader\pstnpz.py", line 67, in col_property self._run_once() File "C:\Users\\AppData\Local\Continuum\anaconda3\envs\gwas_flow\lib\site-packages\pysnptools\pstreader\pstnpz.py", line 82, in _run_once self._row = data['row'] File "C:\Users\\AppData\Local\Continuum\anaconda3\envs\gwas_flow\lib\site-packages\numpy\lib\npyio.py", line 259, in __getitem__ raise KeyError("%s is not a file in the archive" % key) KeyError: 'row is not a file in the archive' Process finished with exit code 1