New Bed-Reader 0.2.9: Writing Bed files now 3 to 6 times faster
Greetings, We're happy to announce another new version of Bed-Reader, the sub-package of FaST-LMM and PySnpTools for reading and writing PLINK Bed 1.9 files. The new feature is multithreaded writing to Bed files. On my machine, with 6 processors, writing times for big files goes from 66 seconds to 11 seconds. (See this short Reddit/r/rust article<https://www.reddit.com/r/rust/comments/sby5g5/solution_to_a_puzzle_any_nice_...> on the software engineering details.) Also, if anyone needs a Bed file writer that can write the largest files (for example, 1 TB on disk), let me know. I'm interested in creating it, but don't know if there is a need. Yours, Carl Carl Kadie, Ph.D. FaST-LMM & PySnpTools Team<https://fastlmm.github.io/> (Microsoft Research, retired) https://www.linkedin.com/in/carlk/ Join the FaST-LMM user discussion and announcement list via email<mailto:fastlmm-user-join@python.org?subject=Subscribe> (or use web sign up<https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.python.org%2Fmailman3%2Flists%2Ffastlmm-user.python.org&data=02%7C01%7C%7C13a5c33d7cd84cad5cdf08d7bba56e20%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637184191498409587&sdata=2CQWjQEwOpQol2rQ1eoyVTgY8WvInV8UH31Wtl68FzY%3D&reserved=0>)
Hi Carl, That's really cool and very useful, thanks a lot! I personally work with data of GB range and I am fine (fast ready is more important). But sequence coverage is growing all the time :) Greetings Stefanie Am Di., 25. Jan. 2022 um 16:39 Uhr schrieb Carl K <carlk@msn.com>:
Greetings,
We’re happy to announce another new version of Bed-Reader, the sub-package of FaST-LMM and PySnpTools for reading and writing PLINK Bed 1.9 files.
The new feature is multithreaded writing to Bed files. On my machine, with 6 processors, writing times for big files goes from 66 seconds to 11 seconds.
(See this short Reddit/r/rust article <https://www.reddit.com/r/rust/comments/sby5g5/solution_to_a_puzzle_any_nice_...> on the software engineering details.)
Also, if anyone needs a Bed file writer that can write the largest files (for example, 1 TB on disk), let me know. I’m interested in creating it, but don’t know if there is a need.
Yours,
Carl
Carl Kadie, Ph.D. FaST-LMM & PySnpTools Team <https://fastlmm.github.io/> (Microsoft Research, retired) https://www.linkedin.com/in/carlk/
Join the FaST-LMM user discussion and announcement list via email <fastlmm-user-join@python.org?subject=Subscribe> (or use web sign up <https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.python.org%2Fmailman3%2Flists%2Ffastlmm-user.python.org&data=02%7C01%7C%7C13a5c33d7cd84cad5cdf08d7bba56e20%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637184191498409587&sdata=2CQWjQEwOpQol2rQ1eoyVTgY8WvInV8UH31Wtl68FzY%3D&reserved=0> )
_______________________________________________ fastlmm-user mailing list -- fastlmm-user@python.org To unsubscribe send an email to fastlmm-user-leave@python.org https://mail.python.org/mailman3/lists/fastlmm-user.python.org/ Member address: luecks@gmail.com
Indeed, thanks Carl! David From: Stefanie Lück <luecks@gmail.com> Sent: Wednesday, January 26, 2022 10:58 PM To: Carl K <carlk@msn.com> Cc: fastlmm-user@python.org; fastlmm-dev@python.org Subject: [fastlmm-dev] Re: [fastlmm-user] New Bed-Reader 0.2.9: Writing Bed files now 3 to 6 times faster Hi Carl, That's really cool and very useful, thanks a lot! I personally work with data of GB range and I am fine (fast ready is more important). But sequence coverage is growing all the time :) Greetings Stefanie Am Di., 25. Jan. 2022 um 16:39 Uhr schrieb Carl K <carlk@msn.com<mailto:carlk@msn.com>>: Greetings, We're happy to announce another new version of Bed-Reader, the sub-package of FaST-LMM and PySnpTools for reading and writing PLINK Bed 1.9 files. The new feature is multithreaded writing to Bed files. On my machine, with 6 processors, writing times for big files goes from 66 seconds to 11 seconds. (See this short Reddit/r/rust article<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.reddit.com%2Fr%2Frust%2Fcomments%2Fsby5g5%2Fsolution_to_a_puzzle_any_nice_way_to_multithread%2F&data=04%7C01%7C%7C87d6a47165484c45bab008d9e1627029%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637788635179910685%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=QztjKNGk4%2B%2FjbgDLZC%2F9UPrasQg%2BseofjkI%2F8p4o3Eg%3D&reserved=0> on the software engineering details.) Also, if anyone needs a Bed file writer that can write the largest files (for example, 1 TB on disk), let me know. I'm interested in creating it, but don't know if there is a need. Yours, Carl Carl Kadie, Ph.D. FaST-LMM & PySnpTools Team<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ffastlmm.github.io%2F&data=04%7C01%7C%7C87d6a47165484c45bab008d9e1627029%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637788635179910685%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=QQfq86Ilo8KPaImJBnh5g6EtJdjVVqZ2HmqgGeKB4Rk%3D&reserved=0> (Microsoft Research, retired) https://www.linkedin.com/in/carlk/<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fin%2Fcarlk%2F&data=04%7C01%7C%7C87d6a47165484c45bab008d9e1627029%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637788635179910685%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=850RD11QHSw4RqyOvfGhawmuHV7fG0EReXdKenOHRP4%3D&reserved=0> Join the FaST-LMM user discussion and announcement list via email<mailto:fastlmm-user-join@python.org?subject=Subscribe> (or use web sign up<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.python.org%2Fmailman3%2Flists%2Ffastlmm-user.python.org&data=04%7C01%7C%7C87d6a47165484c45bab008d9e1627029%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637788635179910685%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=%2F%2F7XGIgKzgDyhFgITxJpckHtkAmpgTyUqoSw3dT3Tsw%3D&reserved=0>) _______________________________________________ fastlmm-user mailing list -- fastlmm-user@python.org<mailto:fastlmm-user@python.org> To unsubscribe send an email to fastlmm-user-leave@python.org<mailto:fastlmm-user-leave@python.org> https://mail.python.org/mailman3/lists/fastlmm-user.python.org/<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmail.python.org%2Fmailman3%2Flists%2Ffastlmm-user.python.org%2F&data=04%7C01%7C%7C87d6a47165484c45bab008d9e1627029%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637788635179910685%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Xsev%2BmAgli5WniFGP5lfmZyMCkwdt29957WXh%2FmM0Fk%3D&reserved=0> Member address: luecks@gmail.com<mailto:luecks@gmail.com>
participants (3)
-
Carl K -
David Heckerman -
Stefanie Lück