Mailman 3 July 2024 - Pandas-dev

pandas new infrastructure (OVH donation)
by Marc Garcia Sept. 21, 2024

Sept. 21, 2024

Hi all, pandas has received a donation from OVHcloud <https://www.ovhcloud.com/> to support the project infrastructure, with OVHcloud public cloud credits (an initial amount of 10,000 EUR for a period of one year). OVH is open to sponsor longer term and also other projects of the ecosystem (or NumFOCUS as a whole), but we started with this to have feedback at a smaller scale first. The credits will be used initially for: - Hosting of the pandas website - Running the pandas benchmarks - Speeding up the project CI I detail next what I have in mind to set up for each. If anyone is interested in getting involved, or has ideas, comments... please let me know. I'll publish updates here as there is progress on this. Website: I'm planning to experiment on splitting the website in two (it'll be transparent for users). The website and the stable docs which receive most of the traffic can probably be stored in Cloudflare pages. We're already using Cloudflare as a CDN, so instead of using it as a cache, we can publish the documents there. The rest of the docs (old versions and the dev version) can be hosted in bucket storage of the OVHcloud. Response times may be a bit slower, but our website is bigger than the Cloudflare quota, and having old docs rarely accessed in a CDN seems unnecessary anyway. - Benchmarks: OVHcloud instances have guaranteed hardware, and we'll be checking if this is enough for the results of the benchmarks to be consistent over runs, or if there is too much variability and we need dedicated hardware. If consistency is good enough that would be great, since our benchmarks mostly use one core, and using dedicated hardware is likely to be a decent waste of resources, since most servers will likely have 16 cores or more. We'll discuss with OVH if dedicated hardware is needed, as at the moment their public cloud doesn't offer it (there is an alpha for providing dedicated instances, but we need to check with them). - Faster CI: Our GitHub runners are small, and most builds take around one hour or more to finish. We should be able to use bigger OVH instances for our existing CI pretty easily, via their OpenStack API and CIrun.

5 9

February 2023 bi-monthly community meeting (Wednesday February 8, UTC 18:00)
by Joris Van den Bossche Sept. 11, 2024

Sept. 11, 2024

Hi all, A late reminder that the next bi-monthly (twice a month) dev call is today in a bit more than 1 hour (Wednesday, February 8) at 18:00 UTC. Our calendar is at https://pandas.pydata.org/docs/development/meeting.html#calendar to check your local time. The pandas Community Meeting is a regular sync meeting for the project's maintainers which is open to the community. All are welcome to attend! Video Call: https://us06web.zoom.us/j/84484803210?pwd=TjUxNmcyNHcvcG9SNGJvbE53Y21GZz09 Meeting notes: https://docs.google.com/document/u/1/d/1tGbTiYORHiSPgVMXawiweGJlBw5dOkVJLY-… Joris

1 1

Cost of pandas.read_pickle()
by c.buhtz＠posteo.jp July 25, 2024

July 25, 2024

Hello, my question in short: When using pandas.read_pickle() what cause the most costs in terms of performance and time? The file need to be read from the file system (IO performance) into RAM. Unpickeling will cost CPU time. I think about how to speed up my workflow. A database is not an option, mostly because of administrative reasons. I have several data frames with multiple of 10 Mio lines. My pipeline is separated into several scripts where each one is one step. Between this steps I temporary store the data frames as pickle files on the hard drive. Am I assuming it correct that the unpickeling has the biggest part in this process? In case of that it would make sense (to me) to parallelize that. In my workflow it is no problem to cut a big data frame into several pieces before pickling it. Then reading each pickle on its own CPU core and finally concat them back together. Does this sound reasonable to you? Best, Christian Buht

3 4

Boolean: convert string to boolean
by Luc Kesters July 23, 2024

July 23, 2024

Hi All, Something that confuses people sometimes when trying to convert a column with string values “true”, “false”, “no”, “yes”,… If one uses the read_csv method there is the option to handle this with the parameters true_values and false_values. In case one has columns that are still in dtype object, one can make the mistake by using the as_type method. I have looked in the documentation and issues on github but I can’t find any discussion on this subject. There doesn’t seem to be a pandas.to_bool(…) method like there is for example for datetimes (pandas.to_datetime(..)). So my conclusion is that one should use the apply method with a custom function. Or am I missing something? Kind regards, Luc

2 1

[PyCon KR] Request for a Congratulatory Video for PyCon Korea's 10th Anniversary
by 박성민 July 8, 2024

July 8, 2024

Hello, This is the Session Team from PyCon Korea Organizing Team. PyCon Korea is celebrating its 10th anniversary this year, and we are preparing a special video to mark this milestone. In this regard, we would be honored to include a congratulatory message from you, as you have been a significant source of inspiration within the Python community. Your congratulatory message would be deeply meaningful and inspirational to the many people who have been part of PyCon Korea's history. If possible, we would greatly appreciate it if you could record a short video of about 5-10 minutes, including the following points: - A brief self-introduction and congratulations on PyCon Korea's 10th anniversary - Examples of how you have utilized Python - Your opinions on Python/the ecosystem (impact, interest, affection, etc.) - Personal experiences/impressive event content/what you hope for the event - Your thoughts on the future or strategies for industries utilizing Python - Closing remarks - Any additional comments you would like to add - Guide Link : https://docs.google.com/presentation/d/1_NIQkthjn0OWqrYT31F4CfgC5m1UopBD1GD… Please refer to the submission deadline and format below and reply to this email with your video. If you have any questions, please feel free to contact us at any time. - Submission deadline: 23:59:59 (KST), August 15, 2024 - Submission format: .mp4 or .mov (HEVC, H.265, or H.264) - Additional Information/Script: If you think that it contains information that is technical or should not be misunderstood, please attach explanatory text or script files on the email. Thank you very much for taking the time out of your busy schedule. We look forward to your congratulatory message making PyCon Korea's 10th anniversary even more special. Thank you. Best regards, PyCon Korea Organizing Team

1 0