Mailman 3 January 2024 - Pandas-dev

pandas new infrastructure (OVH donation)
by Marc Garcia Sept. 21, 2024

Sept. 21, 2024

Hi all, pandas has received a donation from OVHcloud <https://www.ovhcloud.com/> to support the project infrastructure, with OVHcloud public cloud credits (an initial amount of 10,000 EUR for a period of one year). OVH is open to sponsor longer term and also other projects of the ecosystem (or NumFOCUS as a whole), but we started with this to have feedback at a smaller scale first. The credits will be used initially for: - Hosting of the pandas website - Running the pandas benchmarks - Speeding up the project CI I detail next what I have in mind to set up for each. If anyone is interested in getting involved, or has ideas, comments... please let me know. I'll publish updates here as there is progress on this. Website: I'm planning to experiment on splitting the website in two (it'll be transparent for users). The website and the stable docs which receive most of the traffic can probably be stored in Cloudflare pages. We're already using Cloudflare as a CDN, so instead of using it as a cache, we can publish the documents there. The rest of the docs (old versions and the dev version) can be hosted in bucket storage of the OVHcloud. Response times may be a bit slower, but our website is bigger than the Cloudflare quota, and having old docs rarely accessed in a CDN seems unnecessary anyway. - Benchmarks: OVHcloud instances have guaranteed hardware, and we'll be checking if this is enough for the results of the benchmarks to be consistent over runs, or if there is too much variability and we need dedicated hardware. If consistency is good enough that would be great, since our benchmarks mostly use one core, and using dedicated hardware is likely to be a decent waste of resources, since most servers will likely have 16 cores or more. We'll discuss with OVH if dedicated hardware is needed, as at the moment their public cloud doesn't offer it (there is an alpha for providing dedicated instances, but we need to check with them). - Faster CI: Our GitHub runners are small, and most builds take around one hour or more to finish. We should be able to use bigger OVH instances for our existing CI pretty easily, via their OpenStack API and CIrun.

5 9

February 2023 bi-monthly community meeting (Wednesday February 8, UTC 18:00)
by Joris Van den Bossche Sept. 11, 2024

Sept. 11, 2024

Hi all, A late reminder that the next bi-monthly (twice a month) dev call is today in a bit more than 1 hour (Wednesday, February 8) at 18:00 UTC. Our calendar is at https://pandas.pydata.org/docs/development/meeting.html#calendar to check your local time. The pandas Community Meeting is a regular sync meeting for the project's maintainers which is open to the community. All are welcome to attend! Video Call: https://us06web.zoom.us/j/84484803210?pwd=TjUxNmcyNHcvcG9SNGJvbE53Y21GZz09 Meeting notes: https://docs.google.com/document/u/1/d/1tGbTiYORHiSPgVMXawiweGJlBw5dOkVJLY-… Joris

1 1

Is it time for a bi-weekly pandas call?
by Marco Gorelli May 13, 2024

May 13, 2024

Currently, there's a monthly pandas call, which has been in place for several years. Seeing as there's now more people working on pandas as part of their jobs, might it be time to increase the frequency? E.g. to meet every 2 weeks, instead of once a month?

6 11

OSS Validation and Verification Project Assistance
by Ganesh, Sriram Jan. 29, 2024

Jan. 29, 2024

Good Evening, I am currently part of a team of students involved in an OSS Validation and Verification Project as part of an Advanced Software Engineering Graduate class at Purdue University. We were tasked with picking an OSS, and finding defects such as (1)overhauling a test suite; or (2) applying some state-of the art techniques to the project. We are specifically interested in pandas. Our goal is to address issues identifying and addressing bugs, vulnerabilities, and security holes through testing and validation. We have looked at the list of issues provided at https://github.com/pandas-dev/pandas/issues. Is there any specific issue that you would like looked at as part of this Project? Ideally this would be something we accomplish with a two month timeframe, but we are flexible. We are open to any suggestions on issues to look into and open to chat as well. Thanks for the help! Thanks, Sriram Ganesh BS Purdue ME 23' MS Purdue ME 25' e: ganesh24(a)purdue.edu

1 0

Conbench PoC - benchmarks
by Dea María Jan. 23, 2024

Jan. 23, 2024

Hi All, I have been working on a PoC to use conbench with pandas asv benchmarks, and it is ready to be reviewed. The setup is ready to start sending regression/improvement alerts by email. An email group will be created soon where those of you who are interested can subscribe. I shared a document on Slack that includes a quick user guide. Or it can be seen here: https://drive.google.com/file/d/1xTrLbocmKBoHtRzvBNUMJGTnxojRh_Sp/view?usp=… The link to the PoC is here: http://57.128.112.95:5000/ I look forward to hearing your thoughts. Dea

1 0

ANN: pandas v2.2.0
by Thomas Li Jan. 22, 2024

Jan. 22, 2024

We are pleased to announce the release of pandas 2.2.0. This release includes some new features, bug fixes, and performance improvements. We recommend that all users upgrade to this version. See the [full whatsnew](https://pandas.pydata.org/pandas-docs/version/2.2 .0/whatsnew/v2.2.0.html) for a list of all the changes. Pandas 2.2.0 supports Python 3.9 and higher. The release will be available on the defaults and conda-forge channels: conda install pandas Or via PyPI: python3 -m pip install --upgrade pandas Please report any issues with the release on the [pandas issue tracker]( https://github.com/pandas-dev/pandas/issues). Thanks to all the contributors who made this release possible.

1 0

How to replace a cell value with each of its contour cells and yield the corresponding datasets seperately in a list according to a Pandas-way?
by marc nicole Jan. 21, 2024

Jan. 21, 2024

Hello, I have an initial dataframe with a random list of target cells (each cell being identified with a couple (x,y)). I want to yield four different dataframes each containing the value of one of the contour (surrounding) cells of each specified target cell. the surrounding cells to consider for a specific target cell are : (x-1,y), (x,y-1),(x+1,y);(x,y+1), specifically I randomly choose 1 to 4 cells from these and consider for replacement to the target cell. I want to do that through a pandas-specific approach without having to define the contour cells separately and then apply the changes on the dataframe (but rather using an all in one approach): for now I have written this example which I think is not Pandas specific: *def select_target_values(dataframe, number_of_target_values): target_cells = [] for _ in range(number_of_target_values): row_x = random.randint(0, len(dataframe.columns) - 1) col_y = random.randint(0, len(dataframe) - 1) target_cells.append((row_x, col_y)) return target_cellsdef select_contours(target_cells): contour_coordinates = [(0, 1), (1, 0), (0, -1), (-1, 0)] contour_cells = [] for target_cell in target_cells: # random contour count for each cell contour_cells_count = random.randint(1, 4) try: contour_cells.append( [tuple(map(lambda i, j: i + j, (target_cell[0], target_cell[1]), contour_coordinates[iteration_])) for iteration_ in range(contour_cells_count)]) except IndexError: continue return contour_cells def apply_contours(target_cells, contour_cells): target_cells_with_contour = [] # create one single list of cells for idx, target_cell in enumerate(target_cells): target_cell_with_contour = [target_cell] target_cell_with_contour.extend(contour_cells[idx]) target_cells_with_contour.append(target_cell_with_contour) return target_cells_with_contour def create_possible_datasets(dataframe, target_cells_with_contour): all_datasets_final = [] dataframe_original = dataframe.copy() #check for nans list_tuples_idx_cells_all_datasets = list(filter(lambda x: utils_tuple_list_not_contain_nan(x), [list(tuples) for tuples in list(itertools.product( *target_cells_with_contour))])) target_original_cells_coordinates = list(map(lambda x: x[0], [target_and_contour_cell for target_and_contour_cell in target_cells_with_contour])) for dataset_index_values in list_tuples_idx_cells_all_datasets: all_datasets = [] for idx_cell in range(len(dataset_index_values)): dataframe_cpy = dataframe.copy() dataframe_cpy.iat[ target_original_cells_coordinates[idx_cell][1], target_original_cells_coordinates[idx_cell][ 0]] = dataframe_original.iloc[dataset_index_values[idx_cell][1], dataset_index_values[idx_cell][0]] all_datasets.append(dataframe_cpy) all_datasets_final.append(all_datasets) return all_datasets_final* If you have a better Pandas approach (unifying all these methods into one that make use of dataframe methods only) please let me know. thanks!

2 1