Mailman 3 May 2024 - Pandas-dev

pandas new infrastructure (OVH donation)
by Marc Garcia Sept. 21, 2024

Sept. 21, 2024

Hi all, pandas has received a donation from OVHcloud <https://www.ovhcloud.com/> to support the project infrastructure, with OVHcloud public cloud credits (an initial amount of 10,000 EUR for a period of one year). OVH is open to sponsor longer term and also other projects of the ecosystem (or NumFOCUS as a whole), but we started with this to have feedback at a smaller scale first. The credits will be used initially for: - Hosting of the pandas website - Running the pandas benchmarks - Speeding up the project CI I detail next what I have in mind to set up for each. If anyone is interested in getting involved, or has ideas, comments... please let me know. I'll publish updates here as there is progress on this. Website: I'm planning to experiment on splitting the website in two (it'll be transparent for users). The website and the stable docs which receive most of the traffic can probably be stored in Cloudflare pages. We're already using Cloudflare as a CDN, so instead of using it as a cache, we can publish the documents there. The rest of the docs (old versions and the dev version) can be hosted in bucket storage of the OVHcloud. Response times may be a bit slower, but our website is bigger than the Cloudflare quota, and having old docs rarely accessed in a CDN seems unnecessary anyway. - Benchmarks: OVHcloud instances have guaranteed hardware, and we'll be checking if this is enough for the results of the benchmarks to be consistent over runs, or if there is too much variability and we need dedicated hardware. If consistency is good enough that would be great, since our benchmarks mostly use one core, and using dedicated hardware is likely to be a decent waste of resources, since most servers will likely have 16 cores or more. We'll discuss with OVH if dedicated hardware is needed, as at the moment their public cloud doesn't offer it (there is an alpha for providing dedicated instances, but we need to check with them). - Faster CI: Our GitHub runners are small, and most builds take around one hour or more to finish. We should be able to use bigger OVH instances for our existing CI pretty easily, via their OpenStack API and CIrun.

5 9

February 2023 bi-monthly community meeting (Wednesday February 8, UTC 18:00)
by Joris Van den Bossche Sept. 11, 2024

Sept. 11, 2024

Hi all, A late reminder that the next bi-monthly (twice a month) dev call is today in a bit more than 1 hour (Wednesday, February 8) at 18:00 UTC. Our calendar is at https://pandas.pydata.org/docs/development/meeting.html#calendar to check your local time. The pandas Community Meeting is a regular sync meeting for the project's maintainers which is open to the community. All are welcome to attend! Video Call: https://us06web.zoom.us/j/84484803210?pwd=TjUxNmcyNHcvcG9SNGJvbE53Y21GZz09 Meeting notes: https://docs.google.com/document/u/1/d/1tGbTiYORHiSPgVMXawiweGJlBw5dOkVJLY-… Joris

1 1

PDEP-14: Dedicated string data type for pandas 3.0
by Joris Van den Bossche June 14, 2024

June 14, 2024

Hi all, I want to notify you of a new PDEP enhancement proposal being discussed at https://github.com/pandas-dev/pandas/pull/58551. Rendered version of the current text: https://jorisvandenbossche.github.io/pandas-website-preview/pdeps/0014-stri… *Summary* This PDEP proposes to introduce a dedicated string dtype that will be used by default in pandas 3.0: - In pandas 3.0, enable a "string" dtype by default, using PyArrow if available or otherwise a string dtype using numpy object-dtype under the hood as fallback. - The default string dtype will use missing value semantics (using NaN) consistent with the other default data types. This will give users a long-awaited proper string dtype for 3.0, while 1) not (yet) making PyArrow a *hard* dependency, but only a dependency used by default, and 2) leaving room for future improvements (different missing value semantics, using NumPy 2.0 strings, etc) after 3.0. *Backwards compatibility* The biggest compatibility issues will be present for users of the existing StringDtype. Users can already specify dtype="string" for several years, and right now that gives a string dtype that uses pd.NA as the missing value sentinel, while with this proposal that will change to NaN by default in pandas 3.0. The current proposal is to start raising a deprecation warning for this in pandas 2.3 before changing the behaviour in pandas 3.0, and users will have the option to explicitly keep using the variant of the dtype that uses pd.NA (although most users should be fine with the new default). See the "Backwards compatibility" section in the PDEP text. For the full proposal and background, see the links above. Feedback welcome!

4 6

Privacy Policy - Panda
by Louise Kitney May 29, 2024

May 29, 2024

Good Morning, I work for a UK based Academy Trust as a Data Compliance Officer, and one of our academies has requested to install Pandas. As part of our own internal processes for this, I have been tasked with reviewing the GDPR impact of installing and using this software. I have been unable to locate a privacy policy on your website; are you able to provide one or advise on the types of personal data the software requires upon install or to use? Many thanks Louise *Louise Kitney* *Data Compliance Officer* Leigh Academies Trust Carnation Road, Strood, Rochester, Kent, ME2 2SX *Email:* louise.kitney(a)latrust.org.uk -- Visit our new* **IT Porta*l <https://lat.sharepoint.com/sites/ITServices> for easy access to all of our policies and forms. -- This e-mail communication contains information that is confidential and may also be privileged. It is intended for the exclusive use of the addressee(s). If you have received this communication in error, please notify Leigh Academies Trust immediately (Tel: +44(0)1634 412 200). This email is sent on behalf of Leigh Academies Trust which is registered in England. The company registration no. is 2336587 with registered office address at Carnation Road, Strood, Rochester, Kent, ME2 2SX, UK. For more information about Leigh Academies Trust please visit www.leighacademiestrust.org.uk <http://www.leighacademiestrust.org.uk> Disclaimer The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful. This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast, a leader in email security and cyber resilience. Mimecast integrates email defenses with brand protection, security awareness training, web security, compliance and other essential capabilities. Mimecast helps protect large and small organizations from malicious activity, human error and technology failure; and to lead the movement toward building a more resilient world. To find out more, visit our website.

2 1

Is it time for a bi-weekly pandas call?
by Marco Gorelli May 13, 2024

May 13, 2024

Currently, there's a monthly pandas call, which has been in place for several years. Seeing as there's now more people working on pandas as part of their jobs, might it be time to increase the frequency? E.g. to meet every 2 weeks, instead of once a month?

6 11