Mailman 3 June 2018 - Pandas-dev

Temptative guidelines for indexing code
by Pietro Battiston June 29, 2018

June 29, 2018

Hi all, maybe they need discussion, or vice-versa they are quite obvious, but just to have a clear map of how we can improve indexing code, and "interpret" new issues/PRs/discussions regarding it, I put down some guidelines in the wiki: https://github.com/pandas-dev/pandas/wiki/(Temptative)-rules-for-restru cturing-indexing-code Joris suggested we can move it to the repo itself, as a sort of "design document", if at any time we want to ease a line-by-line review; for the moment, any comment is welcome. Pietro

2 1

Pandas development hangout - Thursday June 21 at 13:00 UTC
by Joris Van den Bossche June 19, 2018

June 19, 2018

Hi all, We're having a dev chat this Friday (May 25) at 9:00 Eastern / 14:00 UTC+1 / 15:00 CEST (Europe). All are welcome to attend. We will mainly discuss 0.23.2 release and preparation for the dev sprint before Scipy Conf. Hangout: https://hangouts.google.com/hangouts/_/calendar/am9yaXN2YW5kZW5ib3NzY2hlQGd… Calendar invite: https://calendar.google.com/event?action=TEMPLATE&tmeid=N2Z0ODh0NzdyMG1mMTF… Minutes: https://docs.google.com/document/d/1tGbTiYORHiSPgVMXawiweGJl Bw5dOkVJLY-licoBmBU/edit?usp=sharing Joris

1 1

How does Pandas from python reads the timestamp to datetime64? (HDF5)
by Arthuro Verissimo June 14, 2018

June 14, 2018

i'm creating an application on C# that generates a hdf5 document with one row of the compound data being a timestamp (using unix timestamp), and i wish i could load this hdf5 in pandas and it load the timestamp column as a datetime64. I've tried to create the file from pandas and i saw it creates the dataset with a lot of attributes that i assume pandas read and do some logic to convert it while reading the file (the datetime inside the file created from pandas is a timestamp too), i can't figure it out... Here is my dataset creation code and some screenshots of what i had investigated public void CriaHDF5Customizado(PackingConfigFile pf) { // PopulaOPC(pf); H5FileId arquivoHDF5 = H5F.create("C:/Users/arthuro/Desktop/timestampteste.h5", H5F.CreateMode.ACC_TRUNC); H5GroupId datasetGroup = H5G.create(arquivoHDF5, "Datasets"); H5GroupId infosGroup = H5G.create(arquivoHDF5, "Informations"); H5G.close(infosGroup); opcSt opHelper = new opcSt(); opHelper.dt = (Int64)DateTime.Parse("1996-11-9").Subtract(new DateTime(1970, 1, 1,12,00,00, DateTimeKind.Utc)).TotalSeconds; opHelper.qlt = (Int64)DateTime.Now.Subtract(new DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc)).TotalSeconds; opHelper.vl = 123456.7f; int structsize = Marshal.SizeOf(opHelper); H5DataTypeId myOPCType = H5T.create(H5T.CreateClass.COMPOUND, structsize); //H5T.insert(myOPCType, "TimeStamp", 0, new H5DataTypeId(H5T.H5Type.NATIVE_LONG)); H5T.insert(myOPCType, "Quality", 8, new H5DataTypeId(H5T.H5Type.NATIVE_LONG)); H5T.insert(myOPCType, "Value", 16, new H5DataTypeId(H5T.H5Type.NATIVE_FLOAT)); long[] ds = new long[1]; ds[0] = 1; H5DataTypeId dtDatetime = H5T.create_array(new H5DataTypeId(H5T.H5Type.NATIVE_LLONG), ds); H5T.insert(myOPCType, "TimeStamp", 0, dtDatetime); foreach (BasicVariable bv in pf.basicVariableList.bvList) { bv.PopulateOPCUA(); var bvnow = bv; var bvNro = pf.basicVariableList.bvList.IndexOf(bv); long[] dims = new long[1]; dims[0] = bv.bvData.Count; H5DataSpaceId myDataSpace = H5S.create_simple(1, dims); H5DataSetId bvDset = H5D.create(datasetGroup, bv.bvTag, myOPCType, myDataSpace); var arrayaux = new List<opcSt>(); foreach (OPC_UA opc in bv.bvData) { var aux = new opcSt(opc.timeStamp, opc.quality, (float)opc.data); arrayaux.Add(aux); } H5D.write(bvDset, myOPCType, new H5Array<opcSt>(arrayaux.ToArray())); string[] stringteste = { "datetime64" }; H5DataTypeId attrDt = H5T.copy(H5T.H5Type.C_S1); H5T.setSize(attrDt, stringteste[0].Length); var longsz = new long[] { 1 }; var enc = new System.Text.ASCIIEncoding(); var array1 = enc.GetBytes(stringteste[0]); var charArray = new byte[stringteste[0].Length + 1]; array1.CopyTo(charArray, 0); charArray[stringteste[0].Length] = 0; H5DataSpaceId atribDS = H5S.create_simple(1, longsz); H5AttributeId Attribs = H5A.create(bvDset, "TimeStamp_dtype", attrDt, atribDS); H5A.write(Attribs, attrDt, new H5Array<byte>(charArray)); stringteste[0] = "(lp0L0La."; H5T.setSize(attrDt, stringteste[0].Length); longsz = new long[] { 1 }; enc = new System.Text.ASCIIEncoding(); array1 = enc.GetBytes(stringteste[0]); charArray = new byte[stringteste[0].Length + 1]; array1.CopyTo(charArray, 0); charArray[stringteste[0].Length] = 0; atribDS = H5S.create_simple(1, longsz); H5AttributeId Attribs2 = H5A.create(bvDset, "TimeStamp_kind", attrDt, atribDS); H5A.write(Attribs2, attrDt, new H5Array<byte>(charArray)); stringteste[0] = "N."; H5T.setSize(attrDt, stringteste[0].Length); longsz = new long[] { 1 }; enc = new System.Text.ASCIIEncoding(); array1 = enc.GetBytes(stringteste[0]); charArray = new byte[stringteste[0].Length + 1]; array1.CopyTo(charArray, 0); charArray[stringteste[0].Length] = 0; atribDS = H5S.create_simple(1, longsz); H5AttributeId Attribs3 = H5A.create(bvDset, "TimeStamp_meta", attrDt, atribDS); H5A.write(Attribs3, attrDt, new H5Array<byte>(charArray)); H5S.close(atribDS); H5A.close(Attribs); H5A.close(Attribs2); H5A.close(Attribs3); H5D.close(bvDset); H5S.close(myDataSpace); bv.bvData = new List<OPC_UA>(); } H5G.close(datasetGroup); H5F.close(arquivoHDF5); } Pandas reading (timestamp and datetime): being 1- The value read inside jupyter notebook (processed by pandas) 2- The real value inside the HDF5 File https://i.stack.imgur.com/ZZJ3A.png <https://github.com/pandas-dev/pandas/issues/url> Ahd these are the attributes of the table created by pandas (that i've tried to recreate in my code) https://i.stack.imgur.com/U1Zom.png <https://github.com/pandas-dev/pandas/issues/url>

1 0

ANN: Pandas 0.23.1 Released
by Tom Augspurger June 13, 2018

June 13, 2018

Hi all, I'm happy to announce pandas that pandas 0.23.1 has been released. This is a minor bug-fix release in the 0.23.x series and includes some regression fixes, bug fixes, and performance improvements. We recommend that all users upgrade to this version. See the full whatsnew <https://pandas.pydata.org/pandas-docs/version/0.23.1/whatsnew.html#v0-23-1> for a list of all the changes. The release can be installed with conda from the default channel and conda-forge:: conda install pandas Or via PyPI: python -m pip install --upgrade pandas A total of 30 people contributed patches to this release. People with a “+” by their names contributed a patch for the first time. - Adam J. Stewart - Adam Kim + - Aly Sivji - Chalmer Lowe + - Damini Satya + - Dr. Irv - Gabe Fernando + - Giftlin Rajaiah - Jeff Reback - Jeremy Schendel + - Joris Van den Bossche - Kalyan Gokhale + - Kevin Sheppard - Matthew Roeschke - Max Kanter + - Ming Li - Pyry Kovanen + - Stefano Cianciulli - Tom Augspurger - Uddeshya Singh + - Wenhuan - William Ayd - chris-b1 - gfyoung - h-vetinari - nprad + - ssikdar1 + - tmnhat2001 - topper-123 - zertrin +

1 0

Pandas dev sprint July 5-8 (pre-Scipy Conf)
by Joris Van den Bossche June 12, 2018

June 12, 2018

Dear all, We are planning to do a pandas sprint just before the Scipy Conference (which is in Austin, Texas, July 9-13). The sprint would then be from July 5 or 6 to July 8, also in Austin. We will meet there with some of the core devs, but we also want to invite other pandas (or related libraries) contributors. Note: the sprint is not meant to jump-start newcomers to contributing to open source (for that, there is a sprint at the end of the Scipy). The exact planning of the sprint still needs to be discussed (will open a separate thread for that), but topics will probably include: general issue clean-up, discussing project management, roadmap to pandas 1.0 (what do we still want to get in / change before that) and evolution of pandas afterwards, ... and of course actual coding. *So if you are interested, certainly let something know! * We probably want to keep it somewhat limited in number, but don't hesitate to state your interest. If you would like to participate, but not sure if you would fit at such a sprint, don't hesitate to mail me personally. If you would like to participate, but would need financial support, you can also mail me (we might have some funds, but not fully sure yet). Best, Joris

1 1