From jmwright at waveapps.com Thu Sep 12 08:29:38 2024 From: jmwright at waveapps.com (Jean-Mark Wright) Date: Thu, 12 Sep 2024 08:29:38 -0400 Subject: [DB-SIG] Better Database Visibility Message-ID: Hi, I'm aiming to improve database trace quality to better understand what's happening in our database. I'd love to get index information, query planning information, performance statistics and richer errors into our database spans. I don't know what the best layer is for this enrichment (e.g. database module, OpenTelemetry, in our application, etc...). I'm looking for the following specifically: - Thoughts on the possibilities of getting this information and where it would come from? - Thoughts on whether this enrichment aligns with the database specification's purpose? - Any contacts who could help answer or provide more questions. This article captures the issues I've faced in greater detail. Thanks in advance for your help! -- *JEAN-MARK **WRIGHT* | Staff Engineer Join our community on Facebook , LinkedIn , or Instagram This message and any attachments are intended only for the use of the addressee and should be considered confidential. If you are not an intended recipient, you may not review, copy or distribute this message. If you have received this communication in error or would like to stop receiving these emails, please notify the sender by replying to this email. Wave is located at 500-155 Queens Quay East, Box 3, Toronto ON, M5A 0W4. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Sat Sep 14 17:00:56 2024 From: mal at egenix.com (Marc-Andre Lemburg) Date: Sat, 14 Sep 2024 23:00:56 +0200 Subject: [DB-SIG] Better Database Visibility In-Reply-To: References: Message-ID: <4930dfcf-251a-4c4d-a280-29cbecea81c7@egenix.com> Hi Jean-Mark, this list is about Python database connectivity. How does your request related to Python ? You also don't mention the database in question. There are plenty tools out there for monitoring and tracing databases out there, but those are typically not Python specific. Thanks. On 12.09.2024 14:29, Jean-Mark Wright via DB-SIG wrote: > Hi, > I'm aiming to improve database trace quality to better understand what's > happening in our database. I'd love to get index information, query > planning information, performance statistics and richer errors into our > database spans. I don't know what the best layer is for this enrichment > (e.g. database module, OpenTelemetry, in our application, etc...). > > I'm looking for the following specifically: > > * Thoughts on the possibilities of getting this information and where > it would come from? > * Thoughts on whether this enrichment aligns with the database > specification's purpose? > * Any contacts who could help answer or provide more questions. > > > This article > captures the issues I've faced in greater detail. > > Thanks in advance for your help! > > > -- > *JEAN-MARK **WRIGHT* | Staff Engineer > > > > Join our community onFacebook ,LinkedIn > ,?orInstagram > > > This message and any attachments are intended only for the use of the > addressee and should be considered confidential. If you are not an > intended recipient, you may not review, copy or distribute this message. > If you have received this communication in error or would like to stop > receiving these emails, please notify the sender by replying to this > email. Wave is located at 500-155 Queens Quay East, Box 3, Toronto ON, > M5A 0W4. > > _______________________________________________ > DB-SIG maillist - DB-SIG at python.org > https://mail.python.org/mailman/listinfo/db-sig -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Sep 14 2024) >>> Python Projects, Coaching and Support ... https://www.egenix.com/ >>> Python Product Development ... https://consulting.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/ From mal at egenix.com Mon Sep 16 15:01:04 2024 From: mal at egenix.com (Marc-Andre Lemburg) Date: Mon, 16 Sep 2024 21:01:04 +0200 Subject: [DB-SIG] Better Database Visibility In-Reply-To: References: <4930dfcf-251a-4c4d-a280-29cbecea81c7@egenix.com> Message-ID: On 16.09.2024 20:12, Jean-Mark Wright wrote: > Hi Marc-Andre, > > Thanks for your response. > > My request isn't directly related to Python. I reached out because > we're mostly a Python shop. Python is the primary space I'd be looking > to solve this problem for. I figure some folks would understand some > of the intricacies and challenges of achieving my goals and could help > me with pointers. Possibly I could make some amendments to the > database interface, or something else. We're using Postgres. > > > There are plenty tools out there for monitoring and tracing databases > > out there, but those are typically not Python-specific. > > We're using DataDog and switching providers isn't probable currently. > There's some space to augment the information we currently have. I'm > not aware of all the possibilities though. That's why I've reached out. PostgreSQL provides plenty of statistics to monitor and insight tooling: https://www.postgresql.org/docs/current/monitoring-stats.html https://www.postgresql.org/docs/current/monitoring.html You can access these using psycopg from Python, of course, and write custom logic to alert you. Or you can use Grafana or a similar tool to visualize them (via Prometheus or OpenTelemetry): https://grafana.com/solutions/postgresql/monitor/ It's also possible to ingest the data in OpenTelemetry, Nagios/Icinga, Zabbix, Checkmk, etc. There also plenty of commercial tools or commercial offerings for the OSS tools out there. Most of these (if not all), give you alerting, visuals, and data recording. This is a vast space and where to dig deeper depends a lot on your use cases, architecture and data sizes. E.g. if you're into tracing query plans, you could have a look at PG Ferret: https://github.com/ChrisBellew/pg-ferret Now, getting all this data is easy. What is hard, is making sense of it :-) > > On Sat, Sep 14, 2024 at 5:00?PM Marc-Andre Lemburg wrote: > > Hi Jean-Mark, > > this list is about Python database connectivity. How does your > request > related to Python ? You also don't mention the database in question. > > There are plenty tools out there for monitoring and tracing databases > out there, but those are typically not Python specific. > > Thanks. > > > On 12.09.2024 14:29, Jean-Mark Wright via DB-SIG wrote: > > Hi, > > I'm aiming to improve database trace quality to better > understand what's > > happening in our database. I'd love to get index information, query > > planning information, performance statistics and richer errors > into our > > database spans. I don't know what the best layer is for this > enrichment > > (e.g. database module, OpenTelemetry, in our application, etc...). > > > > I'm looking for the following specifically: > > > >? ?* Thoughts on the possibilities of getting this information > and where > >? ? ?it would come from? > >? ?* Thoughts on whether this enrichment aligns with the database > >? ? ?specification's purpose? > >? ?* Any contacts who could help answer or provide more questions. > > > > > > This article > > > > captures the issues I've faced in greater detail. > > > > Thanks in advance for your help! > > > > > > -- > > *JEAN-MARK **WRIGHT* | Staff Engineer > > > > > > > > > > Join our community onFacebook > ,LinkedIn > > ,?orInstagram > > > > > > This message and any attachments are intended only for the use > of the > > addressee and should be considered confidential. If you are not an > > intended recipient, you may not review, copy or distribute this > message. > > If you have received this communication in error or would like > to stop > > receiving these emails, please notify the sender by replying to > this > > email. Wave is located at 500-155 Queens Quay East, Box 3, > Toronto ON, > > M5A 0W4. > > > > _______________________________________________ > > DB-SIG maillist? - DB-SIG at python.org > > https://mail.python.org/mailman/listinfo/db-sig > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Experts (#1, Sep 14 > 2024) > ?>>> Python Projects, Coaching and Support ... https://www.egenix.com/ > ?>>> Python Product Development ... https://consulting.egenix.com/ > ________________________________________________________________________ > > ::: We implement business ideas - efficiently in both time and > costs ::: > > ? ? eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > ? ? ?D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > ? ? ? ? ? ? Registered at Amtsgericht Duesseldorf: HRB 46611 > https://www.egenix.com/company/contact/ > https://www.malemburg.com/ > > > > -- > *JEAN-MARK **WRIGHT* | Staff Engineer > > > > Join our community onFacebook > ,LinkedIn > ,?orInstagram > > > This message and any attachments are intended only for the use of the > addressee and should be considered confidential. If you are not an > intended recipient, you may not review, copy or distribute this > message. If you have received this communication in error or would > like to stop receiving these emails, please notify the sender by > replying to this email. Wave is located at 500-155 Queens Quay East, > Box 3, Toronto ON, M5A 0W4. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Sep 16 2024) >>> Python Projects, Coaching and Support ... https://www.egenix.com/ >>> Python Product Development ... https://consulting.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 https://www.egenix.com/company/contact/ https://www.malemburg.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jmwright at waveapps.com Mon Sep 16 14:12:33 2024 From: jmwright at waveapps.com (Jean-Mark Wright) Date: Mon, 16 Sep 2024 14:12:33 -0400 Subject: [DB-SIG] Better Database Visibility In-Reply-To: <4930dfcf-251a-4c4d-a280-29cbecea81c7@egenix.com> References: <4930dfcf-251a-4c4d-a280-29cbecea81c7@egenix.com> Message-ID: Hi Marc-Andre, Thanks for your response. My request isn't directly related to Python. I reached out because we're mostly a Python shop. Python is the primary space I'd be looking to solve this problem for. I figure some folks would understand some of the intricacies and challenges of achieving my goals and could help me with pointers. Possibly I could make some amendments to the database interface, or something else. We're using Postgres. > There are plenty tools out there for monitoring and tracing databases > out there, but those are typically not Python-specific. We're using DataDog and switching providers isn't probable currently. There's some space to augment the information we currently have. I'm not aware of all the possibilities though. That's why I've reached out. On Sat, Sep 14, 2024 at 5:00?PM Marc-Andre Lemburg wrote: > Hi Jean-Mark, > > this list is about Python database connectivity. How does your request > related to Python ? You also don't mention the database in question. > > There are plenty tools out there for monitoring and tracing databases > out there, but those are typically not Python specific. > > Thanks. > > > On 12.09.2024 14:29, Jean-Mark Wright via DB-SIG wrote: > > Hi, > > I'm aiming to improve database trace quality to better understand what's > > happening in our database. I'd love to get index information, query > > planning information, performance statistics and richer errors into our > > database spans. I don't know what the best layer is for this enrichment > > (e.g. database module, OpenTelemetry, in our application, etc...). > > > > I'm looking for the following specifically: > > > > * Thoughts on the possibilities of getting this information and where > > it would come from? > > * Thoughts on whether this enrichment aligns with the database > > specification's purpose? > > * Any contacts who could help answer or provide more questions. > > > > > > This article > > < > https://jaywhy13.hashnode.dev/dear-editor-we-need-better-database-observability> > captures the issues I've faced in greater detail. > > > > Thanks in advance for your help! > > > > > > -- > > *JEAN-MARK **WRIGHT* | Staff Engineer > > > > < > https://www.waveapps.com/>< > https://www.waveapps.com/> > > > > Join our community onFacebook ,LinkedIn > > , orInstagram > > > > > > This message and any attachments are intended only for the use of the > > addressee and should be considered confidential. If you are not an > > intended recipient, you may not review, copy or distribute this message. > > If you have received this communication in error or would like to stop > > receiving these emails, please notify the sender by replying to this > > email. Wave is located at 500-155 Queens Quay East, Box 3, Toronto ON, > > M5A 0W4. > > > > _______________________________________________ > > DB-SIG maillist - DB-SIG at python.org > > https://mail.python.org/mailman/listinfo/db-sig > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Experts (#1, Sep 14 2024) > >>> Python Projects, Coaching and Support ... https://www.egenix.com/ > >>> Python Product Development ... https://consulting.egenix.com/ > ________________________________________________________________________ > > ::: We implement business ideas - efficiently in both time and costs ::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > https://www.egenix.com/company/contact/ > https://www.malemburg.com/ > > -- *JEAN-MARK **WRIGHT* | Staff Engineer Join our community on Facebook , LinkedIn , or Instagram This message and any attachments are intended only for the use of the addressee and should be considered confidential. If you are not an intended recipient, you may not review, copy or distribute this message. If you have received this communication in error or would like to stop receiving these emails, please notify the sender by replying to this email. Wave is located at 500-155 Queens Quay East, Box 3, Toronto ON, M5A 0W4. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jmwright at waveapps.com Wed Sep 18 23:23:35 2024 From: jmwright at waveapps.com (Jean-Mark Wright) Date: Wed, 18 Sep 2024 23:23:35 -0400 Subject: [DB-SIG] Better Database Visibility In-Reply-To: References: <4930dfcf-251a-4c4d-a280-29cbecea81c7@egenix.com> Message-ID: Thanks for all the links! I'll check it out. On Mon, Sep 16, 2024 at 3:01?PM Marc-Andre Lemburg wrote: > On 16.09.2024 20:12, Jean-Mark Wright wrote: > > Hi Marc-Andre, > > Thanks for your response. > > My request isn't directly related to Python. I reached out because we're > mostly a Python shop. Python is the primary space I'd be looking to solve > this problem for. I figure some folks would understand some of the > intricacies and challenges of achieving my goals and could help me with > pointers. Possibly I could make some amendments to the database interface, > or something else. We're using Postgres. > > > There are plenty tools out there for monitoring and tracing databases > > out there, but those are typically not Python-specific. > > We're using DataDog and switching providers isn't probable currently. > There's some space to augment the information we currently have. I'm not > aware of all the possibilities though. That's why I've reached out. > > PostgreSQL provides plenty of statistics to monitor and insight tooling: > > https://www.postgresql.org/docs/current/monitoring-stats.html > > https://www.postgresql.org/docs/current/monitoring.html > > You can access these using psycopg from Python, of course, and write > custom logic to alert you. > > Or you can use Grafana or a similar tool to visualize them (via Prometheus > or OpenTelemetry): > > https://grafana.com/solutions/postgresql/monitor/ > > It's also possible to ingest the data in OpenTelemetry, Nagios/Icinga, > Zabbix, Checkmk, etc. > > There also plenty of commercial tools or commercial offerings for the OSS > tools out there. > > Most of these (if not all), give you alerting, visuals, and data recording. > > This is a vast space and where to dig deeper depends a lot on your use > cases, architecture and data sizes. > > E.g. if you're into tracing query plans, you could have a look at PG > Ferret: > > https://github.com/ChrisBellew/pg-ferret > > Now, getting all this data is easy. What is hard, is making sense of it :-) > > > > On Sat, Sep 14, 2024 at 5:00?PM Marc-Andre Lemburg wrote: > >> Hi Jean-Mark, >> >> this list is about Python database connectivity. How does your request >> related to Python ? You also don't mention the database in question. >> >> There are plenty tools out there for monitoring and tracing databases >> out there, but those are typically not Python specific. >> >> Thanks. >> >> >> On 12.09.2024 14:29, Jean-Mark Wright via DB-SIG wrote: >> > Hi, >> > I'm aiming to improve database trace quality to better understand >> what's >> > happening in our database. I'd love to get index information, query >> > planning information, performance statistics and richer errors into our >> > database spans. I don't know what the best layer is for this enrichment >> > (e.g. database module, OpenTelemetry, in our application, etc...). >> > >> > I'm looking for the following specifically: >> > >> > * Thoughts on the possibilities of getting this information and where >> > it would come from? >> > * Thoughts on whether this enrichment aligns with the database >> > specification's purpose? >> > * Any contacts who could help answer or provide more questions. >> > >> > >> > This article >> > < >> https://jaywhy13.hashnode.dev/dear-editor-we-need-better-database-observability> >> captures the issues I've faced in greater detail. >> > >> > Thanks in advance for your help! >> > >> > >> > -- >> > *JEAN-MARK **WRIGHT* | Staff Engineer >> > >> > < >> https://www.waveapps.com/>< >> https://www.waveapps.com/> >> > >> > Join our community onFacebook ,LinkedIn >> >> > , orInstagram >> > >> > >> > This message and any attachments are intended only for the use of the >> > addressee and should be considered confidential. If you are not an >> > intended recipient, you may not review, copy or distribute this >> message. >> > If you have received this communication in error or would like to stop >> > receiving these emails, please notify the sender by replying to this >> > email. Wave is located at 500-155 Queens Quay East, Box 3, Toronto ON, >> > M5A 0W4. >> > >> > _______________________________________________ >> > DB-SIG maillist - DB-SIG at python.org >> > https://mail.python.org/mailman/listinfo/db-sig >> >> -- >> Marc-Andre Lemburg >> eGenix.com >> >> Professional Python Services directly from the Experts (#1, Sep 14 2024) >> >>> Python Projects, Coaching and Support ... https://www.egenix.com/ >> >>> Python Product Development ... https://consulting.egenix.com/ >> ________________________________________________________________________ >> >> ::: We implement business ideas - efficiently in both time and costs ::: >> >> eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 >> D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg >> Registered at Amtsgericht Duesseldorf: HRB 46611 >> https://www.egenix.com/company/contact/ >> https://www.malemburg.com/ >> >> > > -- > *JEAN-MARK **WRIGHT* | Staff Engineer > > > > Join our community on Facebook , LinkedIn > , or Instagram > > > This message and any attachments are intended only for the use of the > addressee and should be considered confidential. If you are not an intended > recipient, you may not review, copy or distribute this message. If you have > received this communication in error or would like to stop receiving these > emails, please notify the sender by replying to this email. Wave is located > at 500-155 Queens Quay East, Box 3, Toronto ON, M5A 0W4. > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Experts (#1, Sep 16 2024) > >>> Python Projects, Coaching and Support ... https://www.egenix.com/ > >>> Python Product Development ... https://consulting.egenix.com/ > ________________________________________________________________________ > > ::: We implement business ideas - efficiently in both time and costs ::: > > eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 > D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg > Registered at Amtsgericht Duesseldorf: HRB 46611 > https://www.egenix.com/company/contact/ > https://www.malemburg.com/ > > -- *JEAN-MARK **WRIGHT* | Staff Engineer Join our community on Facebook , LinkedIn , or Instagram This message and any attachments are intended only for the use of the addressee and should be considered confidential. If you are not an intended recipient, you may not review, copy or distribute this message. If you have received this communication in error or would like to stop receiving these emails, please notify the sender by replying to this email. Wave is located at 500-155 Queens Quay East, Box 3, Toronto ON, M5A 0W4. -------------- next part -------------- An HTML attachment was scrubbed... URL: