Mailman 3 including psutil in the standard library? - Python-ideas

newer
A way to tell the gc it's OK to...

including psutil in the standard library?

older
Store shared/locked state inside...

Stefano Borini

24 Sep 2014 24 Sep '14

2:21 p.m.

I am wondering if it would be possible to include psutil (https://pypi.python.org/pypi/psutil ) in the standard library, and if not, what would be needed. I am not a developer of it, but I am using psutil at work with good success. it provides a good deal of services for querying and managing processes in a cross platform way. Any thoughts?

Show replies by thread

Ryan Gonzalez

24 Sep 24 Sep

3:10 p.m.

I would honestly prefer it to be merged into an already in-place module (maybe platform?) over it being added separately. On Wed, Sep 24, 2014 at 2:21 PM, Stefano Borini < stefano.borini@ferrara.linux.it> wrote:

...

I am wondering if it would be possible to include psutil ( https://pypi.python.org/pypi/psutil ) in the standard library, and if not, what would be needed.

I am not a developer of it, but I am using psutil at work with good success. it provides a good deal of services for querying and managing processes in a cross platform way.

Any thoughts? _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

-- Ryan If anybody ever asks me why I prefer C++ to C, my answer will be simple: "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was nul-terminated." Personal reality distortion fields are immune to contradictory evidence. - srean Check out my website: http://kirbyfan64.github.io/

Jonas Wielicki

3:13 p.m.

On 24.09.2014 22:10, Ryan Gonzalez wrote:

...

I would honestly prefer it to be merged into an already in-place module (maybe platform?) over it being added separately.

That would unneccessarily break code. Also, os would be a better fit, if any. platform contains rather static informational data. regards, jwi

...

On Wed, Sep 24, 2014 at 2:21 PM, Stefano Borini < stefano.borini@ferrara.linux.it> wrote:

...
I am wondering if it would be possible to include psutil ( https://pypi.python.org/pypi/psutil ) in the standard library, and if not, what would be needed.

I am not a developer of it, but I am using psutil at work with good success. it provides a good deal of services for querying and managing processes in a cross platform way.

Any thoughts? _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

Barry Warsaw

3:24 p.m.

On Sep 24, 2014, at 03:10 PM, Ryan Gonzalez wrote:

...

I would honestly prefer it to be merged into an already in-place module (maybe platform?) over it being added separately.

Why platform? That seems out of place. If it were to be adopted into the stdlib, I think a top-level module name would be fine. psutil is great, but has anybody asked upstream if they even want to be part of the stdlib? Cheers, -Barry

M.-A. Lemburg

3:30 p.m.

On 24.09.2014 22:24, Barry Warsaw wrote:

...

On Sep 24, 2014, at 03:10 PM, Ryan Gonzalez wrote:

...
I would honestly prefer it to be merged into an already in-place module (maybe platform?) over it being added separately.

Why platform? That seems out of place.

Agreed. The typical approach is to put a renamed version into the stdlib, so that there's a possibility to continue using the package from PyPI.

...

If it were to be adopted into the stdlib, I think a top-level module name would be fine. psutil is great, but has anybody asked upstream if they even want to be part of the stdlib?

That'll have to happen first, indeed. The next question to answer is: Who will maintain the code in the stdlib ? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 24 2014)

...

...
...
Python Projects, Consulting and Support ... http://www.egenix.com/ mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

2014-09-27: PyDDF Sprint 2014 ... 3 days to go 2014-09-30: Python Meeting Duesseldorf ... 6 days to go ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

Antoine Pitrou

3:38 p.m.

On Wed, 24 Sep 2014 22:30:41 +0200 "M.-A. Lemburg" wrote:

...

On 24.09.2014 22:24, Barry Warsaw wrote:

...
On Sep 24, 2014, at 03:10 PM, Ryan Gonzalez wrote:

...
I would honestly prefer it to be merged into an already in-place module (maybe platform?) over it being added separately.

Why platform? That seems out of place.

Agreed. The typical approach is to put a renamed version into the stdlib, so that there's a possibility to continue using the package from PyPI.

...
If it were to be adopted into the stdlib, I think a top-level module name would be fine. psutil is great, but has anybody asked upstream if they even want to be part of the stdlib?

That'll have to happen first, indeed. The next question to answer is: Who will maintain the code in the stdlib ?

"Upstream" is Giampaolo Rodola, who is a core developer. Regards Antoine.

M.-A. Lemburg

3:44 p.m.

On 24.09.2014 22:38, Antoine Pitrou wrote:

...

On Wed, 24 Sep 2014 22:30:41 +0200 "M.-A. Lemburg" wrote:

...
On 24.09.2014 22:24, Barry Warsaw wrote:

...
On Sep 24, 2014, at 03:10 PM, Ryan Gonzalez wrote:

...
I would honestly prefer it to be merged into an already in-place module (maybe platform?) over it being added separately.

Why platform? That seems out of place.

Agreed. The typical approach is to put a renamed version into the stdlib, so that there's a possibility to continue using the package from PyPI.

...
If it were to be adopted into the stdlib, I think a top-level module name would be fine. psutil is great, but has anybody asked upstream if they even want to be part of the stdlib?

That'll have to happen first, indeed. The next question to answer is: Who will maintain the code in the stdlib ?

"Upstream" is Giampaolo Rodola, who is a core developer.

Cool. That makes the second question a lot easier to answer ;-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 24 2014)

...

...
...
Python Projects, Consulting and Support ... http://www.egenix.com/ mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

Stefano Borini

4:18 p.m.

On 9/24/14 10:38 PM, Antoine Pitrou wrote:

...

"Upstream" is Giampaolo Rodola, who is a core developer.

I already sent him an email about this. I am waiting for his answer. I would enjoy handling the library as a maintainer in the stdlib, but I have never done anything at this level, so I am not sure I am worthy/able to do it ;)

Giampaolo Rodola'

25 Sep 25 Sep

12:15 p.m.

On Wed, Sep 24, 2014 at 11:18 PM, Stefano Borini < stefano.borini@ferrara.linux.it> wrote:

...

On 9/24/14 10:38 PM, Antoine Pitrou wrote:

...
"Upstream" is Giampaolo Rodola, who is a core developer.

I already sent him an email about this. I am waiting for his answer. I would enjoy handling the library as a maintainer in the stdlib, but I have never done anything at this level, so I am not sure I am worthy/able to do it ;)

Hello all and thanks for the positive feedback. It's good to know psutil is so appreciated and it's the best payback for all the hard work. Personally I would be glad to offer psutil for inclusion into the stdlib but at the moment I'm crazily busy with the relocation (I moved to Berlin from Italy) and I wouldn't really have time to dedicate to such an expensive task (which would probably also deserve a PEP) and careful thinking. Also, I still want to work on a couple of new features first (NIC addresses and stats), address a couple of high-priority issues such as: https://github.com/giampaolo/psutil/issues/512 https://github.com/giampaolo/psutil/issues/496 https://github.com/giampaolo/psutil/issues/428 ...and be 100% sure that I'm happy with the API as it is right now (I already went through a major breakage once, see http://grodola.blogspot.com/2014/01/psutil-20-porting.html). In summary, I'd love to do this but not right now as I'm not quite ready yet. -- Giampaolo - http://grodola.blogspot.com

Victor Stinner

13 Oct 13 Oct

4:40 p.m.

2014-09-24 21:21 GMT+02:00 Stefano Borini :

...

I am wondering if it would be possible to include psutil (https://pypi.python.org/pypi/psutil ) in the standard library, and if not, what would be needed.

The problem is that Python only releases a new major version every 18 months (or something like that), so adding a single new function will take so much time. Supporting a new platform, architecture, major version of an OS, etc. may also need to wait a new Python major release. Operating systems are evolving quite fast. For example, there is the new architecture AArch64, containers are more and more popular, systemd can also be used to provide features similar to psutil and it supports containers (running systemd, it even supports recursive containers....). Network becomes more virtual with Software Defined Networks (SDN, NFV, etc.). The Linux kernel is also extended at each release. Example of "recent" addition: /proc/pid/fdinfo/ directory. Does psutil provide NUMA information? NUMA also becomes more information nowadays for performances. The psutil also still evolves. For example, I see that the version 2.1 released at April, 2014 adds "netstat-like functionalities". The version 2.0 was only released one month before (March, 2014). The API of psutil changed a lot between 1.x and 2.0. The version 2.0 was only released a few months ago. Is it enough to consider that the API is now stable enough (was enough tested)? Giampaolo (author of the module) doesn't look to be confident in its own API ;-) He wrote "I still want to work on a couple of new features first (...) and be 100% sure that I'm happy with the API as it is right now". Maybe I'm wrong and the psutil is stable and can be "frozen" in the standard library. Maybe it's possible to extract a subpart of the psutil to keep the most stable part? I'm not strongly opposed to the integration of the psutil module into the stdlib. Is it hard to install the psutil module today? I see wheel packages and .exe installers for Windows. They are Linux packages (Debian, Ubuntu, Fedora, ArchLinux, etc.) Victor

Andrew Barnert

5:49 p.m.

On Oct 13, 2014, at 14:40, Victor Stinner wrote:

...

2014-09-24 21:21 GMT+02:00 Stefano Borini :

...
I am wondering if it would be possible to include psutil (https://pypi.python.org/pypi/psutil ) in the standard library, and if not, what would be needed.

The problem is that Python only releases a new major version every 18 months (or something like that), so adding a single new function will take so much time.

Supporting a new platform, architecture, major version of an OS, etc. may also need to wait a new Python major release.

Operating systems are evolving quite fast. For example, there is the new architecture AArch64, containers are more and more popular, systemd can also be used to provide features similar to psutil and it supports containers (running systemd, it even supports recursive containers....). Network becomes more virtual with Software Defined Networks (SDN, NFV, etc.). The Linux kernel is also extended at each release. Example of "recent" addition: /proc/pid/fdinfo/ directory. Does psutil provide NUMA information? NUMA also becomes more information nowadays for performances.

The psutil also still evolves. For example, I see that the version 2.1 released at April, 2014 adds "netstat-like functionalities". The version 2.0 was only released one month before (March, 2014).

The API of psutil changed a lot between 1.x and 2.0. The version 2.0 was only released a few months ago. Is it enough to consider that the API is now stable enough (was enough tested)? Giampaolo (author of the module) doesn't look to be confident in its own API ;-) He wrote "I still want to work on a couple of new features first (...) and be 100% sure that I'm happy with the API as it is right now".

Maybe I'm wrong and the psutil is stable and can be "frozen" in the standard library. Maybe it's possible to extract a subpart of the psutil to keep the most stable part?

I'm not strongly opposed to the integration of the psutil module into the stdlib.

Is it hard to install the psutil module today? I see wheel packages and .exe installers for Windows. They are Linux packages (Debian, Ubuntu, Fedora, ArchLinux, etc.)

This might be a good idea. It seems to me that there are two common uses for psutil. Some people want just a little bit more than Python makes available, and they want it in a cross-platform (at least gnu vs. bsd vs. sysv, if not unix vs. windows) way. That doesn't change that much over time, and adding that much from psutil to the stdlib might make sense. Other people want all kinds of detailed process information, in a platform-specific way, without having to grub through low-level APIs. That changes frequently, and for those people, pip install psutil should probably remain the right answer. (Of course there are also people who use psutil to do the exact same things subprocess and os can already do better; for those people, the answer is to stop doing that...) I guess the real question is, what are the use cases for the "stable core"? Once we know that, we can identify whether it's stable enough and simple enough to extract.

Nick Coghlan

14 Oct 14 Oct

2:56 a.m.

On 14 October 2014 08:49, Andrew Barnert wrote:

...

On Oct 13, 2014, at 14:40, Victor Stinner wrote:

...
Maybe I'm wrong and the psutil is stable and can be "frozen" in the standard library. Maybe it's possible to extract a subpart of the psutil to keep the most stable part?

I'm not strongly opposed to the integration of the psutil module into the stdlib.

Is it hard to install the psutil module today? I see wheel packages and .exe installers for Windows. They are Linux packages (Debian, Ubuntu, Fedora, ArchLinux, etc.)

This might be a good idea.

It seems to me that there are two common uses for psutil.

Some people want just a little bit more than Python makes available, and they want it in a cross-platform (at least gnu vs. bsd vs. sysv, if not unix vs. windows) way. That doesn't change that much over time, and adding that much from psutil to the stdlib might make sense.

Other people want all kinds of detailed process information, in a platform-specific way, without having to grub through low-level APIs. That changes frequently, and for those people, pip install psutil should probably remain the right answer.

(Of course there are also people who use psutil to do the exact same things subprocess and os can already do better; for those people, the answer is to stop doing that...)

I guess the real question is, what are the use cases for the "stable core"? Once we know that, we can identify whether it's stable enough and simple enough to extract.

Right, "use case driven design" is the way to go. In particular, some of the key drivers for standard libraries additions in recent times have been: 1. Adding a module because we want to use it in the standard library (enum) 2. Adding a module because it smooths the learning curve for folks learning programming and other computing concepts (ipaddress, statistics) 3. Adding a module as part of a broader Python ecosystem interoperability effort (asyncio) 4. Adding a module because it is closely coupled to internal interpreter details (tracemalloc) A "minimal psutil" strikes me as something that could be useful for learning about computers by way of learning to program in Python - listing processes, users, network connections, memory usage, etc. Having that kind of basic capability in the standard library could be quite beneficial to a broad audience. The more sophisticated capabilities in psutil around resource limiting, and providing alternative interfaces to standard library functionality would likely make less sense in a standard library module (especially as these are the ones more likely to evolve over time). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Tshepang Lekhonkhobe

3:32 p.m.

On Tue, Oct 14, 2014 at 12:49 AM, Andrew Barnert wrote:

...

(Of course there are also people who use psutil to do the exact same things subprocess and os can already do better; for those people, the answer is to stop doing that...)

I was of the thought that psutil is better in everything it does compared to anything comparable in the stdlib. The only example I have in mind is its (far more featureful) Popen. What did I miss?

Jim Baker

4:15 p.m.

What are the implications of placing psutil in the stdlib for cross implementation support, eg Jython/IronPython? FWIW, we are planning to start work on Jython 3.5 at the PyCon sprints in Montreal. On Tue, Oct 14, 2014 at 2:32 PM, Tshepang Lekhonkhobe wrote:

...

...
(Of course there are also people who use psutil to do the exact same

On Tue, Oct 14, 2014 at 12:49 AM, Andrew Barnert wrote: things subprocess and os can already do better; for those people, the answer is to stop doing that...)

I was of the thought that psutil is better in everything it does compared to anything comparable in the stdlib. The only example I have in mind is its (far more featureful) Popen. What did I miss? _______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

-- - Jim jim.baker@{colorado.edu|python.org|rackspace.com|zyasoft.com} twitter.com/jimbaker github.com/jimbaker bitbucket.com/jimbaker

Giampaolo Rodola'

6 Nov 6 Nov

12:59 p.m.

On Tue, Oct 14, 2014 at 12:49 AM, Andrew Barnert < abarnert@yahoo.com.dmarc.invalid> wrote:

...

On Oct 13, 2014, at 14:40, Victor Stinner wrote:

...
2014-09-24 21:21 GMT+02:00 Stefano Borini < stefano.borini@ferrara.linux.it>:

...
I am wondering if it would be possible to include psutil (https://pypi.python.org/pypi/psutil ) in the standard library, and if not, what would be needed.

The problem is that Python only releases a new major version every 18 months (or something like that), so adding a single new function will take so much time.

Supporting a new platform, architecture, major version of an OS, etc. may also need to wait a new Python major release.

Operating systems are evolving quite fast. For example, there is the new architecture AArch64, containers are more and more popular, systemd can also be used to provide features similar to psutil and it supports containers (running systemd, it even supports recursive containers....). Network becomes more virtual with Software Defined Networks (SDN, NFV, etc.). The Linux kernel is also extended at each release. Example of "recent" addition: /proc/pid/fdinfo/ directory. Does psutil provide NUMA information? NUMA also becomes more information nowadays for performances.

The psutil also still evolves. For example, I see that the version 2.1 released at April, 2014 adds "netstat-like functionalities". The version 2.0 was only released one month before (March, 2014).

The API of psutil changed a lot between 1.x and 2.0. The version 2.0 was only released a few months ago. Is it enough to consider that the API is now stable enough (was enough tested)? Giampaolo (author of the module) doesn't look to be confident in its own API ;-) He wrote "I still want to work on a couple of new features first (...) and be 100% sure that I'm happy with the API as it is right now".

Maybe I'm wrong and the psutil is stable and can be "frozen" in the standard library. Maybe it's possible to extract a subpart of the psutil to keep the most stable part?

I'm not strongly opposed to the integration of the psutil module into the stdlib.

Is it hard to install the psutil module today? I see wheel packages and .exe installers for Windows. They are Linux packages (Debian, Ubuntu, Fedora, ArchLinux, etc.)

This might be a good idea.

It seems to me that there are two common uses for psutil.

Some people want just a little bit more than Python makes available, and they want it in a cross-platform (at least gnu vs. bsd vs. sysv, if not unix vs. windows) way. That doesn't change that much over time, and adding that much from psutil to the stdlib might make sense.

Other people want all kinds of detailed process information, in a platform-specific way, without having to grub through low-level APIs. That changes frequently, and for those people, pip install psutil should probably remain the right answer.+

(Of course there are also people who use psutil to do the exact same things subprocess and os can already do better; for those people, the answer is to stop doing that...)

I guess the real question is, what are the use cases for the "stable core"? Once we know that, we can identify whether it's stable enough and simple enough to extract.

I think this might make sense. Skimming through http://pythonhosted.org/psutil/ should give an idea of what's more "necessary" and what not. What's probably "less" necessary is: - psutil.disk_partitions() - psutil.net_connections() - psutil.users() - psutil.Process.open_files() (it turned out this is not reliable on Windows and FreeBSD) - psutil.Process.connections() - psutil.Process.memory_percent() ('cause it's a very simple utility method) APIs which are currently duplicated in Python stdlib: - psutil.disk_usage() (shutil.disk_usage, see http://bugs.python.org/issue12442) - psutil.Process.nice() (os.get/setpriority(), see: http://bugs.python.org/issue10784) - psutil.Process.cpu_affinity() (os.sched_get/setaffinity, see https://docs.python.org/3.5/library/os.html#os.sched_setaffinity) - psutil.cpu_count() (available as multiprocessing.cpu_count() but psutil can also distinguish between logical and physical cores) The rest is probably worth it for inclusion. As for my "doubts" regarding the stability of the API they basically comes down to one issue only: https://github.com/giampaolo/psutil/issues/428. In order to properly fix that issue I'll probably have to introduce a new "ZombieProcess" exception class other than "NoSuchProcess". That means users will have to change their code from this: for p in psutil.process_iter(): try: p.something() except psutil.NoSuchProcess: pass ...to this: for p in psutil.process_iter(): try: p.something() except (psutil.NoSuchProcess, psutil.ZombieProcess): pass I would consider the rest of the API quite stable. Please note that psutil currently supports: - Linux >= 2.6 (perhaps also 2.4) - Windows => XP - OSX => 10.4 (not sure about older versions) - FreeBSD => 9.0, perhaps also 8.0 - OpenSolaris (tested on OpenSolaris 10, not sure about other versions) With that said I expect that the moment psutil is included in the buildbot system a lot of currently undetected issues will be discovered as psutil has only been actively developed and tested on the platforms listed above (which are the ones I have access to).

...

Is it hard to install the psutil module today? I see wheel packages and .exe installers for Windows. They are Linux packages (Debian, Ubuntu, Fedora, ArchLinux, etc.)

I would say it is reasonably easy, especially now that I'm providing wheels for Windows. -- Giampaolo - http://grodola.blogspot.com

Stefano Borini

11 Nov 11 Nov

3:51 p.m.

New subject: hardware stdlib package (Was: Re: including psutil in the standard library?)

On 11/6/14 7:59 PM, Giampaolo Rodola' wrote:

...

- psutil.disk_partitions() - psutil.net_connections()

Actually, I was thinking that some of these should be part of a standard library "hardware" module, where you can inquire about the underlying hardware in a platform independent way. I checked around and I was unable to find a definitive package for this, which is a pity, because I consider it an important functionality that should definitely be a battery included. What I have found is, however, plenty of sparse code and packages trying to address this very need. To throw an idea, maybe it would be a good idea to aggregate all this code in a "hardware" package, stabilize it, and then include it as part of the standard library. I created a tentative github https://github.com/stefanoborini/hardware I'll scout around to see how to attack the problem, but maybe the ML has some important feedback I am definitely interested in. Stefano

3453

Age (days ago)

3501

Last active (days ago)

List overview

Download

15 comments

12 participants

participants (12)

Andrew Barnert
Antoine Pitrou
Barry Warsaw
Giampaolo Rodola'
Jim Baker
Jonas Wielicki
M.-A. Lemburg
Nick Coghlan
Ryan Gonzalez
Stefano Borini
Tshepang Lekhonkhobe
Victor Stinner

including psutil in the standard library?

Stefano Borini

Stefano Borini

Stefano Borini

tags

participants (12)