On Tue, Oct 14, 2014 at 12:49 AM, Andrew Barnert <abarnert@yahoo.com.dmarc.invalid> wrote:
On Oct 13, 2014, at 14:40, Victor Stinner <victor.stinner@gmail.com> wrote:

> 2014-09-24 21:21 GMT+02:00 Stefano Borini <stefano.borini@ferrara.linux.it>:
>> I am wondering if it would be possible to include psutil
>> (https://pypi.python.org/pypi/psutil ) in the standard library, and if not,
>> what would be needed.
>
> The problem is that Python only releases a new major version every 18
> months (or something like that), so adding a single new function will
> take so much time.
>
> Supporting a new platform, architecture, major version of an OS, etc.
> may also need to wait a new Python major release.
>
> Operating systems are evolving quite fast. For example, there is the
> new architecture AArch64, containers are more and more popular,
> systemd can also be used to provide features similar to psutil and it
> supports containers (running systemd, it even supports recursive
> containers....). Network becomes more virtual with Software Defined
> Networks (SDN, NFV, etc.). The Linux kernel is also extended at each
> release. Example of "recent" addition: /proc/pid/fdinfo/ directory.
> Does psutil provide NUMA information? NUMA also becomes more
> information nowadays for performances.
>
> The psutil also still evolves. For example, I see that the version 2.1
> released at April, 2014 adds "netstat-like functionalities". The
> version 2.0 was only released one month before (March, 2014).
>
> The API of psutil changed a lot between 1.x and 2.0. The version 2.0
> was only released a few months ago. Is it enough to consider that the
> API is now stable enough (was enough tested)? Giampaolo (author of the
> module) doesn't look to be confident in its own API ;-) He wrote "I
> still want to work on a couple of new features first (...) and be 100%
> sure that I'm happy with the API as it is right now".
>
> Maybe I'm wrong and the psutil is stable and can be "frozen" in the
> standard library. Maybe it's possible to extract a subpart of the
> psutil to keep the most stable part?
>
> I'm not strongly opposed to the integration of the psutil module into
> the stdlib.
>
> Is it hard to install the psutil module today? I see wheel packages
> and .exe installers for Windows. They are Linux packages (Debian,
> Ubuntu, Fedora, ArchLinux, etc.)

This might be a good idea.

It seems to me that there are two common uses for psutil.

Some people want just a little bit more than Python makes available, and they want it in a cross-platform (at least gnu vs. bsd vs. sysv, if not unix vs. windows) way. That doesn't change that much over time, and adding that much from psutil to the stdlib might make sense.

Other people want all kinds of detailed process information, in a platform-specific way, without having to grub through low-level APIs. That changes frequently, and for those people, pip install psutil should probably remain the right answer.+

(Of course there are also people who use psutil to do the exact same things subprocess and os can already do better; for those people, the answer is to stop doing that...)

I guess the real question is, what are the use cases for the "stable core"? Once we know that, we can identify whether it's stable enough and simple enough to extract.

I think this might make sense. Skimming through http://pythonhosted.org/psutil/ should give an idea of what's more "necessary" and what not.
What's probably "less" necessary is:

- psutil.disk_partitions()
- psutil.net_connections()
- psutil.users()
- psutil.Process.open_files()  (it turned out this is not reliable on Windows and FreeBSD)
- psutil.Process.connections()
- psutil.Process.memory_percent() ('cause it's a very simple utility method)

APIs which are currently duplicated in Python stdlib:

- psutil.disk_usage()  (shutil.disk_usage, see http://bugs.python.org/issue12442)
- psutil.Process.nice()  (os.get/setpriority(), see: http://bugs.python.org/issue10784)
- psutil.Process.cpu_affinity() (os.sched_get/setaffinity, see https://docs.python.org/3.5/library/os.html#os.sched_setaffinity)
- psutil.cpu_count()  (available as multiprocessing.cpu_count() but psutil can also distinguish between logical and physical cores)

The rest is probably worth it for inclusion.
As for my "doubts" regarding the stability of the API they basically comes down to one issue only: https://github.com/giampaolo/psutil/issues/428.
In order to properly fix that issue I'll probably have to introduce a new "ZombieProcess" exception class other than "NoSuchProcess".
That means users will have to change their code from this:

for p in psutil.process_iter():
     try:
          p.something()
    except psutil.NoSuchProcess:
         pass

...to this:

for p in psutil.process_iter():
     try:
          p.something()
    except (psutil.NoSuchProcess, psutil.ZombieProcess):
         pass

I would consider the rest of the API quite stable.
Please note that psutil currently supports:
- Linux >= 2.6 (perhaps also 2.4)
- Windows => XP
- OSX => 10.4 (not sure about older versions)
- FreeBSD => 9.0, perhaps also 8.0
- OpenSolaris (tested on OpenSolaris 10, not sure about other versions)

With that said I expect that the moment psutil is included in the buildbot system a lot of currently undetected issues will be discovered as psutil has only been actively developed and tested on the platforms listed above (which are the ones I have access to).

> Is it hard to install the psutil module today? I see wheel packages
> and .exe installers for Windows. They are Linux packages (Debian,
> Ubuntu, Fedora, ArchLinux, etc.)

I would say it is reasonably easy, especially now that I'm providing wheels for Windows.

--
Giampaolo - http://grodola.blogspot.com