How suitable is Python to write system utilities?

[Alice Bevan–McGregor]

On 2011-01-06 01:35:58 -0800, Rohit Coder said:

> Is Python suitable to write low-level system utilities like Defrag, 
> Malware Removal Tools and Drivers?

Yes and no.

Python does include libraries (and has available third-party libraries) 
to interface with external low-level libraries of every kind, has 
Python-native third-party libraries to do things like examine ELF 
object files / executables, manipulate raw IP datagrams, etc, and it is 
possible to access glib (and other C libraries or even Windows DLLs) 
directly from within Python without creating wrapper interfaces.

While you -can- do all of these things, the question becomes do you 
really -want to-?

I've implemented server monitoring suites, FUSE filesystem drivers, and 
other strange low-level things in Python.  This doesn't mean I'm 
getting the best "bang for the buck" when it comes to performance or 
overhead; I'm trading these things for the ease of prototyping, 
debugging, and the ability to utilize other high-level interfaces.  My 
FUSE driver won't be as performant as it would have been had I written 
it in C.  My server monitoring suite consumes more RAM than an 
equivalent solution in C.  When it comes down to it, if you want it 
done efficiently, use C.  ;)

(As an aside, you -can- create hideous frankenstein monsters by using 
compiled CPython modules with glue code between the driver API, e.g. 
FUSE, and your CPython driver implementation; but in that case you're 
adding the overhead of Python for no gain whatsoever.)

For anything system critical, Python might not be the best choice.  
Malware removal tools are themselves the target of malware (e.g. virii 
attempting to disable scanners and removal tools), and utilizing Python 
adds (IMHO) too many points of failure.

Also, file fragmentation is a non-issue on all modern filesystems 
(ext3/4, reiser, ntfs, hfs+, etc.) as they perform live-system 
defragmentation to varying degrees.  I have never seen a production 
server of mine (utilizing reiserfs) go above 11% fragmentation 
(non-contiguous extant allocations), and even that resolved itself 
within a few hours of active use.

Note that non-contiguous extants are a distinct problem, reducing file 
read performance substantially, thus why filesystem drivers generally 
handle solving this problem by themselves.  The other forms of 
fragmentation (free space fragmentation and file scattering / 
related-file fragmentation) substantially less so.  Certain filesystems 
have features designed to avoid the latter (e.g. squashfs for ordering 
files on bootable CDs) and the former becomes more of an issue as you 
attempt to allocate extremely large contiguous files.  (Becoming worse 
as free space is exhausted as more and more files of greater size need 
to be shuffled around the disk platter in order to free up a contiguous 
run of extants.)

Hope this helps,

	- Alice.