As different kinds of indirection and detection are added to easy_install, it should probably have a verbosity option. I think this would probably best work by using the logging module, so that Installer takes a logger object or name, and reports all messages to that logger, and in main() it directs the logger to stdout and sets the log level based on the verbosity indicated. Tools that work with easy_install outside of the command-line could probably also make good use of the logging output. I could add this, but the number of parallel patches is probably going to leave everyone (including me) confused. (I'm working on installation of packages without setup.py files right now) -- Ian Bicking / ianb@colorstudy.com / http://blog.ianbicking.org
At 09:02 PM 5/29/2005 -0500, Ian Bicking wrote:
As different kinds of indirection and detection are added to easy_install, it should probably have a verbosity option. I think this would probably best work by using the logging module, so that Installer takes a logger object or name, and reports all messages to that logger, and in main() it directs the logger to stdout and sets the log level based on the verbosity indicated. Tools that work with easy_install outside of the command-line could probably also make good use of the logging output.
Yeah, this is actually exactly what I plan to do in an unspecified future release, probably after I add filesystem virtualization/sandboxing, or maybe in conjunction with it. I hadn't really thought about the part where main() would set it up, but that part makes good sense too. I was planning to pass verbosity options through to the setup() script, and temporarily redirect its stdout/stderr to pass through the logger. The idea there is that GUI tools that wrap easy_install may want to display the progress messages, etc. in a progress dialog or status bar or something of that sort. This is especially true for the downloading part, which can take a while. By the way, the sandboxing feature is where I plan to replace a few key 'os' module functions and builtins with ones that "notice" if the setup script is trying to modify files outside the installer's temporary directory. Initially this will just be so I can analyze those scripts' behavior, but it might eventually grow into a facility to make them think they're modifying files in the real filesystem, but they would actually be getting redirected to the right temporary subdirectory for stuff to be added into the egg. I don't know if this feature is really achievable, but most all file access in Python boils down to either 'open()' or a call to an 'os' function imported from the platform-specific extension ('nt', 'mac', 'posix', etc.). Anyway, arguably people should "fix" their packages, but in practice some folks won't, so a working sandbox could be helpful. But I'll probably want the sandbox tools to do a lot of output at various levels of detail, hence why I'd probably use the logging package to do this.
I could add this, but the number of parallel patches is probably going to leave everyone (including me) confused. (I'm working on installation of packages without setup.py files right now)
Cool. You're really taking the ball and running with it, which is what I hoped would happen, as it's the specific reason I felt I had to release an initial, usable version of EasyInstall this very weekend. :)
Phillip J. Eby wrote:
Yeah, this is actually exactly what I plan to do in an unspecified future release, probably after I add filesystem virtualization/sandboxing, or maybe in conjunction with it. I hadn't really thought about the part where main() would set it up, but that part makes good sense too. I was planning to pass verbosity options through to the setup() script, and temporarily redirect its stdout/stderr to pass through the logger.
I still haven't seriously used the logging module, despite its widespread applicability to things I do. Go figure. But it seems like this would be a very good application of it, except for the actual exec of the setup file, which would have to use capturing of stdout and stderr. I'm guessing that redirection should be done to a sublogger, so that a caller can apply different filters and capturing to it.
The idea there is that GUI tools that wrap easy_install may want to display the progress messages, etc. in a progress dialog or status bar or something of that sort. This is especially true for the downloading part, which can take a while.
Progress wouldn't fit in very well to logging, though. I'm guessing there would have to be some notification object, with an API like: class INotifier: def describe_current_job(message): """Describe the currently-performed job, in the form of a short one-line text description""" def set_progress(progress, total, eta=None, units=None): """Indicate progress of the current job; progress and total are integers or floats. eta is the estimated time to finish, in seconds (if not explicitly given the INotifier instance may guess). units is an optional unit for progress and total (e.g., 'Mb', 'files')""" def reset_progress(): """Reset any progress and estimated times""" At least, that's my initial thought. Potentially you could have two progress indicators, one over-all progress, and another for the current job.
By the way, the sandboxing feature is where I plan to replace a few key 'os' module functions and builtins with ones that "notice" if the setup script is trying to modify files outside the installer's temporary directory. Initially this will just be so I can analyze those scripts' behavior, but it might eventually grow into a facility to make them think they're modifying files in the real filesystem, but they would actually be getting redirected to the right temporary subdirectory for stuff to be added into the egg. I don't know if this feature is really achievable, but most all file access in Python boils down to either 'open()' or a call to an 'os' function imported from the platform-specific extension ('nt', 'mac', 'posix', etc.).
Anyway, arguably people should "fix" their packages, but in practice some folks won't, so a working sandbox could be helpful. But I'll probably want the sandbox tools to do a lot of output at various levels of detail, hence why I'd probably use the logging package to do this.
Yikes, that sounds complicated. Though sandboxing would be kind of neat in general, e.g., for mock filesystems in unit tests. Are there particular packages you have in mind that need this?
I could add this, but the number of parallel patches is probably going to leave everyone (including me) confused. (I'm working on installation of packages without setup.py files right now)
Cool. You're really taking the ball and running with it, which is what I hoped would happen, as it's the specific reason I felt I had to release an initial, usable version of EasyInstall this very weekend. :)
Yes, I think The Time Is Right -- this sort of thing is really important for projects that integrate lots of libraries, and I think we're seeing that need from a few different directions. The code seems very simple too, which is a good sign that it's solving the right problem. -- Ian Bicking / ianb@colorstudy.com / http://blog.ianbicking.org
At 12:09 AM 5/30/2005 -0500, Ian Bicking wrote:
At least, that's my initial thought. Potentially you could have two progress indicators, one over-all progress, and another for the current job.
I was thinking that for download progress, I'd use whatever urllib already does. :) However for other things, you'd probably be looking at log messages to get some idea of the progress.
By the way, the sandboxing feature is where I plan to replace a few key 'os' module functions and builtins with ones that "notice" if the setup script is trying to modify files outside the installer's temporary directory. Initially this will just be so I can analyze those scripts' behavior, but it might eventually grow into a facility to make them think they're modifying files in the real filesystem, but they would actually be getting redirected to the right temporary subdirectory for stuff to be added into the egg. I don't know if this feature is really achievable, but most all file access in Python boils down to either 'open()' or a call to an 'os' function imported from the platform-specific extension ('nt', 'mac', 'posix', etc.). Anyway, arguably people should "fix" their packages, but in practice some folks won't, so a working sandbox could be helpful. But I'll probably want the sandbox tools to do a lot of output at various levels of detail, hence why I'd probably use the logging package to do this.
Yikes, that sounds complicated. Though sandboxing would be kind of neat in general, e.g., for mock filesystems in unit tests. Are there particular packages you have in mind that need this?
Yes. I've been finding that a surprising number manage to install stuff in e.g. site-packages "behind my back". Some are mentioned on the EasyInstall experiences page, but once I actually implemented a prototype sandbox, I discovered that some of the packages I thought were clean, were in fact installing stuff behind my back! Anyway, the virtualization is fairly uncomplicated if all you want to do is to note when a package is being naughty; I've already implemented that in my working copy. Redirecting the paths to a sane location is more complex; I may have to move the virtualizer into bdist_egg and apply it there. Essentially, what we want to do is that during the main setup script, we don't want to allow it to write anything outside the temp directory, and fail the script as an "unsafe package" (a nice bit of FUD to help shame package authors into cleaning up that sort of uncontrolled installation activity). For writes occurring during bdist_egg's invocation of install_data and install_lib, writes to site-packages should be redirected to bdist_egg's staging area, and all other writes outside the setup script directory should be a fatal "unsafe package" abort. Anyway, virtualization is surprisingly easy to implement; all of the Python-supplied filesystem APIs are grounded in the 'os' module, and the 'open'/'file' builtin. If you replace these two, you've pretty much got it made. Even os.path functions like 'isdir()' and such work by calling os.stat(), so replacing it in os means you're good to go. And, there are fewer than 30 os primitives in all to replace, with only four distinct input/output signatures. A little metaprogramming goes a long way here, such that AbstractSandbox is only about 100 lines, and a simple LoggingSandbox class is about 20 more lines. To use these sandboxes for unit tests would probably be a pain, though; I'm focused only on either disallowing actions entirely, or changing what directory they happen in, sort of like a Python-only pseudo-chroot(). A unit test system would probably want to virtualize all the operations, which means more operations to virtualize, and with more signatures.
participants (2)
-
Ian Bicking -
Phillip J. Eby