has wrote:

- Eliminate DU1's "Swiss Army" tendencies. Separate the build, install and register procedures for higher cohesion and lower coupling. This will make it much easier to refactor design of each in turn.

I'd like to second that. Example: compilation of extension modules. Scons is aiming at providing an abstraction layer for portable compilation. DU2 should at least allow to just delegate compilation of extension modules to scons. (and as I said previously, I think anything that doesn't allow to wrap traditional build systems based on 'make' and possibly the autotools is 'not good enough' in as a general solution).

- Every Python module should be distributed, managed and used as a single folder containing ALL resources relating to that module: sub-modules, extensions, documentation (bundled, generated, etc.), tests, examples, etc. (Note: this can be done without affecting backwards-compatibility, which is important.) Similar idea to OS X's package scheme, where all resources for [e.g.] an application are bundled in a single folder, but less formal (no need to hide package contents from user).

are you really talking about 'package' here when you say 'module' ? I don't think that mandating modules to be self contained is a good idea. Often modules only 'make sense' in the context of the package that contains them. Also, are you talking about how to distribute packages, or about the layout of the installed files ? I don't think DU2 should mandate any particular layout for the target installation. It may well suggest layout of the files inside the (not yet installed) package.

- Question: is there any reason why modules should not be installable via simple drag-n-drop (GUI) or mv (CLI)? A standard policy of "the package IS the module" (see above) would allow a good chunk of both existing and proposed DU "features" to be gotten rid of completely without any loss of "functionality", greatly simplifying both build and install procedures.

Again, I don't think it is DU2's role to impose anything concerning the target layout. This is often platform dependent anyways.

--Replace current system where user must explicitly state what they want included with one where user need only state what they want excluded.

That depends on how much control users want over the process. I believe both are equally valid, and should be supported (similar in spirit to the MANIFEST.in syntax 'include' and 'exclude')

-- In particular, removing most DU involvment from build procedures would allow developers to use their own development/build systems much more easily.

yes !! Though that's more easily said than done: a minimum of collaboration between the two is required, at least the adherence to some conventions.

- Installation and compilation should be separate procedures. Python already compiles .py files to .pyc on demand; is there any reason why .c/.so files couldn't be treated the same? Have a standard 'src' folder containing source files, and have Python's module mechanism look in/for that as part of its search operation when looking for a missing module; c.f. Python's automatic rebuilding of .pyc files from .py files when former isn't found. (Q. How would this folder's contents need to be represented to Python?)

As a starting point, the whole 'build_ext' mechanism should be re-evaluated. The current 'Extension' mechanism is by far not abstract enough. Either the build_ext or the Extension class should be made polymorphic to wrap any external build system that could be used (make, scons, jam, ...)

- What else may setup.py scripts do apart from install modules (2) and build extensions (3)?

* building documentation (that, too, is highly domain specific. From Latex over Docbook to doxygen...) * running unit tests

- Remove metadata from setup.py and modules.

I don't quite agree in general. What metadata are we talking about anyways ? There's metadata that is to be provided to the packager backends, i.e. a package description of some sort. Some of these can be generated automatically (such as MANIFEST.in -> MANIFEST, build / host platform, etc.), others have to be explicitely provided (maintainer address, package description). Having 'all metadata' lumped together brings us back to the 'swiss army knife' syndrome.

- Improve version control. Junk current "operators" scheme (=, <, >, >=, <=) as both unnecessarily complex and inadequate (i.e. stating module X requires module Y (>= 1.0) is useless in practice as it's impossible to predict _future_ compatibility). Metadata should support 'Backwards Compatibility' (optional) value indicating earliest version of the module that current version is backwards-compatible with. Dependencies list should declare name and version of each required package (specifically, the version used as package was developed and released).

Good idea, though this issue highly depends on the packager backend used.

- Make it easier to have multiple installed versions of a module.

That, too, isn't really an DU2 issue, or is it ?

- Reject PEP 262 (installed packages database). Complex, fragile, duplication of information, single point of failure reminiscent of Windows Registry. Exploit the filesystem instead - any info a separate db system would provide should already be available from each module's metadata.

I don't quite agree. I couldn't live without rpm these days.

On Fri, May 14, 2004 at 03:16:31PM +0100, has wrote:

Recommend:

- Before adding new features/complexity, refactor current _design_ to simplify it as much as possible. Philosophy here is much more hands-off than DU1; less is more; power and flexibility through simplicity: make others (filesystem, generic tools, etc.) do as much of the work as possible; don't create dependencies.

I'd translate this to "an implementation that effectively manages package metadata". If the metadata is consistently represented, translating to differing build, install and packaging modules becomes simpler and more straight forward. The idea of different setup.cfg sections for different bdist commands needs to die. Every bdist command should be able to process effectively off the same set of metadata.

-- e.g. c.f. Typical OS X application installation procedure (mount disk image and copy single application package to Applications folder; no special tools/actions required) versus typical Windows installation procedure (run InstallShield to put lots of bits into various locations, update Registry, etc.) or typical Unix installation procedure (build everything from source, then move into location). Avoiding overreliance on rigid semi-complex procedures will allow DU2 to scale down very well and provide more flexibility in how it scales up.

Distutils cannot dictate to any platform how packages are to be installed. It needs to go the other way. Distutils needs to meet the platforms' requirements for packaging. I know perl-mongers (and some pythoneers as well) hold up CPAN as a shining example of how things should work, but I shoot any of my admins that use plain CPAN for installation. It doesn't register with the platform's native software manager and, therefore, completely destroys any attempt to manage system inventory information. No big deal on a system or two, maybe, but at 3:00 in the morning, with an hour outage to move an application from one system to another, suddenly discovering that you didn't prepare the destination host with 37 required CPAN (or Distutils) modules because there was no record of them ever having been installed really, really, sucks. IMHO, if Distutils leverages native packager bdist commands for as many platforms as possible, it will kick CPAN's behind in the enterprise enviroment. I, for one, do not want to visit hundreds of machines to "python setup.py install", even if it downloads it for me. I don't need or want a full development environment on production machines. I _need_ a package once (per platform), install -> hundreds of boxes AND I need those hundreds of boxes to be able to tell me exactly what is installed on them for disaster recovery and application portability.

- Eliminate DU1's "Swiss Army" tendencies. Separate the build, install and register procedures for higher cohesion and lower coupling. This will make it much easier to refactor design of each in turn.

I think DU1 should be viewed in a "lessons learned" context. What needs to be done is to document the internal APIs, extrapolate what those APIs should really be now that there's some real-life experience available, then refactor the APIs to be more representative of actual use.

- Every Python module should be distributed, managed and used as a single folder containing ALL resources relating to that module: sub-modules, extensions, documentation (bundled, generated, etc.), tests, examples, etc. (Note: this can be done without affecting backwards-compatibility, which is important.) Similar idea to OS X's package scheme, where all resources for [e.g.] an application are bundled in a single folder, but less formal (no need to hide package contents from user).

Maybe for modules, but this is patently false for applications. Applications have configuration files and data that may be different for different instances or users of the application on the same box. Dumping all that into site-packages is impractical for both usability principles (separation of programs and data) and space considerations (how do I size /usr if /usr/lib/pythonx.x/site-packages/package is going to contain variable user data?). This begs the questions "Should Distutils handle modules and applications the same?" and "Does it make more sense to package modules one way and applications another?" I don't know.

- Question: is there any reason why modules should not be installable via simple drag-n-drop (GUI) or mv (CLI)? A standard policy of "the package IS the module" (see above) would allow a good chunk of both existing and proposed DU "features" to be gotten rid of completely without any loss of "functionality", greatly simplifying both build and install procedures.

Yes, because that's not how all platforms manage their software inventory. Servers may not (most, in fact, _should_ not) have GUI capable resources installed on them. "mv" (or "cp" or any other command) is not a software management tool. I use Distutils because it helps be manage my software inventory. If all I wanted was to be able to make something run on a machine, I'd just build a tarball and drop it in where I needed it. The problem is that once I've done that, I don't have any record of the fact that I needed it on any particular machine and I have lost the ability to easily replicate a machine's environment on another machine, be it for failover or upgrades or for a test/QA environment.

--Replace current system where user must explicitly state what they want included with one where user need only state what they want excluded. Simpler and less error-prone; fits better with user expectations (meeting the most common requirement should require least amount of work, ideally none). Manifest system would no longer be needed (good riddance). Most distributions could be created simply by zipping/tar.gzipping the module folder and all its contents, minus any .pyc and [for source-only extension distributions] .so files.

This I agree with. I recently did a quick setup.py on something (can't remember what) and key .xml files where ignored. If it's in the source tree, it's probably needed for something. If it's not, then I should be able to exclude it, but I'd rather be sure everything necessary "gets there" by default. There are still a lot of python modules available that don't have setup.py included. It would be far easier for a 3rd party to submit Distutils enabling setup.py "patches" if they didn't have to do a complete code analysis to determine what's necessary and what's not.

-- In particular, removing most DU involvment from build procedures would allow developers to use their own development/build systems much more easily.

I may be on the fence on this one. Since mastering simple setup.py formats, I'm not sure I remember how to manually build a C extension anymore ;) Fro my perspective, DU1 has got what it takes to support a newbie's foray into building extensions and I'd hate to see that get lost. A cleaner API and documentation, however, would support better integration of Distutils _into_ alternative build systems.

- Installation and compilation should be separate procedures. Python already compiles .py files to .pyc on demand; is there any reason why .c/.so files couldn't be treated the same? Have a standard 'src' folder containing source files, and have Python's module mechanism look in/for that as part of its search operation when looking for a missing module; c.f. Python's automatic rebuilding of .pyc files from .py files when former isn't found. (Q. How would this folder's contents need to be represented to Python?)

They already are. "python setup.py build" and "python setup.py install" can be done on separate machines (of the same architecture). My argument continues to be that "python setup.py install" is not a software mangement tool. "python setup.py bdist_whatever" produces the installation packages I need to support the "whatever" architecture with a build once, install anywhere and -- most importantly -- effectively manage the software configuration of N hosts of "whatever" architecture. bdist_* is the true value of Distutils. Without the bdist commands, Distutils does nothing more effectively than `./configure && make && make install`.

- What else may setup.py scripts do apart from install modules (2) and build extensions (3)?

Install and optionally upgrade configuration files. Build native packages including preinstall, preremove, postinstall, postremove and reconfiguration scripts. Dynamically relocate packages to the target host's python installation directory. These are all features of the bdist_command capability that I think you're missing.

-- Most packages should not require a setup.py script to install. Users can, of course, employ their own generic shell script/executable to [e.g.] unzip downloaded packages and mv them to their site-packages folder.

They can do that today. Where's the "value added?" Maybe setup.py could be replaced by something that scans a directory for __init__.py files and make some deductions, and maybe that's a good thing for simple packages. Personally, I'd like to see DU2 go further in supporting the simple production of multiple binary packages from a single source tree. Marc's egenix packages come to mind as a good example of the type of thing that it would be really nice to NOT have to sub-class a bunch of Distutils classes in a mega-setup.py script in order to produce a set of related, but not necessarily interdependent, packages.

-- Extensions distributed as source will presumably require some kind of setup script in 'src' folder. Would this need to be a dedicated Python script or would something like a standard makefile be sufficient?

My belief is that Distutils role is to reduce the need to distribute extenstions as source. One, good, Distutils configuration by a Python for Windows developer should simply repackage (absent win32 dependencies) for any supported platform simply by changing "python setup.py wininst" to "python setup.py myplatform". This drives back to my focus on getting the meta-data right and in a consistent format regardless of the original development platform or initially perceived target.

-- Build operations should be handled by separate dedicated scripts when necessary. Most packages should only require a generic shell script/executable to zip up package folder and its entire contents (minus .pyc and, optionally, .so files).

Again, this provides no software configuration management. What, then, do I gain by using Distutils at all?

- Remove metadata from setup.py and modules. All metadata should appear in a single location: meta.txt file included in every package folder. Use a single metadata scheme in simple structured nested machine-readable plaintext format (modified Trove); example:

Isn't that what setup.cfg is? I agree that there shouldn't be redundancy between what can go in setup.py and what can go in setup.cfg.

------------------------------------------------------------------ Name roundup [snip]

There's a whole slew of meta-data required for native packagers that's required for software configuration management. Most (iirc) are addressed in various PEP's, although it would be good to revisit the fields and establish a matrix between Distutils meta-data fields and various native package manager fields to make sure all posibilities are covered.

- Improve version control. Junk current "operators" scheme (=, <, >, >=, <=) as both unnecessarily complex and inadequate (i.e. stating module X requires module Y (>= 1.0) is useless in practice as it's impossible to predict _future_ compatibility). Metadata should support 'Backwards Compatibility' (optional) value indicating earliest version of the module that current version is backwards-compatible with. Dependencies list should declare name and version of each required package (specifically, the version used as package was developed and released). Version control system can then use both values to determine compatibility. Example: if module X is at v1.0 and is backwards-compatible to v0.5, then if module Y lists module X v0.8 as a dependency then X 1.0 will be deemed acceptable, whereas if module Z lists X 0.4.5 as a dependency then X 1.0 will be deemed unacceptable and system should start looking for an older version of X.

This is more appropriately addressed in the context of what/how native package managers support version control. A point implicit in this discussion, however, is that a package name registry is required. Unless package names are registered with some authority, you can have multiple packages of the same name which shoves a huge bone done the throat of dependency resolution.

- Make it easier to have multiple installed versions of a module. Ideally this would require including both name and version in each module name so that multiple modules may coexist in same site-packages folder. Note that this naming scheme would require alterations to Python's module import mechanism and would not be directly compatible with older Python versions (users could still use modules with older Pythons, but would need to strip version from module name when installing).

This is conditional on support of the platform's native package manager. Some support multiple installations and some do not. Most can be dealt with in any case with package named fudging and intelligent install scripts. I don't see a need for multiple instances in the same site-packages, however. Futzing with the import mechanism would be fixing something that ain't broke. Installing to an alternate path and optionally having postinstall scripts update site.py or requiring the user modify PYTHONPATH is adequate. I do this on HP-UX, which supports multiple installs, in different locations, of the same binary package, which allows users to install into their own target python library. When an alternate path is selected, the installer spits out all the necessary steps required to make use of the alternate path.

- Reject PEP 262 (installed packages database). Complex, fragile, duplication of information, single point of failure reminiscent of Windows Registry. Exploit the filesystem instead - any info a separate db system would provide should already be available from each module's metadata.

I agree with rejecting 262 as well, but not in favor of the filesystem but in favor of the native platform tools via bdist support. Solaris people use pkgtools for everything. RH and friends use RPM. HP people use SD-UX. Debianites use dpkg. etc. etc. etc.... God help those of us supporting multiple platforms. In each case, absent [expletive deleted] commercial package installs, all software and configuration management is consistent and, more importantly, effective. Anything on top of that; CPAN, Distutils, PEP 262, rogue admins with tarballs; _anything_ at all and people who have to deal with anything over a handfull of machines WILL eventually get caught with their pants down. If Distutils does not support simple, native package manager integration, then it ceases to be a solution and becomes just one more problem. A successfull implementation that creates native packages gets immediate support from apt, yum, yast, urpmi, pkg-get, swinstall and anything else, now and in the future. /me steps off the soapbox mwa -- Mark W. Alexander slash@dotnetslash.net The contents of this message authored by Mark W. Alexander are released under the Creative Commons Attribution-NonCommercial license. Copyright of quoted materials are retained by the original author(s). http://creativecommons.org/licenses/by-nc/1.0/

[Deep breath, everyone; it's gonna get longer before it gets shorter...;] Stefan Seefeld wrote:

Example: compilation of extension modules.

Scons is aiming at providing an abstraction layer for portable compilation. DU2 should at least allow to just delegate compilation of extension modules to scons. (and as I said previously, I think anything that doesn't allow to wrap traditional build systems based on 'make' and possibly the autotools is 'not good enough' in as a general solution).

I'm far from expert on build systems (interpreted language weenie), but do think it makes sense for DU to hand off compilation duties to a third-party as quickly as it can. That third-party might be a separate Python-based system, or makefile, or anything else; however DU shouldn't need to know this. Thoughts on how one would separate extension compilation from the rest of the installation procedure... Let's say we had a standard 'src' folder to contain everything needed to produce a module's .so file(s), and we treat that folder basically as a self-contained entity. How would we instruct it to compile and deliver those .so files? The party requesting the compile operation should not need to know anything about the compilating system used. Presumably the easiest way to decouple the two is to have a standard 'compile.py' within the 'src' folder that is executed whenever somebody wants .so files created. Whatever code that compile.py file then executes is its own business, and if it needs any information from the OS/Python installation then it's up to it to request that information itself; ideally through existing Python APIs if possible, or through a specific DU API if not. Once that's done, it should be easy for developers to select their own build system from all those available to them. The Python-based build system that's currently incorporated into DU could, of course, be spun off as a peer to make, etc. - giving developers one more option to choose from without forcing it upon them. BTW, once .so compilation is decoupled from installation, it should be possible/practical/easy? to defer .so compilation to import time (as is currently done for .pyc files).

...

- Every Python module should be distributed, managed and used as a single folder containing ALL resources relating to that module: sub-modules, extensions, documentation (bundled, generated, etc.), tests, examples, etc. (Note: this can be done without affecting backwards-compatibility, which is important.) Similar idea to OS X's package scheme, where all resources for [e.g.] an application are bundled in a single folder, but less formal (no need to hide package contents from user).

are you really talking about 'package' here when you say 'module' ? I don't think that mandating modules to be self contained is a good idea. Often modules only 'make sense' in the context of the package that contains them. Also, are you talking about how to distribute packages, or about the layout of the installed files ? I don't think DU2 should mandate any particular layout for the target installation. It may well suggest layout of the files inside the (not yet installed) package.

(Like I say, rough notes; please keep pointing out where I'm not making sense.:) Basically, what I'm proposing is that module developers stop distributing 'naked' Python modules and use package format only (even when there's only a single .py file involved). We then take all the other stuff that's traditionally been bundled alongside the module/package - documentation, unit tests, examples, etc. - and put those into the package folder too. The term 'package' would basically become redundant; you could just describe everything as 'modules' and 'sub-modules'. It's largely a philosophical shift from treating .py and .so files as separate from source files, documentation, unit tests, examples, etc. to treating _all_ of them equally: each being an integral component of the module/package as a whole. It won't require any modifications to Python itself, since Python's import mechanism already supports the package format. For module developers, it's really just a logistical shift from being able to distribute 'bare' modules to always using package format. Module developers should be happy with this, given that it's much more accommodating towards documentation, unit tests, examples, etc. Stuff that they already need to put somewhere, and where better than as part of the module/package itself? And users should benefit too, as they'll always know where to look for documentation, etc. DU will benefit too in that the distributions will become much simpler to create: in most cases the only thing the developer will have to do is zip the package folder before uploading it, something that won't even require DU to do. (That's what I'm hoping for, anyway. In practice there might be some reason that I'm unaware of why certain platforms would require all that extra shuffle that DU currently does when installing packages - creating folders, copying files, etc. I'm not a cross-platform expert. But I'd be kinda surprised if that were the case.) [Sidenote: in an ideal world, a Python end-user should _never_ need to know whether FooLib exists in bare module or package form; the transition from operating in a file-based namespace to class-/object-based namespace would be seamless. Python's import statement is a bit flawed here; e.g. import foo.bar can be used when bar is a module/package within package foo, but not when it's a attribute in module foo.]

...

- Question: is there any reason why modules should not be installable via simple drag-n-drop (GUI) or mv (CLI)? A standard policy of "the package IS the module" (see above) would allow a good chunk of both existing and proposed DU "features" to be gotten rid of completely without any loss of "functionality", greatly simplifying both build and install procedures.

Again, I don't think it is DU2's role to impose anything concerning the target layout. This is often platform dependent anyways.

Not quite sure if we're talking on same wavelength here. Let me try to clarify my previous point first, then maybe you can explain yours to me (feel free to phrase it in terms even an idiot like me can understand; I won't be offended;). I'm talking of how a module/package gets put in a suitable Python directory (e.g. site-packages), which I'm assuming (unless proven otherwise) only requires that one knows which directory to put it in and moving the module/package to it. I'm also assuming that DU should not need to rearrange the contents of that package folder when installing it (except perhaps in special cases where it must install one of several platform-specific versions of a file, say; but that'll be the exception rather than the rule, and packages that don't require such special handling shouldn't need to go through the same in-depth procedures to install). I can't immediately see anything that DU adds to this process of duplicating a package folder from A to B, apart from filling my Terminal window with lots of technical-looking stuff about how it's creating new directories in site-packages and copying files over to them. Which looks impressive, but I'm not convinced is really necessary given a single 'mv' command can shift the package directory and all its contents over just fine from what I can tell. And if 95-100% of modules can be installed with just a simple mv, then let's make that the default procedure for installing modules and squeeze DU out of that part of the process too.

--Replace current system where user must explicitly state what they want included with one where user need only state what they want excluded.

That depends on how much control users want over the process. I believe both are equally valid, and should be supported (similar in spirit to the MANIFEST.in syntax 'include' and 'exclude')

<ASIDE> Quick bit of background info so you know where I'm coming from... I'm also big on the "There should be [preferably] only one way to do it" philosophy (one of the things that attracts me to Python). This is as much out of necessity as anything, mind: I'm absolutely awful at absorbing and retaining technical information, especially compared to 'real' programmers who seem to soak up knowledge like a sponge. e.g. I admire Perl for its "hell, let's totally go for broke and put in _everything_ we can possibly think of" approach and am glad there's somebody out there doing it cos then other languages can look at Perl to see what's worked and what hasn't and steal the best stuff for themselves. But it's not a language I can really use; my brain capacity is far too limited to accommodate more than a fraction of Perl's vast featureset and rules, so I much prefer to stick to tighter languages like Python where I can work at a decent clip without having to look up some 1000-page reference book at every other line. Thus I tend to set the bar for feature inclusion pretty high; probably much higher than most other programmers who can happily cope with a bit of API flab without any problem. Don't take my feature-flaying tendencies as a religious thing. It's more a matter of simple survival: I can't keep up with y'all otherwise. ;) </ASIDE> The problem I see is that manifests seem to be involved whether you need/want them or not. If [as I'm assuming] the majority of distributions are trivial to assemble, then manifests should be the exception, not the rule. I dunno how other folks work, but in my Home folder I have a PythonDev folder containing folders for each of my module projects - FooDev, BarDev, etc. Within each of these I have a folder named Distro, which contains all the files and folders that'll go into my distribution. For me, manifests are nothing but a menace: this folder setup already makes clear what I want put into the distribution, and I can't see why I should have to explain it twice to the stupid machine. There's been several occasions where an error or omission in a manifest file has gone unnoticed until I've received an email from a user to say that the package they downloaded is missing some parts (embarrassing). Right now I manually unzip and check distributions before uploading, but this is kinda crazy; I shouldn't have to worry that DU might have screwed up a build, seeing as one of the reasons for automating the process should be to avoid making such mistakes. Thus my conclusion: explicit inclusion is inherently unsafe; a single mistake or forgetting to update the manifest file to keep it in sync with changes to the package can easily result in a broken distribution. A much more sensible default is to include everything by default, and leave it to the developer to exclude anything they don't want included. The worst accident likely to occur here with any regularity is that you forget to strip out a few .pyc files resulting in a distribution that's a few KB bigger than it really needs to be. Plus it adheres to the philosophy that the most common case should require the least amount of work: in this case, the majority of modules won't ever require a manifest file and can safely skip it. [BTW, will check out the include/exclude feature which I wasn't previously aware of. Though my argument would be that I shouldn't need to know about such 'advanced' features just to produce a simple, reliable distribution: the process should be as simple as falling off a log to begin with.] ... We can take this manifest issue quite a bit further, btw. Another big frustration with the things is they're quite brain-dead. All I want to say is "Package everything in Folder X for distribution except for .pyc and .so files". Thus a more pragmatic approach might be to do away with dumb manifest files completely, and leave the developer to optionally supply a 'build.py' script that will be automatically executed as part of the build process.

...

-- In particular, removing most DU involvment from build procedures would allow developers to use their own development/build systems much more easily.

yes !! Though that's more easily said than done: a minimum of collaboration between the two is required, at least the adherence to some conventions.

Of course (see earlier comments). Just how many... no, _few_ conventions would be needed?

...

- Installation and compilation should be separate procedures.

As a starting point, the whole 'build_ext' mechanism should be re-evaluated. The current 'Extension' mechanism is by far not abstract enough. Either the build_ext or the Extension class should be made polymorphic to wrap any external build system that could be used (make, scons, jam, ...)

Or invert and decouple the process to put the [e.g.] 'src/compile.py' script in control.In this case, I think we could greatly simplify the extension building process if DU can say to the 'src' folder: "Build me some .so files", then stand back and let it get on with it (while being happy to lend any support if/when it's asked for). [i.e. It's a mental trick I often try when trying to resolve an API design: seeing if I can switch from a complex 'push' process to a simpler 'pull' process (or from a complex 'pull' process to a simpler 'push' process). It can make quite a difference.]

...

- What else may setup.py scripts do apart from install modules (2) and build extensions (3)?

* building documentation (that, too, is highly domain specific. From Latex over Docbook to doxygen...)

Yup. So let's say we have a standard 'docs' folder within a package that may optionally contain a 'format.py' script that will be called as necessary.

* running unit tests

Have a standard 'tests' folder containing an optional 'test.py' script. (Hey, think I see a pattern evolving here...)

...

- Remove metadata from setup.py and modules.

I don't quite agree in general. What metadata are we talking about anyways ? There's metadata that is to be provided to the packager backends, i.e. a package description of some sort. Some of these can be generated automatically (such as MANIFEST.in -> MANIFEST, build / host platform, etc.), others have to be explicitely provided (maintainer address, package description).

I mean user-defined metadata (I'll assume those that generate metadata automatically for their own consumption can be left to handle that as best suits themselves): 1. A module may contain various bits of user-defined metadata, e.g. __version__, __author__. This info is almost certainly recorded elsewhere, so [afaik] shouldn't need to be duplicated here. 2. The setup.py script also contains module name, version, author, etc... potentially quite a lot of metadata, in fact. All mooshed together with code for building and installing packages. We should move this data out of there into a separate, dedicated metadata file that's included in each package.

Having 'all metadata' lumped together brings us back to the 'swiss army knife' syndrome.

Well, swiss armyness is always a concern. If it's really a problem here, we'd just need to have more than one metadata file. But I don't think it'll come to that. Also, one great advantage of pulling metadata out of module and setup.py files is that it'll make it much easier for other clients to access it. Right now it's kinda locked away: the only folks who know how to access and use it are Python (module metadata) and DU (setup.py metadata).

- Improve version control. Junk current "operators" scheme (=, <, >, >=, <=) as both unnecessarily complex and inadequate (i.e. stating module X requires module Y (>= 1.0) is useless in practice as it's impossible to predict _future_ compatibility). Metadata should support 'Backwards Compatibility' (optional) value indicating earliest version of the module that current version is backwards-compatible with. Dependencies list should declare name and version of each required package (specifically, the version used as package was developed and released).

Good idea, though this issue highly depends on the packager backend used.

Could you cite some examples to help me understand the issues involved?

- Make it easier to have multiple installed versions of a module.

That, too, isn't really an DU2 issue, or is it ?

Not really; more a general packaging and Python import issue. But I included it here as I think packaging issues have a big impact on DU policy.

...

- Reject PEP 262 (installed packages database). Complex, fragile, duplication of information, single point of failure reminiscent of Windows Registry. Exploit the filesystem instead - any info a separate db system would provide should already be available from each module's metadata.

I don't quite agree. I couldn't live without rpm these days.

Well, it's not to say that users can't build their own databases listing all their installed gunk if they want to. Ensuring user freedom in such areas is crucial. Perhaps it would be clearer to say that the intention is sound (make information on installed modules easy to retrieve; something I'm all for), but the way 262 proposes to do it is not. In fact, one of my main objections to 262 is that it could well restrict user freedom: by creating lots of dependencies and synchronisation issues, users could find themselves locked into using a single 'official' Package Manager because it's the only one smart enough to deal with all these complexities. Users who venture into their site-packages folder by any other means will quickly find themselves being punished the PackMan Police for unlawful infractions. This should be one of the benefits that comes from decoupling module metadata from implementation as I've suggested above. There'll be no need for a central authority (262's DB) to maintain metadata, because each module already contains and looks after its own. And because there's only one metadata instance in existence for each module, there's no dependency/synchronisation issues to worry about. You can still provide users with exactly the same API that the 262 DB would have done for accessing this info, of course, so you still get all the functionality 262 would have provided, but without any of the headaches. Funnily enough though, one of the possible DB implementations floated for 262 is to put the metadata for each module into a separate file on disk. So perhaps I should say that 262's idea of maintaining a _separate_ database simply isn't necessary: all the info it would have provided can already be retrieved from files in the filesystem; the only difference is that each file is bundled in package. The module/file system _is_ the database, if you like. (After all, what's a filesystem but a big ol' object database by any other name?;) Thanks, has p.s. If you're interested, you can see a module system I designed a couple years back at applemods.sourceforge.net. It actually uses a version of the "module = package = distribution with all batteries included" concept I'm floating here. (Which I think was itself influenced by Python's package system.) p.p.s. Anything folk can do to help me understand the issues involved in cross-platform and extension compilation lest I spout off too much about things I know not will be much appreciated, ta. :) -- http://freespace.virgin.net/hamish.sanderson/

On 15-mei-04, at 0:23, Lars Immisch wrote:

From Bob:

...

Not all operating systems have a usable package management system (Win32, Mac OS X, probably others).

What's wrong with Installer.app and/or PackageMaker?

Both are installers, not package management systems. There is no public interface for listing which packages are installed and uninstalling packages, let alone dependency management. Ronald -- X|support bv http://www.xsupport.nl/ T: +31 610271479 F: +31 204416173

On 14-mei-04, at 22:25, has wrote:

[Deep breath, everyone; it's gonna get longer before it gets shorter...;]

Stefan Seefeld wrote:

...

Example: compilation of extension modules.

Scons is aiming at providing an abstraction layer for portable compilation. DU2 should at least allow to just delegate compilation of extension modules to scons. (and as I said previously, I think anything that doesn't allow to wrap traditional build systems based on 'make' and possibly the autotools is 'not good enough' in as a general solution).

I'm far from expert on build systems (interpreted language weenie), but do think it makes sense for DU to hand off compilation duties to a third-party as quickly as it can. That third-party might be a separate Python-based system, or makefile, or anything else; however DU shouldn't need to know this.

DU should know about it, it should be easier to itegratie DU with other build systems but DU should still know about building extensions. For this reason I also don't like your idea of splitting of various tasks into seperate python scripts, that doesn't really solve anything, e.g. why is writing a script named test.py *inherently* easier than writing a test action for distutils. What's needed is an easier way to write and adapt DU actions, and to get tool-builders to include DU-actions/subactions with their tools. Pyrex already includes an build_ext action that will convert .pyx file to C, SWIG could do the same, as could docutils, unittest frameworks, ... BTW. what I like about distutils is that it provides an easy to recognize standard interface for installing python libraries. When I download a new python library and it includes a setup.py script I'm a happy camper, because I can install using 'python setup.py install' (*). When there's no setup.py script I've to read installation instructions, and often have to do various manual steps before I can install. (*) In theory of course, in practice C extensions often need help to tell them where libraries are and which features are available in those libraries. I guess DU needs autoconf-like functionality for detecting features of (C-) libraries.

...

...

- Question: is there any reason why modules should not be installable via simple drag-n-drop (GUI) or mv (CLI)? A standard policy of "the package IS the module" (see above) would allow a good chunk of both existing and proposed DU "features" to be gotten rid of completely without any loss of "functionality", greatly simplifying both build and install procedures.

Again, I don't think it is DU2's role to impose anything concerning the target layout. This is often platform dependent anyways.

<nod> I do like the idea of having a standard location for documetation and examples though. A prefered source layout would also help (e.g. if you don't specify anything at all distutils could assume everything inside 'Lib' is a python module, everything in 'Doc' is documentation and the unittests are in 'Testsuite').

Not quite sure if we're talking on same wavelength here. Let me try to clarify my previous point first, then maybe you can explain yours to me (feel free to phrase it in terms even an idiot like me can understand; I won't be offended;).

Linux systems often have standard locations for placing documenation (such as /usr/share/doc/<package-name>), and likewise for other unix systems. If Python would place it's documentation elsewhere users would get confused.

I'm talking of how a module/package gets put in a suitable Python directory (e.g. site-packages), which I'm assuming (unless proven otherwise) only requires that one knows which directory to put it in and moving the module/package to it. I'm also assuming that DU should not need to rearrange the contents of that package folder when installing it (except perhaps in special cases where it must install one of several platform-specific versions of a file, say; but that'll be the exception rather than the rule, and packages that don't require such special handling shouldn't need to go through the same in-depth procedures to install).

You seem to be muddying the definition of a package. There's a python package as a specific language construct and there's python package as "a distribution of a python library", which are seperate entities. Merging them is IMHO confusing. With little effort every python library could be placed into it's own subdirectory in site-packages: .../site-packages/ MyPackage.pth MyPackage/ module1.py package1/ __init__.py module2.py If the 'extra_path' argument of setup() were to be required we'd have a directory to that can be used to store additional information, and it would make d&d package management easier for those who like that. Ronald -- X|support bv http://www.xsupport.nl/ T: +31 610271479 F: +31 204416173