[Pythonmac-SIG] PackMan - in defence of Python code in the database

[Bob Ippolito]

On Saturday, Oct 11, 2003, at 18:19 America/New_York, Jack Jansen wrote:

>
> On 10-okt-03, at 22:45, Bob Ippolito wrote:
>>>> 1.  This is because we're using bdist_dumb, which is just as smart 
>>>> as it sounds.  We'll inevitably be using something else in the 
>>>> future that's a lot smarter.
>>>
>>> Definitely (and actually fairly simple[*]), but even then you cannot 
>>> be 100% sure that any binary distribution can be re-targeted to a 
>>> different location. There's nothing stopping the author of setup.py 
>>> to store sys.prefix in a datafile somewhere.
>>
>> There's nothing stopping the author of setup.py from writing an 
>> application that doesn't work either.. or stopping the author from 
>> using Makefile instead of setup.py.  I'm not really concerned with 
>> these people.
>
> That's not fair: having sys.prefix be correct has been the case in 
> distutils since day one.
> So if you're creating an extension module that needs a pathname at 
> runtime or something like
> that it's perfectly legitimate to add dependencies on sys.prefix. But, 
> all that said: there's
> little we can do about it...

Yeah but that should happen at runtime not build time, so I don't think 
it's really that relevant to PackMan.

>>> [*] Now that Python has its own tar module we could have a 
>>> bdist_slightlylessdumb that creates a tarfile that has magic tokens 
>>> for places like sys.prefix, etc. While a normal tar couldn't extract 
>>> these to the right place a Python script to do so is dead simple.
>>
>> That's what I was thinking.  I was actually really surprised that you 
>> were using bdist_dumb and rewriting the sys.prefix out of the paths 
>> to do user installs.  Whatever works, right? :)
>
> Partly, yes. And it's also a result of my attempts to use source 
> distributions unadultered
> (which lowers the maintainance load on the scapegoat). There's one 
> alternative to bdist_dumb,
> that looks much better from the description, but it's not generally 
> available: ActiveState
> have a way of building binary distributions where they basically do 
> "setup.py build", then
> freeze the tree, remove all the sources and pack that into an archive. 
> On the destination
> system they then do something quite like "setup.py install". This 
> doesn't work with standard
> distutils, because the "install" always does the "build" too.
>
> I would like to see something like this standardised in distutils, 
> because it keeps the
> setup.py script, so it allows the package writer to do things like 
> sys.prefix dependencies,
> as long as they're done at install time, not build time.

Sounds like a good solution to me.  This "setup.py" mechanism should be 
part of the "frozen archive", and not in the plist.

>>>> 2.   We shouldn't really be putting things in /System, ever.  
>>>> Anything that does system-wide installation should use the 
>>>> authorization API in order to acquire root access, even if the user 
>>>> has admin gid.
>>>
>>> In theory I agree with this. In practice there are a number of 
>>> problems:
>>> - Not putting anything into /System conflicts with the accepted 
>>> Python usage of packages adding stuff to include/python2.3. Until 
>>> Python/distutils learn about a secondary include directory I think 
>>> we have no options but putting things into /System (or not allowing 
>>> it, at the expensive of disabling Numeric support in many packages).
>>
>> Let's put that into the distutils discussion.  Python packages should 
>> have a canonical way of having headers, scripts, dynamic libraries, 
>> and other arbitrary data (like fonts, icons, etc.) associated with 
>> them *in the same or folder (or a closely related folder with 
>> identical permissions)* as the package itself.  We could propose 
>> something like Apple's bundles for this purpose (sans versioning, 
>> most likely).
>
> For datafiles: yes, definitely. But you can already do that by 
> stuffing the datafile into your package. For headers and scripts the 
> situation is different, as these are really for use outside of the 
> package. I think I see more in a scheme similar to what we discussed 
> here a couple of months ago for site-packages: having a number of 
> these for different "authorization groups". So there's site-packages 
> for vendor, system-wide, per-user, and similar for include files.
>
> Dynamic libraries are a whole different can of worms, I have no ready 
> solution for that.

Well if you keep everything inside some kind of package structure, 
symlinks can make the files available wherever else they're needed.  
This has the added benefit of allowing the package itself to find its 
own documentation files or headers for whatever reason, and it makes it 
pretty easy for the user to figure out what scripts/headers/etc belong 
to which package they installed.

>>> - We don't have the authorization API right now
>>
>> That's not a really good excuse, I can fix that this weekend.
>
> Then please please please do it with bgen, or with another technology 
> that will allow me to import this into the Python core distribution 
> and keep it maintainable.....

I don't think bgen is the right solution, this API has about 10 
functions to wrap and most of them are going to require special 
treatment.  I'm going to prototype it in Pyrex, and I'll take it down 
to hand coded C if it needs to be.

>>> Actually, that wouldn't even be needed due to pimp's ability to 
>>> update itself: just add a module pimp_update to site-packages and 
>>> pimp will override itself with the contents of that module (if the 
>>> version number is higher).
>>
>> That works too, forgot about that.  It's a shame that site-packages 
>> isn't allowed to just replace standard library stuff (by way of 
>> sys.path ordering), since there are so many batteries these days that 
>> don't necessarily update at the same speed as Python and it's quite 
>> possible for someone to botch packaging of a battery, such as you 
>> forgetting to have bsddb installed when you packaged Python
>
> That's by design: installing something should never break the standard 
> stuff. There is a way around it, but I'll only tell you if you promise 
> never to use it:-)
>
> Seriously: I discussed this with Guido and he sort-of agreed that 
> while standard third-parties should never override parts of the 
> standard lib there is one exception, and that's to allow Python core 
> developers responsible for a specific platform to override parts of 
> the standard library in case the platform ships with Python 
> pre-installed. You're free to read that long snippet as "that's > me":-).

The only way I'm aware of is to develop this "upgrade" feature into the 
package itself, which PyXML and PackMan both have.  Other than that the 
only solution I know of is to jigger sys.path or sys.modules yourself, 
but that requires changing site.py or importing something else.

One thing I'd like the power to "upgrade" are some of your bgen-erated 
modules, the QuickTime stuff being the prime example.  Lots of 
functions in your Qt module don't work as they're supposed to, or just 
aren't there..  Basically, you can't create a Qt movie from scratch 
with your version.  I have a modified version where I mostly just added 
functions with new names to your bgen.

-bob