[python-win32] Building pywin32 msi installers

[Mark Hammond]

[oops - sorry for the delay]

On 22/06/2013 3:03 AM, Will Sadkin wrote:
> On 20/06/2013 3:16 AM , Mark Hammond wrote:
>
>> I've wanted bdist_msi to work for ages.  There was some issue with
>> the install scripts that always caused grief, and my memory is hazy
>> - it *might* be that the uninstall process doesn't run the
>> post-install script at the correct time - ie, that it runs *after*
>> the uninstall, which means pywin32 isn't available to undo some of
>> the stuff it did as it can't import pywin32 itself.
>
> There's a comment in the pywin32_postinstall.py script that says:
> elif arg == "-remove": # bdist_msi calls us before uninstall, so we
> can undo what we # previously did.  Sadly, bdist_wininst calls us
> *after*, so # we can't do much at all. if not is_bdist_wininst:
> uninstall()
>
> And I've confirmed that this comment re: bdist_msi is correct.  Given
> this, I expect that these issues were with the bdist_wininst, and so
> a working msi installer might be even better than the existing
> installer on that score.

Awesome :)

>> Another problem I'd expect to find is that the post-install script
>> tries to tell distutils about new files and registry entries
>> created at install time - this probably fails silently now (ie, it
>> might *appear* to work but probably leaves trash behind.)
>
> I couldn't identify the code that does this thing you mention above,

It's the file_created and directory_created methods at the top of 
pywin32_postinstall.py.


> but given this concern, I did some careful checking.  As far as I can
> tell, there are 3 complaints re: trash:
>
> 1) What's left behind in the file system after an install followed by
> an immediate uninstall are 3 site-packages subtrees: win32\lib,
> win32com\{client,server,servers}, and win32comext\{axscript,shell},
> and all of these only contain .pyc files.  Unfortunately, as .pycs
> are generated post-install by the interpreter, and I'm not sure how
> to identify the ones associated with just the pywin32 stuff, so I
> don't quite know how to get rid of them so that these directories are
> cleanly removed.  Further, if the other directories installed get
> invoked, I'm sure that there will be .pycs generated there too.  Is
> there a way to have the postinstall script know (without hardcoding),
> which directories it should recursively clean out the .pycs from?  Or
> is there some other mechanism I can use that I'm not aware of?

I'd think it's OK to be fairly aggressive about removing compiled files 
- so if the only solution we can come up with is to delete all .pyc/.pyo 
files from the pywin32 install dirs (which is a fixed set) I'd be 
inclined to say that's OK.

> 2) There is also much black magic juju-bwana re: create_shortcut that
> is in that script and I don't entirely understand...  But running the
> postinstall.py -install manually on windows 7, it says:
>
> Can't install shortcuts -
> u'C:\\Users\\DevTest7\\AppData\\Roaming\\Microsoft\\Windows\\Start
> Menu\\Programs\\Python 2.7' is not a folder.
>
> I don't yet know why it's trying to use this folder, but this whole
> thing is part of the black magic I don't understand, and there are
> comments in the postinstall script that suggest that none of this
> stuff actually works on windows 7 anyway.  Do we know if this step
> actually works for win7 with the current installer?

In theory, this is where a Python install done "for me" would have stuck 
the start menu items.

>
> 3) And on manually trying the -remove version of the script, it
> complained: Failed to unregister Pythonwin: [Error 5] Access is
> denied
>
> But I tracked this down to a bug in the uninstall code in the
> postinstall.py script; _winreg.DeleteKey() doesn't allow you delete a
> key with subkeys, but you had created two keys with (command) subkeys
> in the install process.  Once I fixed the postinstall script to
> Delete the subkeys and then the root key, the script properly removed
> the pythonwin.exe keys and unregistered it.  (I suspect this is
> actually currently broken in the regular installer as well if you use
> the bdist_wininst distutils mechanism, unless that code is
> circumvented in your .exe installer.)  (I will supply a patch for
> this too.)

huh - I've never seen that before, but what you describe sounds very likely.

> Should I be looking for anything else while I'm at it?

The only thing that does come to mind is earlier Windows versions - if 
you have, eg, XP on a VM it's probably worth doing a test cycle there 
too.  Similarly, testing when Python is installed "for me" or "for 
everyone" would be good (IIRC, that's disabled for Vista, but is enabled 
on Win7?)

>>> 1) I couldn't build from the latest source version (218).  [...]
>> Yeah, I screwed that release up in that regard.  I normally unzip
>> the source package and attempt to build it before release, but
>> forgot that time :(
>
> Ok, well, with luck, the changes to allow construction of an official
> msi can become part of version 219.  ;)
>
>> I have used the patch version in the past - mainly when I've found
>> one single package was uploaded incorrectly due to a build issue
>> rather than due to a bug in the code itself. eg:
>> https://sourceforge.net/projects/pywin32/files/pywin32/Build216/ -
>> where is a "216.1" build for Python 3.2.
>>
>> The other thing I've done in the past is to upload custom pywin32
>> builds for people when they've reported a bug but can't rebuild the
>> world to test it - I usually change that version string immediately
>> after release, so any such "interim" builds can't be confused with
>> the official builds (but such builds don't end up on sourceforge)
>>
>> But I'm happy to change things in this area if it blocks support
>> for MSI.  The other thing I'd want to make sure is that the Python
>> version in that number does something sane - eg, I wouldn't want
>> the MSI infrastructure to think that "2.7.217" was an earlier
>> version (or even related in any way) than "3.2.216" etc.
>
> Yeeeaaaahhh, the docstring on the strict version class says "the
> rationale for this version string numbering system will be explained
> in the distutils documentation," but no such rationale is provided
> there.  Further, here's a snippet of the bdist_msi.py code:
>
> version = metadata.get_version() # ProductVersion must be strictly
> numeric # XXX need to deal with prerelease versions sversion =
> "%d.%d.%d" % StrictVersion(version).version # Prefix ProductName with
> Python x.y, so that # it sorts together with the other Python
> packages # in Add-Remove-Programs (APR) fullname =
> self.distribution.get_fullname() if self.target_version: product_name
> = "Python %s %s" % (self.target_version, fullname) else: product_name
> = "Python %s" % (fullname)
>
> This assumes that the StrictVersion(version).version is literally a
> tuple of 3 components; otherwise the string format would generate an
> exception; fortunately, if you don't provide the 3rd component, it
> appends a 0 for you (see below).   Also, the above code automatically
> sticks "Python x.y" into the product name used, so I don't think we
> need to include those in the version string like I was.
>
> From the distutils\version.py file: class StrictVersion (Version):
> [...] version_re = re.compile(r'^(\d+) \. (\d+) (\. (\d+))?
> ([ab](\d+))?$', re.VERBOSE)
>
> def parse (self, vstring): match = self.version_re.match(vstring) if
> not match: raise ValueError, "invalid version number '%s'" % vstring
>
> (major, minor, patch, prerelease, prerelease_num) = \ match.group(1,
> 2, 4, 5, 6)
>
> if patch: self.version = tuple(map(string.atoi, [major, minor,
> patch])) else: self.version = tuple(map(string.atoi, [major, minor])
> + [0])
>
> if prerelease: self.prerelease = (prerelease[0],
> string.atoi(prerelease_num)) else: self.prerelease = None
>
> Given the code in bdist_msi using the current python release
> major.minor release number already in the product name, this suggests
> that we should just specify build_id as major, and your patch number
> (usually .0) as minor in the version string.  That way, you don't
> have to change a thing with respect to versioning, and instead of my
> previous patch, we can just use your build_id_patch variable as the
> dist version, eg:
>
> dist = setup(name="pywin32", !       version=str(build_id),
> description="Python for Window Extensions", long_description="Python
> extensions for Microsoft Windows\n" "Provides access to much of the
> Win32 API, the\n" "ability to create and use COM objects, and the\n"
> "Pythonwin environment.", --- 2328,2340 ---- 'build_py' :
> my_build_py, 'build_scripts' : my_build_scripts, }
>
> dist = setup(name="pywin32", !       # msi construction needs a
> major.minor version, so we use build_id_patch !       # for this
> instead: !       version=str(build_id_patch), description="Python for
> Window Extensions", long_description="Python extensions for Microsoft
> Windows\n"
>
> and that's the only change required in the setup file!
>
> This produces an msi named pywin32-217.0.win32-py2.7.msi, which also
> (mostly) avoids the repetitive redundancy there too.

Nice!

>>> It turns out that this distutils bug was reported back in August
>>> of 2012, and a patch has even been supplied (see
>>> http://bugs.python.org/issue15797.)  When I applied the supplied
>>> patch, and rebuilt my msi, it properly ran the postinstall
>>> script, and (as far as I can tell) the resulting install works
>>> perfectly!
>>
>> I can probably help push that through.  Although it seems unlikely
>> for it to be accepted into the 2.x branch, which is a problem.
>> OTOH though, if you help get this working and you only care about
>> 3.x, I could certainly live with the fact that MSI installers are
>> only available for 3.x...
>
> Regrettably, I need them for 2.7.
>
>> I wonder if there are heuristics the post-install script could use
>> to work around this itself? You said the uninstall worked - did it
>> actually leave a clean file-system afterwards?  If so, it would
>> seem the issue I mentioned before might not be a problem at all -
>> but it's not clear to me how, even if pywin32 was fully unavailable
>> at uninstall time, how we could determine if this invocation is for
>> a re-install or an uninstall (ie, it's not clear to me that the
>> absence of certain files would be a reliable guide here)
>
> Yeah, I thought about that, and struggled with the same issues:
> without arguments, how would it know whether it was being invoked for
> install or remove?  But when I discovered what the real problem was,
> and went a-googling, I discovered that bug report and patch
> submission against the distutils, and so I thought that was the
> better choice.  If I can resolve the other issues, and submit
> patches, is there any way I can convince you to to apply that
> distutils patch on your system, so that you can then use it to build
> the msi's?  If they work and come from you, I suspect no one will
> care that they were built with an "unsanctioned" version of
> distutils...

Sure, sounds fine.

Really happy to see this moving :)

Mark