[lxml-dev] Building lxml on windows: a guide.
Note: This is my first post on this list. There was an absence of windows binaries for lxml on the net (all the sites that had older versions were down). Unfortunately, there wasn't enough information to make the process of building on windows easy. I wrote this guide to solve both these problems. I hope that the information here can at worst help out others trying to build on windows and at best get included in the documentation and maintenance. I also have a windows installer created for lxml 0.9.1 that is staticly built with iconv-1.9.1, libxml2 2.6.23, libxslt 1.1.15, and zlib 1.2.3. If there is some ftp site or something I can upload it to, in order to give wide access to it, please let me know what to do. David Sankel Building lxml on windows =================== First you'll need to download the latest version of all the required files. Download them all to the same directory. * libxml: Availible from http://codespeak.net/lxml/ * iconv, libxml2, libxslt, and zlib are all availible from xmlsoft.org. The place to go on the ftp site is ftp://xmlsoft.org/libxml2/win32. Your directory should now have something like the following files in it: iconv-1.9.1.win32.zip libxml2-2.6.23.win32.zip libxslt-1.1.15.win32.zip lxml-0.9.1.tgz zlib-1.2.3.win32.zip Now extract each of those files in the _same_ directory. Now you should have something like this: iconv-1.9.1.win32/ iconv-1.9.1.win32.zip libxml2-2.6.23.win32/ libxml2-2.6.23.win32.zip libxslt-1.1.15.win32/ libxslt-1.1.15.win32.zip lxml-0.9.1/ lxml-0.9.1.tgz zlib-1.2.3.win32/ zlib-1.2.3.win32.zip Go to the lxml-0.9.1 directory and edit the Makefile. There should be a section that looks like this:: ext_modules = [ Extension( "lxml.etree", sources = sources, extra_compile_args = ['-w'] + flags('xslt-config --cflags'), extra_link_args = flags('xslt-config --libs') )], Change it to this (Warning: make sure you are using version numbers that correspond to your downloads) (Note: the _a portion of the libraries means that we are statically linking. If you want to use dlls(why?), link to the dll version of the libraries):: ext_modules = [ Extension( "lxml.etree", sources = sources, extra_compile_args = [ "-w", "-I..\\libxml2-2.6.23.win32\\include", "-I..\\libxslt-1.1.15.win32\\include", "-I..\\zlib-1.2.3.win32\\include", "-I..\\iconv-1.9.1.win32\\include" ], extra_link_args = [ "..\\libxml2-2.6.23.win32\\lib\\libxml2_a.lib", "..\\libxslt-1.1.15.win32\\lib\\libxslt_a.lib", "..\\zlib-1.2.3.win32\\lib\\zlib.lib", "..\\iconv-1.9.1.win32\\lib\\iconv_a.lib" ] )], Now you should be able to use setup.py and everything should work well. "python setup.py bdist_wininst" will create a windows installer in the pkg directory.
Hi David, David Sankel wrote:
There was an absence of windows binaries for lxml on the net (all the sites that had older versions were down).
"www.python.org" is rarely down: http://www.python.org/pypi/lxml/0.9.1 That's not the most flashy recent version, but it will do (it's not even two months old). And it has windows binaries (which we do not have for 0.9.2 yet as it was released fairly recently). Easiest way to install is by passing "lxml=0.9.1" to "easy_install". Even if you don't want to install EasyInstall, you can still download the egg and unzip it to a suitable directory by hand. We had so about 90 downloads of the egg so far without any feedback, so I guess there are not too many problems with it. Stefan
Hi David, David Sankel wrote:
This is my first post on this list. There was an absence of windows binaries for lxml on the net (all the sites that had older versions were down). Unfortunately, there wasn't enough information to make the process of building on windows easy. I wrote this guide to solve both these problems.
I hope that the information here can at worst help out others trying to build on windows and at best get included in the documentation and maintenance.
Thanks! I'll check this in.
I also have a windows installer created for lxml 0.9.1 that is staticly built with iconv-1.9.1, libxml2 2.6.23, libxslt 1.1.15, and zlib 1.2.3. If there is some ftp site or something I can upload it to, in order to give wide access to it, please let me know what to do.
Do I understand that you can just install this, import lxml and everything works, without having to worry about installing libxml2 on windows? That'd be great! If you drop me a version I shall put it online in the Python cheeseshop. Regards, Martijn
Hi Martijn, Martijn Faassen wrote:
David Sankel wrote:
This is my first post on this list. There was an absence of windows binaries for lxml on the net (all the sites that had older versions were down). Unfortunately, there wasn't enough information to make the process of building on windows easy. I wrote this guide to solve both these problems.
I hope that the information here can at worst help out others trying to build on windows and at best get included in the documentation and maintenance.
Thanks! I'll check this in.
Please, don't. Last thing I heard was that lxml compiles just fine on Windows. I really don't see why we should tell people to install libraries in weird places and mess around in setup.py. Stefan
Stefan Behnel wrote:
Hi Martijn, [snip]
I hope that the information here can at worst help out others trying to build on windows and at best get included in the documentation and maintenance.
Thanks! I'll check this in.
Please, don't. Last thing I heard was that lxml compiles just fine on Windows. I really don't see why we should tell people to install libraries in weird places and mess around in setup.py.
What weird places? There's nothing weird with these instructions that I can see. *Does* the setup.py just work on windows? xslt-config must be installed on the path, and things just work then? In addition, these instructions allow you, as far as I understand, to create a *static* version of lxml, including libxml2 and the like. That's pretty neat. No more separately downloading the libxml2 libraries; you just get them included. Anyway, if lxml compiles fine on Windows with the current instructions, we should consider the following steps: * what was it that tripped up David? We should look into finding out and adding a bit to the documentation so people don't trip up in the future. * I'd like to publish the static version of lxml for Windows, as that makes deploying lxml on Windows that much more easy. * If we do that, it'd be nice if we had instructions on how to build the static version (which David provides). I won't check in anything until we figure those out. I just think we should not ignore David's contribution either. So, David, what was it that stoppped you from building lxml on Windows? Regards, Martijn
Hi Martijn, Martijn Faassen wrote:
*Does* the setup.py just work on windows? xslt-config must be installed on the path, and things just work then?
There are instructions on how to install those on the libxml2-for-Windows homepage. Remember that you don't normally need to build lxml, so the PATH doesn't matter. lxml is a C-extension, that's not as easy as a Python script. That's why we try to keep users from having to build it themselves. People who want the one-command install should just go with easy_install, which works perfectly on any half-way well-configured system. All that's needed in addition is installing the libxml2/etc. libraries (as their homepage describes).
In addition, these instructions allow you, as far as I understand, to create a *static* version of lxml, including libxml2 and the like.
I really don't mind a static version, despite the ugliness-factor. As I already stated a while ago, Windows heavily lacks any package management. (That's an absolute killer argument against Windows BTW, but, well, it's Windows...) Compiling a static version is easy, it's just that the way to do is not portable, so we can't put it into setup.py.
* what was it that tripped up David?
The main problem seemed to be that he didn't find the binaries - which might have been because of the outdated install docs and because we don't have a Windows egg for 0.9.2 yet. I'm sorry for that.
* I'd like to publish the static version of lxml for Windows, as that makes deploying lxml on Windows that much more easy.
Sure, if someone provides a static egg - one more to put on cheeseshop. Stefan
Hey Stefan, Stefan Behnel wrote:
Martijn Faassen wrote:
*Does* the setup.py just work on windows? xslt-config must be installed on the path, and things just work then?
There are instructions on how to install those on the libxml2-for-Windows homepage. Remember that you don't normally need to build lxml, so the PATH doesn't matter. lxml is a C-extension, that's not as easy as a Python script. That's why we try to keep users from having to build it themselves. People who want the one-command install should just go with easy_install, which works perfectly on any half-way well-configured system. All that's needed in addition is installing the libxml2/etc. libraries (as their homepage describes).
Right, we should separate the compilation discussion from the installation discussion. If you want to install lxml on Windows, use a compiled version. Let's talk about compilation here. The question remains whether setup.py just works on Windows or whether the people who built the windows versions had to hack it. If the latter, it'd be nice to document these instructions for future reference. If everything does work with the plain setup.py, we should figure out what tripped up David as he apparently needed to hack things.
In addition, these instructions allow you, as far as I understand, to create a *static* version of lxml, including libxml2 and the like.
I really don't mind a static version, despite the ugliness-factor. As I already stated a while ago, Windows heavily lacks any package management. (That's an absolute killer argument against Windows BTW, but, well, it's Windows...)
Compiling a static version is easy, it's just that the way to do is not portable, so we can't put it into setup.py.
Right, so people typically have to hack their setup.py to do so, I guess. We can still document it somewhere.
* what was it that tripped up David?
The main problem seemed to be that he didn't find the binaries - which might have been because of the outdated install docs and because we don't have a Windows egg for 0.9.2 yet. I'm sorry for that.
Perhaps David can provide a windows installer for 0.9.2 that includes the static libraries?
* I'd like to publish the static version of lxml for Windows, as that makes deploying lxml on Windows that much more easy.
Sure, if someone provides a static egg - one more to put on cheeseshop.
Windows installer should be easy. But eggs? Is it possible to have a static egg and a non-static egg both? How would people choose? Should we switch to the static procedure on windows altogether? What do people think? Regards, Martijn
Hi Martijn, Martijn Faassen wrote:
Stefan Behnel wrote:
if someone provides a static egg - one more to put on cheeseshop.
Windows installer should be easy. But eggs? Is it possible to have a static egg and a non-static egg both? How would people choose? Should we switch to the static procedure on windows altogether? What do people think?
I was thinking about that, too. I guess all-in-one installers are mainly for new users. People who want to upgrade will likely prefer independent downloads. So, I think it's better to have a full-fledged installer and a dynamic egg. Stefan
To clarify your questions. I'm running windows and want to get things installed as quickly as possible (binaries++). * I went to the website and clicked on lxml 9.0.1 link in the news, saw that it was direct source download and canceled. * I scrolled down, scanning for some sort of binary version and clicked the "lxml at the Python cheeseshop" link. (Note: it wasn't immediately obvious that here is where binaries would be) * Saw that there was no windows binaries there. [Made assumption that there were no windows binaries available from the developers.] * Looked at where I got my previous version from my notes. It was from http://carcass.dhs.org/ and that site no longer exists. At this point I figured I'd have to go with doing it by source. * Downloaded lxml 9.0.1 from link in the news (I guess 9.0.2 news wasn't up yet) * python setup.py build:: src\lxml\etree.c(9) : fatal error C1083: Cannot open include file: 'libxml/encod ing.h': No such file or directory error: command '"c:\Program Files\Microsoft Visual Studio .NET 2003\Vc7\bin\cl.e xe"' failed with exit status 2 * Opened README.txt * Opened INSTALL.txt * Skipped down to "Installation on Windows" section as recommended in the first paragraph. * Read that it would be a hassle to compile :-( * Read that binaries are available on http://carcass.dhs.org/ :-) * Found that that site was down :-( [That was unhelpful, so I figured I'd try to get it to work myself] * Looked to xmlsoft.org to get lxml binaries. * Found that that distribution included includes and everything :-) * That distribution didn't have the xslt-config binary that the normal setup.py required. * Hacked away at the setup.py so it'd work. * Found that the distribution from xmlsoft.org had static binaries :-D (It was a _huge_ pain requiring my developers to download these extra dlls before) (Note: I have heard of python eggs, but haven't gotten around to reading about it yet. I'm not sure if it would have saved me some steps from above.) --
Note that due to my fault the installation documentation on the website was slightly up to date. I've updated it just now, so David, could you check whether this would've helped you getting unstuck trying to build lxml on windows?
Unfortunately the new instructions wouldn't have helped because I used the docs in the source distribution to try to figure out how to install it after not finding a windows binary.
If not, could you suggest how to modify it?
For the main page, you could change the news releases to something like this: lxml 0.9.2 (sourcelink, binarieslink) released (changes for 0.9.2)
Windows installer should be easy. But eggs? Is it possible to have a static egg and a non-static egg both? How would people choose? Should we switch to the static procedure on windows altogether? What do people think?
Having gone from the dll version installer to the static version installer, I can't say enough about how much easier it is to deploy the static version.
Perhaps David can provide a windows installer for 0.9.2 that includes the static libraries?
Sure, no problem. David
Hi David, thanks for the long explanation. That's interesting to read. David Sankel wrote:
To clarify your questions. I'm running windows and want to get things installed as quickly as possible (binaries++).
* I went to the website and clicked on lxml 9.0.1 link in the news, saw that it was direct source download and canceled.
True, I always wondered if it was a good idea to have those links in the news section... :) I removed them and left them only under "downloads".
* I scrolled down, scanning for some sort of binary version and clicked the "lxml at the Python cheeseshop" link. (Note: it wasn't immediately obvious that here is where binaries would be)
That should be clear now.
* Saw that there was no windows binaries there. [Made assumption that there were no windows binaries available from the developers.]
Not for 0.9.2 yet, only for 0.9 and 0.9.1. Note that "windows binaries" are distributed in "egg" form as for most other platforms. (should be clear from the "win32" part in the file name).
* Looked at where I got my previous version from my notes. It was from http://carcass.dhs.org/ and that site no longer exists.
That's ok, that was a temporary solution. The web page is already fixed.
At this point I figured I'd have to go with doing it by source.
:)
* Downloaded lxml 9.0.1 from link in the news (I guess 9.0.2 news wasn't up yet)
Exactly.
* python setup.py build::
src\lxml\etree.c(9) : fatal error C1083: Cannot open include file: 'libxml/encod ing.h': No such file or directory error: command '"c:\Program Files\Microsoft Visual Studio .NET 2003\Vc7\bin\cl.e xe"' failed with exit status 2
You didn't read the installation instructions on the web page. The link is more visible now.
* Opened README.txt * Opened INSTALL.txt * Skipped down to "Installation on Windows" section as recommended in the first paragraph.
That section is gone now that we have eggs.
* Read that it would be a hassle to compile :-(
That's why we have eggs. :)
* Read that binaries are available on http://carcass.dhs.org/ :-) * Found that that site was down :-(
See above.
[That was unhelpful, so I figured I'd try to get it to work myself]
* Looked to xmlsoft.org to get lxml binaries. * Found that that distribution included includes and everything :-)
Sure.
* That distribution didn't have the xslt-config binary that the normal setup.py required.
It's not a binary, more of a script. But that's good to know. Maybe Steve can tell us if there was anything he did to make it work without it.
* Hacked away at the setup.py so it'd work. * Found that the distribution from xmlsoft.org had static binaries :-D
As Martijn said, a static installer would be nice to have for windows users.
(It was a _huge_ pain requiring my developers to download these extra dlls before)
You should tell them to use easy_install. Doesn't help with DLLs, but with everything that's Python. lxml works nicely with it.
(Note: I have heard of python eggs, but haven't gotten around to reading about it yet. I'm not sure if it would have saved me some steps from above.)
Python eggs are the best way to distribute and use various versions and binaries of a Python package. See the EasyInstall link on the lxml install page.
Unfortunately the new instructions wouldn't have helped because I used the docs in the source distribution to try to figure out how to install it after not finding a windows binary.
The web page is built from the text files in doc/.
For the main page, you could change the news releases to something like this: lxml 0.9.2 (sourcelink, binarieslink) released (changes for 0.9.2)
I prefer refering to cheeseshop here. "binarieslink" is a bit misleading for a multi-platform thing.
Having gone from the dll version installer to the static version installer, I can't say enough about how much easier it is to deploy the static version.
It's static. Where's the problem?
Perhaps David can provide a windows installer for 0.9.2 that includes the static libraries?
Sure, no problem.
Cool, please send it to Martijn so that he can copy it to cheeseshop. Martijn, please also rebuild the web pages again. Thanks for the feedback, Stefan
Stefan Behnel a écrit :
Hi David,
thanks for the long explanation. That's interesting to read.
David Sankel wrote:
To clarify your questions. I'm running windows and want to get things installed as quickly as possible (binaries++).
* I went to the website and clicked on lxml 9.0.1 link in the news, saw that it was direct source download and canceled.
True, I always wondered if it was a good idea to have those links in the news section... :)
I removed them and left them only under "downloads".
What about replacing the News section directly by the download section with additional links to the CHANGES file? I find the current front page too long and too redundant. The "easy_install lxml" command should be mntioned on top of the download section so that people first think to use it before getting tempted to click on one the source distrib links. -- Olivier
Hi Olivier, Olivier Grisel wrote:
Stefan Behnel a écrit :
Hi David,
thanks for the long explanation. That's interesting to read.
David Sankel wrote:
To clarify your questions. I'm running windows and want to get things installed as quickly as possible (binaries++).
* I went to the website and clicked on lxml 9.0.1 link in the news, saw that it was direct source download and canceled.
True, I always wondered if it was a good idea to have those links in the news section... :)
I removed them and left them only under "downloads".
What about replacing the News section directly by the download section with additional links to the CHANGES file? I find the current front page too long and too redundant.
Good idea. done.
The "easy_install lxml" command should be mntioned on top of the download section so that people first think to use it before getting tempted to click on one the source distrib links.
Already did that in my last commit. :) Stefan
Hey, Note that due to my fault the installation documentation on the website was slightly up to date. I've updated it just now, so David, could you check whether this would've helped you getting unstuck trying to build lxml on windows? If not, could you suggest how to modify it? Regards, Martijn
participants (4)
-
David Sankel -
Martijn Faassen -
Olivier Grisel -
Stefan Behnel