[lxml-dev] lxml eggs and unicode strings

Hi there, I just found out that there is a hidden incompatibility in the compiled versions of lxml eggs we provide, at least in linux. Our provided versions are compiled with a Python that has 4 bytes unicode support (probably the default on ubuntu on which I built the 2.4 extension). If you try to install such an egg on a machine where unicode support is compiled with 2 bytes only, it'll fail with errors such as: ImportError: /usr/local/lib/python2.4/site-packages/lxml-1.0.2-py2.4-linux-i686.egg/lxml/etree.so: undefined symbol: PyUnicodeUCS4_FromEncodedObject I wonder whether there's anything within the egg distribution mechanism that lets us distinguish between such platforms. If not, I wonder what to do instead -- the simplest would be to add a FAQ entry and tell people to recompile from the sources. By the way, does Pyrex generate different C code depending on whether 4 or 2 byte unicode is used? If so, then that would mean an installation of pyrex as well for these people... Regards, Martijn

On Jul 31, 2006, at 11:41 AM, Martijn Faassen wrote:
Noticed that last week, too. Sorry I forgot to mention it over there.
As far as I know, this is typical of the Ubuntu distribution, and I'm 100% sure this egg was laid from Ubuntu. If the egg system could make a difference between distributions, it would be ok, imho. Charset problems are a plague.
I tried to compile from source on Mandriva, and it failed. I had no time to investigate (low priority for the task I was working on), it could very well have been something very trivial. Yours, --------- Georges Racinet Nuxeo SAS gracinet@nuxeo.com http://nuxeo.com Tel: +33 (0) 1 40 33 71 73

Georges Racinet wrote:
What platform were you on when you noticed this? Mandriva (as you mention below)? [snip]
I think Red Hat has been compiling Python with 4 bytes characters for ages too, so while this was Ubuntu (I did it), I'm also pretty sure it's also the case on Fedora.
Charset problems are a plague.
This is not your common charset problems. Mostly one can avoid the plague by just using unicode, but that's what we're doing here..
Interesting; let us know if you find out more. It's important to have the lxml C sources compile on all platforms, as otherwise people will be forced to use Pyrex, possibly even the forked version of Pyrex Stephan is maintaining. Regards, Martijn

Hi Martijn, Martijn Faassen wrote:
AFAIK, UCS4 is the default on most (though maybe not all) Python desktop/server installations under Linux, including SuSE, Redhat and (apparently) Debian/Ubuntu. Distributors tend to care more about broad support for all possible use cases than about memory requirements.
Sure. These cannot be compatible in current CPython (and that's highly unlikely to change).
I wouldn't know any way egg naming could help here. Google yields some discussions about this topic on the distutils list, but it seems they have not made their way into either distutils or setuptools. http://mail.python.org/pipermail/distutils-sig/2005-October/005222.html Anyway, if you have to recompile your Python version to get UCS2 strings, there's no reason not to require the same for the C extensions. Given the fact that all major distributions seem to use UCS4, a FAQ entry should be enough.
No, the distinction between different unicode encodings is handled completely inside the Python interpreter. The C code is not affected and Pyrex does not rely on it. To support parsing from unicode, lxml even has generic run-time support code to detect the internal unicode encoding, which should work for any encoding supported by libxml2/libiconv. Stefan

Stefan Behnel wrote: [snip]
Anyway, if you have to recompile your Python version to get UCS2 strings, there's no reason not to require the same for the C extensions.
Ah, so current CPython sources builds with 4 byte unicode by default? If this is for sure, then we're fairly safe. If not, then I wonder what to do - you'd like lxml to work with hand-compiled Pythons..
Given the fact that all major distributions seem to use UCS4, a FAQ entry should be enough.
It definitely is encouraging.
Good, that's what I was hoping for. That at least means people should be able to recompile without installing Pyrex first.
Cool! Regards, Martijn

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Martijn Faassen wrote:
Nope. The distros all pass the '--enable-unicode=ucs4' to configure. The default value for that option is 'yes', which maps to 'ucs2' unless you also have a usc4-enabled TCL. Tres. - -- =================================================================== Tres Seaver +1 202-558-7113 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFEzhWT+gerLs4ltQ4RAprdAKDatD1WDYW+wKzJlLZYra0OGXcxLACeJSDs CGNZmUnpCDiYbPuF9lwNO00= =8iYC -----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Tres Seaver wrote:
Perhaps we could use the following test inside 'setup.py', and modify the name of the binary egg to include the 'ucs2' vs. 'ucs4' flag?:: ucs_flag = sys.maxunicode > 65536 and 'ucs4' or 'ucs2' Tres. - -- =================================================================== Tres Seaver +1 202-558-7113 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFEzhdI+gerLs4ltQ4RAtQHAKDaSQm9mvJDj+oGUQJZOgHjdENnagCgh0gZ qQ9dwzju5C7s9KIlJVOJsVs= =qiPy -----END PGP SIGNATURE-----

Tres Seaver wrote:
Right, that's what I witness, too.
While that's nice to have, it doesn't really help us as a) we'd still have to build and ship both eggs (while the current UCS4 eggs seem to fit most users) and b) easy_install doesn't currently handle these extensions, so it would most likely just stop finding the eggs on cheeseshop if we added additional sections to the egg name. I still think it's enough to add a FAQ entry (which I already did) and otherwise ignore the problem for now. That way, the major distros are supported out-of-the-box. And for those who happen to use a UCS2 system, it's really not a big deal to build lxml from sources on a fairly recent and well installed Linux system. Stefan

Stefan Behnel wrote:
There'd be a significant amount of people who just build Python by hand though, and they can't use our eggs... [snip]
I agree that's all we can do on the lxml side. Apart from that, we can also talk to the distutils/setuptools people and raise this issue again. It's a fundamental problem with binary eggs that use unicode as long as Python ships with this configuration option. I'll send off a mail on this to the distutils SIG. Regards, Martijn

On Jul 31, 2006, at 11:41 AM, Martijn Faassen wrote:
Noticed that last week, too. Sorry I forgot to mention it over there.
As far as I know, this is typical of the Ubuntu distribution, and I'm 100% sure this egg was laid from Ubuntu. If the egg system could make a difference between distributions, it would be ok, imho. Charset problems are a plague.
I tried to compile from source on Mandriva, and it failed. I had no time to investigate (low priority for the task I was working on), it could very well have been something very trivial. Yours, --------- Georges Racinet Nuxeo SAS gracinet@nuxeo.com http://nuxeo.com Tel: +33 (0) 1 40 33 71 73

Georges Racinet wrote:
What platform were you on when you noticed this? Mandriva (as you mention below)? [snip]
I think Red Hat has been compiling Python with 4 bytes characters for ages too, so while this was Ubuntu (I did it), I'm also pretty sure it's also the case on Fedora.
Charset problems are a plague.
This is not your common charset problems. Mostly one can avoid the plague by just using unicode, but that's what we're doing here..
Interesting; let us know if you find out more. It's important to have the lxml C sources compile on all platforms, as otherwise people will be forced to use Pyrex, possibly even the forked version of Pyrex Stephan is maintaining. Regards, Martijn

Hi Martijn, Martijn Faassen wrote:
AFAIK, UCS4 is the default on most (though maybe not all) Python desktop/server installations under Linux, including SuSE, Redhat and (apparently) Debian/Ubuntu. Distributors tend to care more about broad support for all possible use cases than about memory requirements.
Sure. These cannot be compatible in current CPython (and that's highly unlikely to change).
I wouldn't know any way egg naming could help here. Google yields some discussions about this topic on the distutils list, but it seems they have not made their way into either distutils or setuptools. http://mail.python.org/pipermail/distutils-sig/2005-October/005222.html Anyway, if you have to recompile your Python version to get UCS2 strings, there's no reason not to require the same for the C extensions. Given the fact that all major distributions seem to use UCS4, a FAQ entry should be enough.
No, the distinction between different unicode encodings is handled completely inside the Python interpreter. The C code is not affected and Pyrex does not rely on it. To support parsing from unicode, lxml even has generic run-time support code to detect the internal unicode encoding, which should work for any encoding supported by libxml2/libiconv. Stefan

Stefan Behnel wrote: [snip]
Anyway, if you have to recompile your Python version to get UCS2 strings, there's no reason not to require the same for the C extensions.
Ah, so current CPython sources builds with 4 byte unicode by default? If this is for sure, then we're fairly safe. If not, then I wonder what to do - you'd like lxml to work with hand-compiled Pythons..
Given the fact that all major distributions seem to use UCS4, a FAQ entry should be enough.
It definitely is encouraging.
Good, that's what I was hoping for. That at least means people should be able to recompile without installing Pyrex first.
Cool! Regards, Martijn

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Martijn Faassen wrote:
Nope. The distros all pass the '--enable-unicode=ucs4' to configure. The default value for that option is 'yes', which maps to 'ucs2' unless you also have a usc4-enabled TCL. Tres. - -- =================================================================== Tres Seaver +1 202-558-7113 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFEzhWT+gerLs4ltQ4RAprdAKDatD1WDYW+wKzJlLZYra0OGXcxLACeJSDs CGNZmUnpCDiYbPuF9lwNO00= =8iYC -----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Tres Seaver wrote:
Perhaps we could use the following test inside 'setup.py', and modify the name of the binary egg to include the 'ucs2' vs. 'ucs4' flag?:: ucs_flag = sys.maxunicode > 65536 and 'ucs4' or 'ucs2' Tres. - -- =================================================================== Tres Seaver +1 202-558-7113 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFEzhdI+gerLs4ltQ4RAtQHAKDaSQm9mvJDj+oGUQJZOgHjdENnagCgh0gZ qQ9dwzju5C7s9KIlJVOJsVs= =qiPy -----END PGP SIGNATURE-----

Tres Seaver wrote:
Right, that's what I witness, too.
While that's nice to have, it doesn't really help us as a) we'd still have to build and ship both eggs (while the current UCS4 eggs seem to fit most users) and b) easy_install doesn't currently handle these extensions, so it would most likely just stop finding the eggs on cheeseshop if we added additional sections to the egg name. I still think it's enough to add a FAQ entry (which I already did) and otherwise ignore the problem for now. That way, the major distros are supported out-of-the-box. And for those who happen to use a UCS2 system, it's really not a big deal to build lxml from sources on a fairly recent and well installed Linux system. Stefan

Stefan Behnel wrote:
There'd be a significant amount of people who just build Python by hand though, and they can't use our eggs... [snip]
I agree that's all we can do on the lxml side. Apart from that, we can also talk to the distutils/setuptools people and raise this issue again. It's a fundamental problem with binary eggs that use unicode as long as Python ships with this configuration option. I'll send off a mail on this to the distutils SIG. Regards, Martijn
participants (4)
-
Georges Racinet
-
Martijn Faassen
-
Stefan Behnel
-
Tres Seaver