Hi, I don't know anyone building Python without Unicode. I would prefer to modify configure to raise an error, and drop #ifdef in the code. (Stop supporting building Python 2 without Unicode.) Building Python 2 without Unicode support is not an innocent change. Python is moving strongly to Unicode: Python 3 uses Unicode by default. So to me it sounds really weird to work on building Python 2 without Unicode support. It means that you may have "Python 2" and "Python 2 without Unicode" which are not exactly the same language. IMO u"unicode" is part of the Python 2 language. --disable-unicode is an old option added while Python 1.5 was very slowly moving to Unicode. I have the same opinion on --without-thread option (we should stop supporting it, this option is useless). I worked in the embedded world, Python used for the UI of a TV set top box. Even if the hardware was slow and old, Python was compiled with threads and Unicode. Unicode was mandatory to handle correctly letters with diacritics, threads were used to handle network and D-Bus for examples. Victor 2014-06-24 10:22 GMT+02:00 Serhiy Storchaka <storchaka@gmail.com>:

I submitted a number of patches which fixes currently broken Unicode-disabled build of Python 2.7 (built with --disable-unicode configure option). I suppose this was broken in 2.7 when C implementation of the io module was introduced.

http://bugs.python.org/issue21833 -- main patch which fixes the io module and adds helpers for testing.

http://bugs.python.org/issue21834 -- a lot of minor fixes for tests.

Following issues fix different modules and related tests:

http://bugs.python.org/issue21854 -- cookielib http://bugs.python.org/issue21838 -- ctypes http://bugs.python.org/issue21855 -- decimal http://bugs.python.org/issue21839 -- distutils http://bugs.python.org/issue21843 -- doctest http://bugs.python.org/issue21851 -- gettext http://bugs.python.org/issue21844 -- HTMLParser http://bugs.python.org/issue21850 -- httplib and SimpleHTTPServer http://bugs.python.org/issue21842 -- IDLE http://bugs.python.org/issue21853 -- inspect http://bugs.python.org/issue21848 -- logging http://bugs.python.org/issue21849 -- multiprocessing http://bugs.python.org/issue21852 -- optparse http://bugs.python.org/issue21840 -- os.path http://bugs.python.org/issue21845 -- plistlib http://bugs.python.org/issue21836 -- sqlite3 http://bugs.python.org/issue21837 -- tarfile http://bugs.python.org/issue21835 -- Tkinter http://bugs.python.org/issue21847 -- xmlrpc http://bugs.python.org/issue21841 -- xml.sax http://bugs.python.org/issue21846 -- zipfile

Most fixes are trivial and are only several lines of a code.

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.co...

Le 24/06/2014 07:04, Skip Montanaro a écrit :

I can't see any reason to make a backwards-incompatible change to Python 2 to only support Unicode. You're bound to break somebody's setup.

Apparently, that setup would already have been broken for years. Regards Antoine.

24.06.14 14:50, Victor Stinner написав(ла):

2014-06-24 13:04 GMT+02:00 Skip Montanaro <skip@pobox.com>:

...

I can't see any reason to make a backwards-incompatible change to Python 2 to only support Unicode. You're bound to break somebody's setup. Wouldn't it be better to fix bugs as Serhiy has done?

According to the long list of issues, I don't think that it's possible to compile and use Python stdlib when Python is compiled without Unicode support. So I'm not sure that we can say that it's an backward-incompatible change.

Python has about 300 modules, my patches fix about 30 modules (only 8 of them cause compiling error). And that's almost all. Left only pickle, json, etree, email and unicode-specific modules (codecs, unicodedata and encodings). Besides pickle I'm not sure that others can be fixed. The fact that only small fraction of modules needs fixes means that Python without unicode support can be pretty usable. The main problem was with testing itself. Test suite depends on tempfile, which now uses io.open, which didn't work without unicode support (at least since 2.7).

If Serhiy wants to spend his time supporting this arcane feature, he can do that. It doesn't really seem worth risking regressions to do this, though. On Tue, Jun 24, 2014, at 01:55, Victor Stinner wrote:

Hi,

I don't know anyone building Python without Unicode. I would prefer to modify configure to raise an error, and drop #ifdef in the code. (Stop supporting building Python 2 without Unicode.)

Building Python 2 without Unicode support is not an innocent change. Python is moving strongly to Unicode: Python 3 uses Unicode by default. So to me it sounds really weird to work on building Python 2 without Unicode support. It means that you may have "Python 2" and "Python 2 without Unicode" which are not exactly the same language. IMO u"unicode" is part of the Python 2 language.

--disable-unicode is an old option added while Python 1.5 was very slowly moving to Unicode.

I have the same opinion on --without-thread option (we should stop supporting it, this option is useless). I worked in the embedded world, Python used for the UI of a TV set top box. Even if the hardware was slow and old, Python was compiled with threads and Unicode. Unicode was mandatory to handle correctly letters with diacritics, threads were used to handle network and D-Bus for examples.

Victor

2014-06-24 10:22 GMT+02:00 Serhiy Storchaka <storchaka@gmail.com>:

...

I submitted a number of patches which fixes currently broken Unicode-disabled build of Python 2.7 (built with --disable-unicode configure option). I suppose this was broken in 2.7 when C implementation of the io module was introduced.

http://bugs.python.org/issue21833 -- main patch which fixes the io module and adds helpers for testing.

http://bugs.python.org/issue21834 -- a lot of minor fixes for tests.

Following issues fix different modules and related tests:

http://bugs.python.org/issue21854 -- cookielib http://bugs.python.org/issue21838 -- ctypes http://bugs.python.org/issue21855 -- decimal http://bugs.python.org/issue21839 -- distutils http://bugs.python.org/issue21843 -- doctest http://bugs.python.org/issue21851 -- gettext http://bugs.python.org/issue21844 -- HTMLParser http://bugs.python.org/issue21850 -- httplib and SimpleHTTPServer http://bugs.python.org/issue21842 -- IDLE http://bugs.python.org/issue21853 -- inspect http://bugs.python.org/issue21848 -- logging http://bugs.python.org/issue21849 -- multiprocessing http://bugs.python.org/issue21852 -- optparse http://bugs.python.org/issue21840 -- os.path http://bugs.python.org/issue21845 -- plistlib http://bugs.python.org/issue21836 -- sqlite3 http://bugs.python.org/issue21837 -- tarfile http://bugs.python.org/issue21835 -- Tkinter http://bugs.python.org/issue21847 -- xmlrpc http://bugs.python.org/issue21841 -- xml.sax http://bugs.python.org/issue21846 -- zipfile

Most fixes are trivial and are only several lines of a code.

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.co...

---

Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/benjamin%40python.org

On 06/24/2014 12:54 PM, Ned Deily wrote:

Yes, we are committed to maintaining Python 2.7 for multiple years but that doesn't mean we have to fix every open issue or even most open issues. Any or all of the above costs may apply to any changes we make. For many of our users, the best maintenance policy for Python 2.7 would be the least change possible.

+1 We need to keep 2.7 running, but we don't need to kill ourselves doing it. If a bug has been there for a while, the affected users are probably working around it by now. ;) -- ~Ethan~

On 25 Jun 2014 07:05, "Ethan Furman" <ethan@stoneleaf.us> wrote:

On 06/24/2014 12:54 PM, Ned Deily wrote:

...

Yes, we are committed to maintaining Python 2.7 for multiple years but that doesn't mean we have to fix every open issue or even most open issues. Any or all of the above costs may apply to any changes we make. For many of our users, the best maintenance policy for Python 2.7 would be the least change possible.

+1

We need to keep 2.7 running, but we don't need to kill ourselves doing

it. If a bug has been there for a while, the affected users are probably working around it by now. ;) Aye, in this case, I'm in the "officially deprecate the feature" camp. Don't actively try to break it further, just slap a warning in the docs to say it is no longer a supported configuration. In my own personal case, I not only wasn't aware that there was still an option to turn off the Unicode support, but I also wouldn't really class a build with it turned off as still being Python. As Jim noted, there are quite a lot of APIs that don't make sense if there's no Unicode type available. Cheers, Nick.

-- ~Ethan~

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe:

https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com

25.06.14 00:03, Jim J. Jewett написав(ла):

It would be good to fix the tests (and actual library issues). Unfortunately, some of the specifically proposed changes (such as defining and using _unicode instead of unicode within python code) look to me as though they would trigger problems in the normal build (where the unicode object *does* exist, but would no longer be used).

This is recomended by MvL [1] and widely used (19 times in source code) idiom. [1] http://bugs.python.org/issue8767#msg159473

Other changes, such as the use of \x escapes, appear correct, but make the tests harder to read -- and might end up removing a test for correct unicode funtionality across different spellings.

Even if we assume that the tests are fine, and I'm just an idiot who misread them, the fact that there is any confusion means that these particular changes may be tricky enough to be for a bad tradeoff for 2.7.

It *might* work if you could make a more focused change. For example, instead of leaving the 'unicode' name unbound, provide an object that simply returns false for isinstance and raises a UnicodeError for any other method call. Even *this* might be too aggressive to 2.7, but the fact that it would only appear in the --disable-unicode builds, and would make them more similar to the regular build are points in its favor.

No, existing code use different approach. "unicode" doesn't exist, while encode/decode methods exist but are useless. If my memory doesn't fail me, there is even special explanatory comment about this historical decision somewhere. This decision was made many years ago.

Before doing that, though, please document what the --disable-unicode mode is actually *supposed* to do when interacting with byte-streams that a standard defines as UTF-8. (For example, are the changes to _xml_dumps and _xml_loads at http://bugs.python.org/file35758/multiprocessing.patch correct, or do those functions assume they get bytes as input, or should the functions raise an exception any time they are called?)

Looking more carefully, I see that there is a bug in unicode-enable build (wrong backporting from 3.x). In 2.x xmlrpclib.dumps produces already utf-8 encoded string, in 3.x xmlrpc.client.dumps produces unicode string. multiprocessing should fail with non-ascii str or unicode. Side benefit of my patches is that they expose existing errors in unicode-enable build.

2014-06-25 14:58 GMT+02:00 Serhiy Storchaka <storchaka@gmail.com>:

24.06.14 22:54, Ned Deily написав(ла):

...

Benefit: - Fixes documented feature that may be of benefit to users of Python in applications with very limited memory available, although there aren't any open issues from users requesting this (AFAIK). No benefit to the overwhelming majority of Python users, who only use Unicode-enabled builds.

Other benefit: patches exposed several bugs in code (mainly errors in backporting from 3.x).

Oh, interesting. Do you have examples of such bugs? Victor

On 26 Jun 2014 01:13, "Serhiy Storchaka" <storchaka@gmail.com> wrote:

25.06.14 16:29, Victor Stinner написав(ла):

...

2014-06-25 14:58 GMT+02:00 Serhiy Storchaka <storchaka@gmail.com>:

...

Other benefit: patches exposed several bugs in code (mainly errors in backporting from 3.x).

Oh, interesting. Do you have examples of such bugs?

In posixpath branches for unicode and str should be reversed. In multiprocessing .encode('utf-8') is applied on utf-8 encoded str (this

is unicode string in Python 3). And there is similar error in at least one other place. Tests for bytearray actually test bytes, not bytearray. That is what I remember. OK, *that* sounds like an excellent reason to keep the Unicode disabled builds functional, and make sure they stay that way with a buildbot: to help make sure we're not accidentally running afoul of the implicit interoperability between str and unicode when backporting fixes from Python 3. Helping to ensure correct handling of str values makes this capability something of benefit to *all* Python 2 users, not just those that turn off the Unicode support. It also makes it a potentially useful testing tool when assessing str/unicode handling in general. Regards, Nick.

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe:

https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com

Le 25/06/2014 19:28, Nick Coghlan a écrit :

OK, *that* sounds like an excellent reason to keep the Unicode disabled builds functional, and make sure they stay that way with a buildbot: to help make sure we're not accidentally running afoul of the implicit interoperability between str and unicode when backporting fixes from Python 3.

Helping to ensure correct handling of str values makes this capability something of benefit to *all* Python 2 users, not just those that turn off the Unicode support. It also makes it a potentially useful testing tool when assessing str/unicode handling in general.

Hmmm... From my perspective, trying to enforce unicode-disabled builds will only lower the (already low) chance that I may want to write / backport bug fixes for 2.7. For the same reason, I agree with Victor that we should ditch the threading-disabled builds. It's too much of a hassle for no actual, practical benefit. People who want a threadless unicodeless Python can install Python 1.5.2 for all I care. Regards Antoine.

2014-06-26 13:04 GMT+02:00 Antoine Pitrou <antoine@python.org>:

For the same reason, I agree with Victor that we should ditch the threading-disabled builds. It's too much of a hassle for no actual, practical benefit. People who want a threadless unicodeless Python can install Python 1.5.2 for all I care.

By the way, adding a buildbot for testing Python without thread support is not enough. The buildbot is currently broken since more than one month and nobody noticed :-p http://buildbot.python.org/all/builders/AMD64%20Fedora%20without%20threads%2... Ok, I noticed, but I consider that I spent too much time on this minor use case. I prefer to leave such task to someone else :-) Victor

On Sat, Jun 28, 2014 at 2:51 AM, Victor Stinner <victor.stinner@gmail.com> wrote:

2014-06-26 13:04 GMT+02:00 Antoine Pitrou <antoine@python.org>:

...

For the same reason, I agree with Victor that we should ditch the threading-disabled builds. It's too much of a hassle for no actual, practical benefit. People who want a threadless unicodeless Python can install Python 1.5.2 for all I care.

By the way, adding a buildbot for testing Python without thread support is not enough. The buildbot is currently broken since more than one month and nobody noticed :-p

I've opened http://bugs.python.org/issue21755 to fix the test a couple of weeks ago. --Berker

http://buildbot.python.org/all/builders/AMD64%20Fedora%20without%20threads%2...

Ok, I noticed, but I consider that I spent too much time on this minor use case. I prefer to leave such task to someone else :-)

Victor _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/berker.peksag%40gmail.com