Increased Understanding Of Unicode In Python (2.7)

I'm observing that in the program # -*- coding: utf-8 -*- words = ['artists', 'Künstler', '艺术家', 'Митець'] for word in words: print word, type(word) it is not abs

Solution 1:

Perhaps this will make it more clear:

# -*- coding: utf-8 -*-
words = ['artists', 'Künstler', '艺术家', 'Митець']
for word in words:
    print word, type(word), repr(word)
words = [u'artists', u'Künstler', u'艺术家', u'Митець']
for word in words:
    print word, type(word), repr(word)

Output:

artists <type'str'> 'artists'
Künstler <type'str'> 'K\xc3\xbcnstler'
艺术家 <type'str'> '\xe8\x89\xba\xe6\x9c\xaf\xe5\xae\xb6'
Митець <type'str'> '\xd0\x9c\xd0\xb8\xd1\x82\xd0\xb5\xd1\x86\xd1\x8c'
artists <type'unicode'> u'artists'
Künstler <type'unicode'> u'K\xfcnstler'
艺术家 <type'unicode'> u'\u827a\u672f\u5bb6'
Митець <type'unicode'> u'\u041c\u0438\u0442\u0435\u0446\u044c'

In the first case you get byte strings encoded in the declared source encoding of UTF-8. They will only display correctly on a UTF-8 terminal.

In the second case you get Unicode strings. They will display correctly on any terminal whose encoding supports the characters.

Here's how the strings display on a Windows code page 437 console, using a Python environment variable to configure Python to replace unsupported characters instead of raising the default UnicodeEncodeError exception for them:

c:\>set PYTHONIOENCODING=cp437:replace
c:\>py -2 x.py
artists <type'str'> 'artists'
K├╝nstler <type'str'> 'K\xc3\xbcnstler'
艺术家 <type'str'> '\xe8\x89\xba\xe6\x9c\xaf\xe5\xae\xb6'
Митець <type'str'> '\xd0\x9c\xd0\xb8\xd1\x82\xd0\xb5\xd1\x86\xd1\x8c'
artists <type'unicode'> u'artists'
Künstler <type'unicode'> u'K\xfcnstler'
??? <type'unicode'> u'\u827a\u672f\u5bb6'
?????? <type'unicode'> u'\u041c\u0438\u0442\u0435\u0446\u044c'

Bytes strings are mostly garbage, but Unicode strings are sensible since Chinese and Russian aren't supported by that code page.

Solution 2:

And so it appears that once coding: utf-8 is specified, all strings are encoded in Unicode. Is that true?

No. It means that byte sequences within the source code are interpreted as UTF-8. You have created bytestrings and the system is interpreting their contents naively (versus creating text with u'...').