1.1. 问题 Problem

You need to deal with data that doesn‘t fit in the ASCII character set.

你需要处理不适合用ASCII字符集表示的数据.

1.2. 解决 Solution

Unicode strings can be encoded in plain strings in a variety of ways, according to whichever encoding you choose:

Unicode字符串可以用多种方式编码为普通字符串, 依照你所选择的编码(encoding):

   1 #将Unicode转换成普通的Python字符串:"编码(encode)"
   2 unicodestring = u"Hello world"
   3 utf8string = unicodestring.encode("utf-8")
   4 asciistring = unicodestring.encode("ascii")
   5 isostring = unicodestring.encode("ISO-8859-1")
   6 utf16string = unicodestring.encode("utf-16")
   7 
   8 
   9 #将普通的Python字符串转换成Unicode: "解码(decode)"
  10 plainstring1 = unicode(utf8string, "utf-8")
  11 plainstring2 = unicode(asciistring, "ascii")
  12 plainstring3 = unicode(isostring, "ISO-8859-1")
  13 plainstring4 = unicode(utf16string, "utf-16")
  14 
  15 assert plainstring1==plainstring2==plainstring3==plainstring4

1.3. 讨论 Discussion

If you find yourself dealing with text that contains non-ASCII characters, you have to learn about Unicode梬hat it is, how it works, and how Python uses it.

如果你发现自己正在处理包含非ASCII码字符的文本, 你必须学习Unicode，关于它是什么,如何工作，而且Python如何使用它。

Unicode is a big topic.Luckily, you don‘t needto know everything about Unicode to be able to solve real-worldproblems with it: a few basic bits of knowledge are enough.First,you must understand the difference between bytes and characters.Inolder, ASCII-centric languages and environments, bytes andcharacters are treated as the same thing.Since a byte can hold upto 256 values, these environments are limited to 256characters.Unicode, on the other hand, has tens of thousands ofcharacters.That means that each Unicode character takes more thanone byte, so you need to make the distinction between charactersand bytes.

Unicode是一个大的主题。幸运地，你并不需要知道关于Unicode码的每件事，就能够用它解决真实世界的问题:一些基本知识就够了。首先，你得了解在字节和字符之间的不同。原先，在以ASCII码为中心的语言和环境中，字节和字符被当做相同的事物。由于一个字节只能有256个值，这些环境就受限为只支持256个字符。Unicode码，另一方面，有数万个字符,那意谓着每个Unicode字符占用多个字节，因此，你需要在字符和字节之间作出区别。

Standard Python strings are really byte strings,and a Python character is really a byte.Other terms for thestandard Python type are "8-bit string" and "plain string.",In thisrecipe we will call them byte strings, to remind you of theirbyte-orientedness.

标准的Python字符串确实是字节字符串，而且一个Python字符真的是一个字节。换个术语,标准的Python字符串类型的是 "8位字符串(8-bit string)"和"普通字符串(plainstring)". 在这一份配方中我们把它们称作是字节串(byte strings),并记住它们是基于字节的。

Conversely, a Python Unicode character is anabstract object big enough to hold the character, analogous toPython‘s long integers.You don‘t have to worry about the internalrepresentation;the representation of Unicode characters becomes anissue only when you are trying to send them to some byte-orientedfunction, such as the write method for files or the send method fornetwork sockets.At that point, you must choose how to represent thecharacters as bytes.Converting from Unicode to a byte string iscalled encoding the string.Similarly, when you load Unicode stringsfrom a file, socket, or other byte-oriented object, you need todecode the strings from bytes to characters.

相反地，一个PythonUnicode码字符是一个大到足够支持(Unicode)字符的一个抽象对象,类似于Python中的长整数。 你不必要为内在的表示担忧;只有当你正在尝试把它们传递给给一些基于字节的函数的时候，Unicode字符的表示变成一个议题,比如文件的write方法或网络套接字的send方法。那时，你必须要选择该如何表示这些(Unicode)字符为字节。从Unicode码到字节串的转换被叫做编码。同样地，当你从文件，套接字或其他的基于字节的对象中装入一个Unicode字符串的时候,你需要把字节串解码为(Unicode)字符。

There are many ways of converting Unicodeobjects to byte strings, each of which is called an encoding.For avariety of historical, political, and technical reasons, there isno one "right" encoding.Every encoding has a case-insensitive name,and that name is passed to the decode method as a parameter. Hereare a few you should know about:

将Unicode码对象转换成字节串有许多方法,每个被称为一个编码(encoding)。由于多种历史的，政治上的，和技术上的原因,没有一个"正确的"编码。每个编码有一个大小写无关的名字，而且那一个名字被作为一个叁数传给解码方法。这里是一些你应该知道的:

• The UTF-8 encoding can handle any Unicode character.It is alsobackward compatible with ASCII, so a pure ASCII file can also beconsidered a UTF-8 file, and a UTF-8 file that happens to use onlyASCII characters is identical to an ASCII file with the samecharacters.This property makes UTF-8 very backward-compatible,especially with older Unix tools.UTF-8 is far and away the dominantencoding on Unix.It‘s primary weakness is that it is fairlyinefficient for Eastern texts.
• UTF-8编码能处理任何的Unicode字符。它也是与ASCII码向后兼容的，因此一个纯粹的ASCII码文件也能被考虑为一个UTF-8文件，而且一个碰巧只使用ASCII码字符的 UTF-8文件和拥有同样字符的ASCII码文件是相同的。这个特性使得UTF-8的向后兼容性非常好,尤其使用较旧的Unix工具时。UTF-8 无疑地是在 Unix上的占优势的编码。它主要的弱点是对东方文字是非常低效的。
• The UTF-16 encoding is favored by Microsoftoperating systems and the Java environment.It is less efficient forWestern languages but more efficient for Eastern ones.A variant ofUTF-16 is sometimes known as UCS-2.
• UTF-16编码在微软的操作系统和Java环境下受到偏爱。它对西方语言是比较低效,但对于东方语言是更有效率的。一个UTF-16 的变体有时叫作UCS-2 。
• The ISO-8859 series of encodings are 256-characterASCII supersets.They cannot support all of the Unicodecharacters;they can support only some particular language or familyof languages.ISO-8859-1, also known as Latin-1, covers most WesternEuropean and African languages, but not Arabic.ISO-8859-2, alsoknown as Latin-2,covers many Eastern European languages such asHungarian and Polish.
• ISO-8859编码系列是256个字符的ASCII码的超集。他们不能够支援所有的Unicode码字符;他们只能支援一些特别的语言或语言家族。ISO-8859-1,也既Latin-1,包括大多数的西欧和非洲语言,但是不含阿拉伯语。ISO-8859-2,也既Latin-2,包括许多东欧的语言,像是匈牙利语和波兰语。

If you want to be able to encode all Unicodecharacters, you probably want to use UTF-8.You will probably needto deal with the other encodings only when you are handed data inthose encodings created by some other application.

如果你想要能够编码所有的Unicode码字符,你或许想要使用UTF-8。只有当你需要处理那些由其他应用产生的其它编码的数据时，你或许才需要处理其他编码。