Unicode

Unicode is a computing industry standard allowing computers to consistently represent and manipulate text expressed in most of the world‘s writing systems. Developed in tandem with the Universal Character Set standard and published in book form as The Unicode Standard, the latest version of Unicode consists of

^l     a repertoire of more than 107,000 characters covering 90 scripts,

^l     a set of code charts for visual reference,

^l     an encoding methodology and set of standard character encodings,

^l     an enumeration of character properties such as upper and lower case,

^l     a set of reference data computer files,

^l     a number of related items, such as character properties, rules for normalization, decomposition, collation, rendering, and bidirectional display order (for the correct display of text containing both right-to-left scripts, such as Arabic or Hebrew, and left-to-right scripts).

Character encoding

In computer science, the terms character encoding, character set, and sometimes character map or code page were historically synonymous, as the same standard would specify a repertoire of characters and how they were to be encoded into a stream of code units — usually with a single character per code unit. The terms now have related but distinct meanings, reflecting the efforts of standards bodies to use precise terminology when writing about and unifying many different encoding systems.^[1] Regardless, the terms are still used interchangeably, with character set being nearly ubiquitous.

Unicode encoding model

Unicode and its parallel standard, the ISO/IEC 10646 Universal Character Set together constitute a modern, unified character encoding. Rather than mapping characters directly to octets (bytes), they separately define what characters are available, their numbering, how those numbers are encoded as a series of "code units" (limited-size numbers), and finally how those units are encoded as a stream of octets. The idea behind this decomposition is to establish a universal set of characters that can be encoded in a variety of ways.^[1] To correctly describe this model one needs more precise terms than "character set" and "character encoding". The terms used in the modern model follow:^[1]

A character repertoire is the full set of abstract characters that a system supports. The repertoire may be closed, i.e. no additions are allowed without creating a new standard (as is the case with ASCII and most of the ISO-8859 series), or it may be open, allowing additions (as is the case with Unicode and to a limited extent the Windows code pages). The characters in a given repertoire reflect decisions that have been made about how to divide writing systems into linear information units. The basic variants of the Latin, Greek, and Cyrillic alphabets, can be broken down into letters, digits, punctuation, and a few special characters like the space, which can all be arranged in simple linear sequences that are displayed in the same order they are read. Even with these alphabets however diacritics pose a complication: they can be regarded either as part of a single character containing a letter and diacritic (known in modern terminology as a precomposed character), or as separate characters. The former allows a far simpler text handling system but the latter allows any letter/diacritic combination to be used in text. Other writing systems, such as Arabic and Hebrew, are represented with more complex character repertoires due to the need to accommodate things like bidirectional text and glyphs that are joined together in different ways for different situations.

A coded character set specifies how to represent a repertoire of characters using a number of non-negative integer codes called code points. For example, in a given repertoire, a character representing the capital letter "A" in the Latin alphabet might be assigned to the integer 65, the character for "B" to 66, and so on. A complete set of characters and corresponding integers is a coded character set. Multiple coded character sets may share the same repertoire; for example ISO/IEC 8859-1 and IBM code pages 037 and 500 all cover the same repertoire but map them to different codes. In a coded character set, each code point only represents one character, i.e., a coded character set is a function.

A character encoding form (CEF) specifies the conversion of a coded character set‘s integer codes into a set of limited-size integer code valuesthat facilitate storage in a system that represents numbers in binary form using a fixed number of bits (i.e. practically any computer system). For example, a system that stores numeric information in 16-bit units would only be able to directly represent integers from 0 to 65,535 in each unit, but larger integers could be represented if more than one 16-bit unit could be used. This is what a CEF accommodates: it defines a way of mapping a single code point from a range of, say, 0 to 1.4 million, to a series of one or more code values from a range of, say, 0 to 65,535.

The simplest CEF system is simply to choose large enough units that the values from the coded character set can be encoded directly (one code point to one code value). This works well for coded character sets that fit in 8 bits (as most legacy non-CJK encodings do) and reasonably well for coded character sets that fit in 16 bits (such as early versions of Unicode). However, as the size of the coded character set increases (e.g. modern Unicode requires at least 21 bits/character), this becomes less and less efficient, and it is difficult to adapt existing systems to use larger code values. Therefore, most systems working with later versions of Unicode use either UTF-8, which maps Unicode code points to variable-length sequences of octets, or UTF-16/UCS-2, which maps Unicode code points to variable-length sequences of 16-bit words.

Next, a character encoding scheme (CES) specifies how the fixed-size integer code values should be mapped into an octet sequence suitable for saving on an octet-based file system or transmitting over an octet-based network. With Unicode, a simple character encoding scheme is used in most cases, simply specifying whether the bytes for each integer should be in big-endian or little-endian order (even this isn‘t needed with UTF-8). However, there are also compound character encoding schemes, which use escape sequences to switch between several simple schemes (such as ISO/IEC 2022), and compressing schemes, which try to minimise the number of bytes used per code unit (such as SCSU, BOCU, and Punycode).

Finally, there may be a higher level protocol which supplies additional information that can be used to select the particular variant of a Unicode character, particularly where there are regional variants that have been ‘unified‘ in Unicode as the same character. An example is the XML attribute xml:lang.

The Unicode model reserves the term character map for historical systems which directly assign a sequence of characters to a sequence of bytes.^[1] Such systems include entities which IBM‘s Character Data Representation Architecture (CDRA) designates with coded character set identifiers (CCIDs) and each of which is variously called a charset, character set, code page, or CHARMAP.^[1] The term charset is also used for similar mappings by MIME and systems based on it.^[1]

Popular character encodings

ISO 8859:

• ISO 8859-1 Western Europe
• ISO 8859-2 Western and Central Europe

?  Chinese Guobiao

• GB 2312
• GBK (Microsoft Code page 936)
• GB 18030

?  Taiwan Big5 (a more famous variant is Microsoft Code page 950)

?  Hong Kong HKSCS

?  Korean

Universal Character Set

The Universal Character Set (UCS), defined by the International Standard ISO/IEC 10646, Information technology — Universal multiple-octet coded character set (UCS) (plus amendments to that standard), is a standard set of characters upon which many character encodings are based.

Mapping of Unicode character planes

The Unicode characters can be categorized in many different ways, Unicode code points can be logically divided into 17 planes, each with 65,536 (= 2¹⁶) code points, although currently only a few planes are used:

• Plane 0 (0000–FFFF): Basic Multilingual Plane (BMP). This is the plane containing most of the character assignments so far. A primary objective for the BMP is to support the unification of prior character sets as well as characters for writing systems in current use.
• Plane 1 (10000–1FFFF): Supplementary Multilingual Plane (SMP).
• Plane 2 (20000–2FFFF): Supplementary Ideographic Plane (SIP)
• Planes 3 to 13 (30000–DFFFF) are unassigned
• Plane 14 (E0000–EFFFF): Supplementary Special-purpose Plane (SSP)
• Plane 15 (F0000–FFFFF) reserved for the Private Use Area (PUA)
• Plane 16 (100000–10FFFF), reserved for the Private Use Area (PUA)

Currently, about ten percent of the potential space is used. Furthermore, ranges of characters have been tentatively blocked out for every current and ancient writing system (script) the Unicode consortium has been able to identify: (see [1]). While Unicode may eventually need to use another of the spare 11 planes for ideographic characters, other planes remain, if previously unknown scripts with tens of thousands of characters are discovered. This 21-bit limit is therefore unlikely to be reached in the near future.

The first plane (plane 0), the Basic Multilingual Plane (BMP),[Chandler1]  is where most characters have been assigned so far. The BMP contains characters for almost all modern languages, and a large number of special characters. Most of the allocated code points in the BMP are used to encode Chinese, Japanese, and Korean (CJK) characters.

(edit template)

Note: Unicode characters visualization will depend on the character support of your web browser and the fonts installed on your system.

GB18030-2000

Introduction

GB18030-2000 is a new character set standard from the PRC that specifies an extended codepage and a mapping table to Unicode.

On March 17, 2000, the Chinese government issued regulations mandating that all operating systems on non-handheld computers sold in the PRC after January 1, 2001 would have to comply with the new multibyte GB18030-2000 standard. However, the initial implementation deadline of January 1, 2001 was later postponed until September 1, 2001.

Evolution of GB18030-2000

All character set standards that originate in the PRC have designations that begin with "GB". GB is an abbreviation for Guojia Biaozhun, meaning "national standard". The GB 2312-1980 character set standard was established in 1981 to represent simplified Chinese characters. GB 2312-1980 is a coded character set that contains 7,445 characters, including 6,763 Hanzi and 682 non-Hanzi characters. With the release of ISO 10646-1/Unicode 2.1 in 1993, the PRC expressed its fundamental consent to support the combined efforts of the ISO/IEC and the Unicode Consortium through publishing a Chinese National Standard that was code- and character-compatible with ISO 10646-1/Unicode 2.1. This standard was named GB 13000.1. Whenever the ISO and the Unicode Consortium changed or revised their common standard, GB 13000.1 subsequently adopted these changes.

To accommodate all additional Hanzi characters specified in GB 13000.1 that are not included in GB 2312-1980, a new specification known as GBK was then introduced. GBK is an abbreviation for "Guojia biaozhun kuozhan", which is the Chinese for "Rules/Specifications defining the extensions of internal codes for Chinese ideograms". GBK is an extension of GB 2312-1980 and the key significant property of GBK is that it leaves the characters and codes as defined in GB 2312-1980 untouched and positions all additional characters around it. The additional characters are mainly those of the Unified Han portion of Unicode 2.1 that go beyond the character repertoire of GB 2312-1980. Thus, code and character compatibility between GBK and GB 2312-1980 is ensured while, at the same time, the complete Unicode Unified Han character set is made available. At the time when GBK was defined, other characters were added that were not available in Unicode.

GBK defines 23,940 code points containing 21,886 characters. At the same time, GBK provides mappings to the code points of Unicode 2.1. However, due to the packed code space used to define GBK, it became obvious that there was no space left for a major addition. The 1,894 code points of GBK‘s three user-defined areas were not even close to providing sufficient space for the CJK Unified Ideographs Extension A, which defines 6,582 new characters in plane 0 of Unicode, version 3.0, the Basic Multilingual Plane (BMP).

Therefore, GB18030-2000 was created as an update of GBK for Unicode 3.0 with an extension that covers all of Unicode. It is fully backward-compatible with GB 2312-1980 and GBK. The mapping table from GB18030-2000 to Unicode is backward-compatible with the mapping table from GB 2312-1980 to Unicode, however, the GBK to Unicode table has a few differences. GBK contains characters which were not defined in Unicode 2.1, but were added in later versions of Unicode.

GB18030-2000 specifies a mapping table that covers all Unicode code points and maintains compatibility of GB-encoded text with GBK and GB 2312-1980.

In graphical form, the following figure shows the space of all 64K possible 2-byte codes. Green and yellow areas are assigned GBK codepoints, red are for user-defined characters. The uncolored areas are invalid byte combinations.

从上面的两个图上可以分厂清晰的看出GB 18030、GBK 1.0、Microsoft Codepage 936、GB 2312这些编码的两个字节覆盖的取值范围。从下面的”Mapping Tables for Character Sets”中，也能看到，一个字节情况下128个gbk取值，还有后面的两个字节的情况，这里是以第一个字节为0x81为例，注意第二个字节的取值是从0x40开始的，0x8100到0x8139这个区间中是没有合法取值的，是空缺的位置。对应上面说到的“The uncolored areas are invalid byte combinations.”

Mapping Tables for Character Sets - GB2312

How to read this chart:

main

main | 81

UTF-8

UTF-8 (8-bit UCS/Unicode Transformation Format) is avariable-length character encoding for Unicode. It isable to represent any character in the Unicode standard.

UTF-16/UCS-2

In computing, UTF-16(16-bit UCS/Unicode Transformation Format) is avariable-length character encoding for Unicode, capableof encoding the entire Unicode repertoire.

UCS-2(2-byteUniversal Character Set) The UCS-2 encodingform is identical to that of UTF-16, except that it does not support surrogatepairs and therefore can only encode characters in the BMP range U+0000 throughU+FFFF.

Encoding of characters outside theBMP

Theimprovement that UTF-16 made over UCS-2 is its ability to encode characters inplanes 1–16, not just those in plane 0 (BMP). This was done by taking anunassigned portion of the 16 bit UCS-2 space, shown to scale by color here:

||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||

注意：在上面的BMP（mapping of Unicodecharacter planes）中0xD8到0xF8都是“utf-16 surrogates and private use”，这里的前面8个区域就是用来支持utf-16表示BMP以外的unicode编码用的。

UTF-16represents non-BMP characters (those from U+10000 through U+10FFFF) using apair of 16-bit words, known as a surrogate pair. First 10000₁₆is subtracted from the code point to give a 20-bit value. This is then splitinto two separate 10-bit values each of which is represented as a surrogatewith the most significant half placed in the first surrogate. To allow safe useof simple word-oriented stringprocessing, separate ranges of values are used for the two surrogates:0xD800–0xDBFF for the first, most significant surrogate (marked brown) and0xDC00-0xDFFF for the second, least significant surrogate (marked azure).

For example,the character at code point U+10000 becomes the code unit sequence 0xD8000xDC00, and the character at U+10FFFD, the upper limit of Unicode, becomesthe sequence 0xDBFF 0xDFFD.[Chandler2]  Unicode and ISO/IEC 10646 do not, and will never, assigncharacters to any of the code points in the U+D800–U+DFFF range, so anindividual code value from a surrogate pair does not ever represent acharacter.