Note: Strictly speaking, it makes no sense to talk of narrow and wide for neutral characters, but since for all practical purposes they behave like Na, they are treated as narrow characters (the same as Na) under the recommendations below.
| Revision | 4 |
| Authors | Asmus Freytag |
| Date | August 20, 1999 |
| This Version | http://www.unicode.org/unicode/reports/tr11-4 |
| Previous Version | http://www.unicode.org/unicode/reports/tr11-3 |
| Latest Version | http://www.unicode.org/unicode/reports/tr11 |
Summary
This report presents the specifications of a informative property for Unicode characters that is useful when interoperating with East Asian Legacy character sets.
Status of this document
This document contains informative material which has been considered and approved by the Unicode Technical Committee for publication as a Technical Report and as part of the Unicode Standard, Version 3.0 (forthcoming). Any reference to version 3.0 of the Unicode Standard automatically includes this technical report. The content of all technical reports must be understood in the context of the appropriate version of the Unicode Standard. References in this technical report to sections of the Unicode Standard refer to the Unicode Standard, Version 3.0. See http://www.unicode.org/unicode/standard/versions/ for more information. This technical report may undergo further editorial work before the final release of the Unicode Standard, Version 3.0. Please mail corrigenda and other comments to the author.
Contents
§1 Overview
§2 Scope
§3 Description
§3 Definitions
§4 Conformance
§6 Recommendation
§7 Composition Exclusion Table
§8 Background
§8 Acknowledgements
§9 Changes from previous revisions
In mixed-width, East Asian, legacy encodings there is a concept of an inherent width of a character. For a fixed pitch font, this width translates to a display width of either one half or a whole unit width. A common name for this unit width is "Em". It is customarily the height of the letter 'M', but since in East Asian fonts the standard character cell is square, it is the same as the unit width.
NOTE:
the character width for a fixed pitch Latin font like Courier is generally 3/5 of an em.
Layout and line breaking (to cite only two examples) in an East Asian context show systematic variations depending on the value of the East-Asian Width property (even for non-fixed pitch fonts). Further, the same information is useful in creating correct transcoding tables for East Asian character sets.
The East Asian Width property provides a useful concept for implementations that
This Unicode Technical Report does not provide rules or specifications of how this property might be used in font design or line layout, since, while a useful property for this purpose, it is only one of several character properties that would need to be considered.
By convention, 1/2 Em wide characters of East Asian legacy encodings are called "half-width" (or hankaku characters in Japanese), the others are called correspondingly "full-width" (or zenkaku) characters. Legacy encodings often use a single byte for the half-width characters and two bytes for the full-width characters. In the Unicode Standard, no such distinction is made, but understanding the distinction is often necessary when interchanging data with legacy systems, especially when fixed size buffers are involved.
Some character blocks in the compatibility zone contain characters that are explicitly marked "half-width" and "full-width" in their character name but for all other characters the width property must be implicitly derived. Some characters behave differently in East Asian context than in non-East Asian content. Their default width property is considered ambiguous and needs to be resolved into an actual width property based on context.
This technical report assigns to each Unicode character one of the six values Ambiguous, Full Width, Half Width, Narrow, Wide, or Not East Asian Neutral (defined below) as its default width property. For any given operation, these six default properties resolve into only two property values narrow and wide, depending on context.
East Asian Width - in the context of interoperating with East Asian legacy character encodings and implementing East Asian typography, character width is an abstract concept. It can take on two values, narrow and wide. In legacy implementations, there is often a corresponding difference in encoding length (one or two bytes) as well as a difference in displayed width. However, the actual display width of a glyph is given by the font and may be further adjusted by layout. An important class of fixed width legacy fonts contains glyphs of just two widths with the wider glyphs twice as wide as the narrower glyphs.
East Asian FullWidth (F) - characters that are defined as FULL WIDTH and therefore are compatibility equivalents of implicitly narrow but unmarked characters elsewhere in the Unicode Standard.
East Asian Half-width (H) - characters that are explicitly defined as HALF WIDTH in the Unicode Standard and therefore are compatibility characters of implicitly wide, but unmarked characters elsewhere in the Unicode Standard.
East Asian Wide (W) - characters that are implicitly wide (such as the Unified Han Ideographs or Squared Katakana Symbols) because they occur only in the context of East Asian typography where they are wide characters.
East Asian Ambiguous (A) - characters that occur in East Asian legacy character sets as wide characters, but are displayed as narrow (i.e. normal-width) characters in their own local or non-East Asian usage (Examples are the Greek and Cyrillic alphabet found in East Asian character sets, but also some of the mathematical symbols). Ambiguous characters require context to resolve their width. Private Use characters are considered ambiguous, since additional information is required to know whether they should be treated as wide or narrow.
Note: Because East Asian legacy character sets do not always include complete case pairs of Latin characters, two members of a pair may have different East Asian Width properties:
Ambiguous: 01D4 LATIN SMALL LETTER U WITH CARON NEA Neutral: 01D3 LATIN CAPITAL LETTER U WITH CARON
East Asian Narrow (Na) - characters that are implicitly narrow, since they have explicit full-width clones (all of ASCII is an example).
Note: These are characters are marked with (Na) and not (H) in the data table, because it is useful to distinguish characters explicitly defined as half-width from other characters that have a full-width equivalent. In particular, half-width punctuation behaves in some important ways like ideographic punctuation.
Not East Asian (Neutral) - all other characters. Neutral characters do not occur in legacy East Asian character sets. By extension, they also do not occur in East Asian typography. For example, there is no traditional Japanese way of typesetting Devanagari.
Note: Strictly speaking, it makes no sense to talk of narrow and wide for neutral characters, but since for all practical purposes they behave like Na, they are treated as narrow characters (the same as Na) under the recommendations below.
Figure 1: Venn diagram showing the set relations for the five of the six categories.
In a broad sense, wide characters include W, F, and A (when in EA context), while narrow characters include N, Na, H, and A (when not in EA context).
When converting a DBCS mixed-width encoding to and from Unicode, the full-width characters in such a mixed-width encoding are mapped to the full-width compatibility characters in the FFxx block, whereas the corresponding half-width characters are mapped to ordinary Unicode characters (e.g. ASCII in U+0021..U+007E, plus a few other scattered characters).
In the context of interoperability with DBCS character encodings, that restricted set of Unicode characters in the General Scripts area can be construed as half-width, rather than full-width. (This applies only to the restricted set of characters which can be paired with the full-width compatibility characters.)
In the context of interoperability with DBCS character encodings, all other Unicode characters which are not explicitly marked as half-width can be construed as full-width.
In any other context, Unicode characters not explicitly marked as being either full-width or half-width compatibility forms should be construed as neutral as to half-width versus full-width status.
Seen in this light, the "half-width" and "full-width" properties are not unitary character properties in the same sense as "space" or "combining" or "alphabetic". They are, instead, relational properties of a pair of characters, one of which is explicitly encoded as a half-width or full-width form for compatibility in mapping to DBCS mixed-width character encodings.
What is "full-width" by default today could in theory become "half-width" tomorrow by the introduction of another character on the SBCS part of a mixed-width code page somewhere, requiring the introduction of another full-width compatibility character to complete the mapping. Since the single byte part of mixed-width character sets is limited, there are not going to be many candidates and neither UTC and WG2 have any intention to add additional compatibility characters for this purpose.
Ambiguous width characters are all those characters that can occur as full-width characters in any of a number of East Asian legacy character encodings. They have a 'resolved' width of either narrow or wide depending on the context of their use. If they are not used in context of the specific legacy encoding they belong to, their width resolves to narrow. Otherwise it resolves to full-width or half-width. The context includes extra information such as the source codepage, font information, or language identification. For example, Greek characters resolve to narrow when used with a standard Greek font, since there is no East Asian legacy context. Private use character codes and the replacement character have ambiguous width, since they may stand in for characters of any width.
East Asian Width is an informative character property.
When interchanging data
When processing or displaying data
The classifications presented here are based on the most widely used mixed-width legacy character sets in use in East Asia as of this writing. In particular, the assignment of the neutral or ambiguous categories depend on the contents of these character sets. For example, an implementation that knows a-priori, that it only needs to interchange data with the Japanese Shift-JIS character set, but not other East Asian character sets, could reduce the number of characters in the ambiguous classification to those actually encoded in Shift-JIS. Or such a reduction could be done implicitly at runtime in the context of interoperating with Shift-JIS fonts or data sources. Conversely, if additional character sets are created and widely adopted for legacy purposes, more characters would need to be classified as ambiguous.
All unassigned characters are by default classified as non-East Asian neutral, except for the range U-00020000 to U-0002FFFD, since all code positions from U-0002000 to U-0002FFFD are intended for CJK ideographs (W). All Private use characters are by default classified as ambiguous, since their definition depends on context. This includes surrogate pairs.
Combining marks have been classified and are given a property assignment based on their typical applicability. For example combining marks typically applied to characters of class N, Na or W are classified as A. Combining marks for purely non-East Asian scripts are marked as N, and non-spacing marks used only with wide characters are given a W.
Particularly non-spacing marks do not possess actual advance width, therefore, even when displaying combining marks, the East Asian Width property cannot be related to the advance width of these characters. However, it could conceivably be related to the width of the glyph image or the encoding length in a legacy encoding. Even more so than for other characters, East Asian width is an abstract concept for non-spacing marks. It is often convenient to treat all characters with general category Mn (non-spacing marks) as if they had the East Asian Width property of their base character.
This information is available as the file EastAsianWidth.txt in the Unicode Character Database. This is a tab-delimited three column plain text file, with code position, East Asian Width designator and character name (for reference purpose only). The compact way of listing the ideographic, Hangul, Surrogate and Private use ranges is the same as in UnicodeData.txt
ISO 10646 is silent on the terms "half-width" and "full-width" except to say that the characters so named are provided for compatibility.
The Unicode Standard, Version 2.0 stated (p. 6-130):
In the context of conversion to and from such mixed-width encodings, all characters in the General Scripts area [i.e. 0000-1FFF] should be construed as half-width (hankaku) characters.
This sentence, as it stood, was misleading in that it implies that everything in the range U+0000..U+1FFF is half-width.
All characters in the CJK Phonetics and Symbols area [i.e. 3000-33FF] and the Unified CJK Ideograph area [i.e. 4E00-9FFF], along with the characters in the CJK Compatibility Ideographs [i.e. F900-FAFF], CJK Compatibility Forms [i.e. FE30-FE4F], and Small Form Variants blocks [i.e. FE50-FE6F], should be construed as full-width (zenkaku) characters.
This was correct, with one exception, 303F IDEOGRAPHIC HALF FILL SPACE
Other Compatibility Area [i.e. F900-FFFF] characters outside of the current block should be construed as half-width characters. The characters of the Symbols Area are neutral regarding their width semantics.
Like the first, this sentence, this was misleading in that it fails to account for the ambiguous width property of many symbols.
It should clearly be noted that statements made in the Unicode Standard, Version 2.1 in Chapter 6 (Character Block Descriptions) did not have normative status. Chapters 3, 4, and 7 (Charts) had normative status. The rest of the book, including Chapter 6 is provided to give as much information as possible to help people understand and implement the characters correctly. But it is dangerous to make legalistic arguments based on the text of Chapter 6, since there is rather large leeway for the editors of the Unicode Standard to modify and augment such explanatory text as new issues arise or old ones require more clarification.
The intent of the text in 2.1 was not yet to create a property but to account for the fact that there are full-width forms encoded in the ranges U+FF01..U+FF5E and U+FFE0..U+FFE6.
Unicode 3.0 formally introduces East Asian Width as an informative character property. The discussion of this issue has moved to section 10.3 Katakana, and defers to this Technical Report for details. The data file with the classifications is now one of the contributory files the Unicode Character Database. See http://www.unicode.org/unicode/standard/versions for more information on versions of this standard and UnicodeCharacterDatabase.html on the Unicode Character Database for more information on the Unicode Character Database.
Michel Suignard provided extensive input into the analysis and source material for the detail assignments of these properties.
First draft technical report version. Extensive formatting to fit the template. Split Wide into Wide and FullWidth to capture the characters with explicit FullWidth characteristics.
First Technical Report Version. Remove list of 'unassigned' characters. Add some informative text and make other editorial changes requested at UTC meeting #78.
Second Technical Report Version. Added UTF-8 and names annotations to the table. Minor wording changes. HTML fixes.
Third Technical Report Version: Added the classifications for new characters in Unicode 3.0. Changed classification of PUA to A.Moved the classifications into EastAsianWidth.txt in the Unicode Character Database. Minor wording changes, including some more explanatory text on private use characters and surrogates.
Fourth Tecnical Report Version: Slight rewording of the definition language as well as additional notes to clarify the terms. Better match to the data file in that the definition of (W) no longer implies that it is a superset of (F). Fix the status language to mark inclusion into Unicode 3.0. Add explanation on combining marks. No change to the data file.
Copyright © 1998-1999 Unicode, Inc. All Rights Reserved. The Unicode Consortium makes no expressed or implied warranty of any kind, and assumes no liability for errors or omissions. No liability is assumed for incidental and consequential damages in connection with or arising out of the use of the information or programs contained or accompanying this technical report.
Unicode and the Unicode logo are trademarks of Unicode, Inc., and are registered in some jurisdictions.
Unicode Home Page: http://www.unicode.org
Unicode Technical Reports: http://www.unicode.org/unicode/reports/techreports.html