SURVEY OF UNICODE CHARACTER PROPERTIES AND GUIDELINES

The Unicode Standard makes these specific statements about overriding properties:

Some normative behavior is default behavior; this behavior can be overridden by higher-level protocols. However, in the absence of such protocols, the behavior must be observed so as to follow the character semantics.

Disclaimer

The content of all character property tables has been verified as far as possible by the Unicode Consortium. However, the Unicode Consortium does not guarantee that the tables printed in this volume or on the CD-ROM are correct in every detail, and it is not responsible for errors that may occur either in the character property tables or in software that implements these tables. The contents of all the tables in this chapter may be superseded or augmented by information on the Unicode Web site.

For the directional types of Unicode characters, see the Unicode Character Database on the CD-ROM.

            * may behave like Ps or Pe depending on usage

Limitations. Some of the values in the General Category values are better defined than others. Letters, digits, combining marks, spaces, and formatting characters are some of the well defined values. The subdivision of the remainder into subclasses is then based on an implicit and unstated hierarchy where the less well defined categories, such as "Punctuation" and "Symbol", represent the left-overs.  The subdivision of punctuation into subtypes, such as opening and closing cannot be uniquely done (see the quotation marks) and is less detailed than needed for important implementations, such as line breaking. In addition, the assignment of Symbol cannot be made coherent, since there are instances of letters functioning as symbols (e.g., the letterlike symbols), and instances of symbols functioning as letters (e.g., some of the modifier letters). Where detailed analysis of symbols or punctuation is required, it is recommended that implementers not rely solely on General Category assignments, but consider other, more specific properties as well.

6.6 Numeric Value—Normative

Numeric value is a normative property of characters that represent numbers. This group includes characters such as fractions, subscripts, superscripts, Roman numerals, currency numerators, encircled numbers, and script-specific digits. In many traditional numbering systems, letters are used with a numeric value. Examples include Greek and Hebrew letters as well as Latin letters used in outlines (II.A.1.b). These special cases are not included here as numbers.

Decimal digits form a large subcategory of numbers consisting of those digits that can be used to form decimal-radix numbers. They include script-specific digits, not characters such as Roman numerals (1 + 5 = 15 = fifteen, but I + V = IV = four), subscripts, or super-scripts. Numbers other than decimal digits can be used in numerical expressions, but it is up to the user to determine the specialized uses.

The Unicode Standard assigns distinct codes to the forms of digits that are specific to a given script or language. Examples are the digits used with the Arabic script, Chinese numbers, or those of the Indic languages. For naming conventions, see the introduction to Section 8.2, Arabic.

Table 4 - 6 gives the numeric values of Unicode characters that can represent numbers. Some

CJK ideographs also have numeric values; those are not included in Table 4 - 6, but are discussed following the table.

CJK ideographs from the Unified Repertoire and Ordering also may have numeric values. The primary numeric ideographs are shown in Table 4 - 7 . When used to represent numbers in decimal notation, zero is represented by U+3007. Otherwise, zero is represented by U+96F6.

Ideographic accounting numbers are commonly used on checks and other financial instruments to minimize the possibilities of misinterpretation or fraud in the representation of numerical values. The set of accounting numbers varies somewhat between Japanese, Chinese, and Korean usage. Table 4 - 8 gives a fairly complete listing of the known accounting characters. Some of these characters are ideographs with other meanings pressed into service as accounting numbers; others are used only as accounting numbers.

Table 7. Primary Numeric Ideographs

U+96F6	0
U+4E00	1
U+4E8C	2
U+4E09	3
U+56DB	4
U+4E94	5
U+516D	6
U+4E03	7
U+516B	8
U+4E5D	9
U+5341	10
U+767E	100
U+5343	1,000
U+4E07	10,000
U+5104	100,000,000 (10,000 × 10,000)
U+5146	1,000,000,000,000 (10,000 × 10,000 × 10,000)

Table 8. Ideographs Used as Accounting Numbers

1	U+58F9, U+58F1, U+5F0C a
2	U+8CAE a , U+8D30 a , U+5F10 a , U+5F0D a
3	U+53C3, U+53C2, U+53C1 a , U+5F0E a
4	U+8086
5	U+4F0D
6	U+9678, U+9646
7	U+67D2 b
8	U+634C
9	U+7396
10	U+62FE
100	U+4F70 a , U+964C
1,000	U+4EDF
10,000	U+842C

a. These characters are used only as accounting numbers, and have no other meaning.
b. In Japan, U+67D2 is also pronounced urusi, meaning "lacquer," and is treated as a variant of the standard character for "lacquer" U+6F06.

6.7 Mirrored—Normative

Mirrored is a normative property of characters such as parentheses, whose images are mirrored horizontally in text that is laid out from right to left. For example, U+0028 LEFT PARENTHESIS is interpreted as opening parenthesis; in a left-to-right context it will appear as "(", while in a right-to-left context it will appear as the mirrored glyph ")". The list of mirrored characters appears in Table 4 - 9 . Note that mirroring is not limited to paired characters, but that any character with the mirrored property will need two mirrored glyphs. This requirement is necessary to render the character properly in a bidirectional context.

6.8 Unicode 1.0 Names

The Unicode 1.0 character name is an informative property of the characters defined in Version 1.0 of the Unicode Standard. The names of Unicode characters were changed in the process of merging the standard with ISO/IEC 10646. The Version 1.0 character names can be obtained from the CD-ROM accompanying the standard or from the ftp site. See also Appendix D, Changes from Unicode Version 2.0. Where the Version 1.0 character name provides additional useful information, it is listed in Chapter 14, Code Charts. For example, U+00B6 PILCROW SIGN has its Version 1.0 name, PARAGRAPH SIGN, listed for clarity.

6.9 Mathematical Property

The mathematical property is an informative property of characters that are used as operators in mathematical formulas. The mathematical property may be useful in algorithms that deal with the display of mathematical text and formulas. However, a number of these characters have multiple usages and may occur with nonmathematical semantics. For example, U+002D HYPHEN-MINUS may also be used as a hyphen—and not as a mathematical minus sign. Other characters, including some alphabetic, numeric, punctuation, spaces, arrows, and geometric shapes, are used in mathematical expressions as well, but are even more dependent on the context for their identification. The Unicode characters in the following lists have the mathematical property.

6.10 Letters and Other Useful Properties

The CD-ROM that accompanies the Unicode Standard contains data files that list other useful, informative properties of Unicode characters. The full list of those properties can be found in the data files; see, in particular, PropList.txt. This section highlights some of those properties that have a bearing on such implementation issues as parsing of identifiers. (See also Section 5.16, Identifiers.)

Letter. This informative property applies to characters that are used to write words. This group includes characters such as capital letters, small letters, ideographs, hangul, and spacing modifier letters. Combining marks generally assume the letter property of the preceding base character. For example, when searching for word boundaries, combining characters don’t break from previous letters. The letter property mappings can be obtained from the CD-ROM accompanying the standard.

Alphabetic. The alphabetic property is an informative property of the primary units of alphabets and/or syllabaries, whether combining or noncombining. Included in this group would be composite characters that are canonical equivalents to a combining character sequence of an alphabetic base character plus one or more combining characters; letter digraphs; contextual variants of alphabetic characters; ligatures of alphabetic characters; contextual variants of ligatures; modifier letters; letterlike symbols that are compatibility equivalents of single alphabetic letters; and miscellaneous letter elements. Notably, U+00AA FEMININE ORDINAL INDICATOR and U+00BA MASCULINE ORDINAL INDICATOR are simply abbreviatory forms involving a Latin letter and should be considered alphabetic rather than nonalphabetic symbols.

Ideographic. The ideographic property is an informative property of the Unified CJK Ideograph set (U+4E00..U+9FA5); the CJK Ideograph Extension A set (U+3400..U+4DB5); the CJK Compatibility Ideograph set (U+F900..U+FA2D); U+3007 IDEOGRAPHIC NUMBER ZERO; U+3006 IDEOGRAPHIC CLOSING MARK; and the Hangzhou-style numerals (U+3021..U+3029, U+3038..U+303A).

6.11 East Asian Width

East Asian Width is a limited property, classifying characters based on their width in East Asian legacy implementations. While presented as a closed enumeration, the EAW value "Neutral" essentially is the 'does not apply' value.

6.12 ISO comment

ISO/IEC 10646 provides for a comment field in the character name itself, while Unicode uses a more extensive set of annotations. Combining the information in this field with the Unicode Character name, provides the complete name as it appears in ISO/IEC 10646. 

6.13 Sort Weights

<see TR 10>

6.14 Line Breaking

<see TR 14>

6.15 Decompositions

<see TR 15>

6.16 Special Property

Table 3-9 in [Unicode] lists those characters that have particular properties. Their properties are described individually in chapter 13 of [Unicode]. These characters can be grouped as follows.

• Line Boundary Control
• Hyphenation Control
• Fraction Formatting
• Special Behavior with nonspacing marks
• Double nonspacing marks
• Joining Control
• Bidirectional Ordering Control
• Alternate Formatting (deprecated)
• Syriac abbreviation
• Indic dead-character formation
• Mongolian variation selectors
• Ideographic variation indication
• Ideographic description
• Interlinear Annotation
• Object Replacement
• Code Conversion Fallback
• Byte Order Signature

• Math Variant
• Combining Grapheme Joiner
• Word joiner
• Ligature formation control
• Musical scoping
• Invisible operators

6.21 Case Mapping

<see TR21>

6.22 Character Mappings

<see TR22>

6.24 Script Name

<see UTR 24: Script Names>

7. Updating Properties and Extending the Standard 

7.1 Updating Properties

Updates to the Unicode Character Database can be required for three reasons

1. To cover new characters added to the Unicode Standard
2. To add new properties
3. To change the assigned values for a property for some characters

Changing a characters property assignment invalidates existing implementations and is therefore something that is done judiciously and with great care when there is no better alternative.

• Unless explicitly stated otherwise, The Unicode Consortium may change the properties of characters.

  The consortium will endeavor to keep the values of all character properties as stable as possible, but some circumstances may arise that require changing them. In particular, as Unicode encodes less-well documented scripts (such as for minority languages in Thailand) the exact character properties and behavior may not be known at the time the script is first encoded.
  

7.2 Guarantees

For some properties, some of the following aspects are guaranteed to be invariant.

•  stability of assignment 
•  stability of set of values for a property
•  stability of file formats
•  stability of relation to another property

The status of a property as normative does not imply a stability guarantee.

• Once a character is encoded, its code position and name are immutable properties.

  Mistakes in naming are noted in the nameslist in a note or by using an alias, but the formal name remains unchanged.

• Once a character is encoded, its canonical combining class and decomposition (canonical or combining) are stable with respect to normalization.

  If a string contains only characters from a given version of the Unicode Standard (say Unicode 3.1), and it is put into a normalized form in accordance with that version of Unicode, then it will be in normalized form when according to any past or future versions of Unicode.

  Note: If an implementation normalizes a string that contains characters that are not assigned in the version of Unicode that it supports, that string may not be assessed as being in normalized form according to a future version of Unicode. For example, suppose that a Unicode 3.0 program normalizes a string that contains new Unicode 3.1 characters. That string may not be normalized according to Unicode 3.1.
  

• The General Category and Bidi Category are closed enumerations.

  In other words they will not be further subdivided.

• The structure of certain property values in the Unicode Character Database will not be changed:

  Further description of these is provided in described in UnicodeData.html:
  

  • Combining classes are limited to the values 0 to 255.
  • All characters other than those of General Category M* have the combining class 0.
  • Canonical and Compatibility mappings are always in canonical order, and the resulting recursive decomposition will also be in canonical order.
  • Canonical mappings are always limited either to a single value or to a pair. The second character in the pair cannot itself have a canonical mapping.
    

8. Special Property Values

8.1 N/A Value

Limited properties apply to only a subset of characters. Where these properties are implemented as a partition (required property) the characters to which the property does not apply is given a special value denoting that the property does not apply.

8.2 Default Value

Implementations often need specific properties for all code points, including those that are unassigned. To meet this need, the Unicode standard assigns default properties to ranges of unassigned code points.

All implementations of the Unicode Standard should endeavor to handle additions to the character repertoire gracefully. In some cases this may require that an implementation attempts to 'anticipate' likely property values for Code points for which characters have not yet been defined, but where surrounding characters exist that make it probable that similar characters will be assigned to the Code point in question.

There are three strategies

1. Rely on the recommendation from The Unicode Consortium. For example, for the Bidirectional Class, the Unicode Consortium has published recommended default values for all code points.
2. Treat the unassigned areas of a block as if they had property values common to other characters of the block. A variation of this scheme bridges 'holes' in the allocation by using the property values for the characters bracketing the hole.
3. Give unassigned code location a different default property that will result in graceful, if not completely correct behavior if encoded characters are later encountered at that location.

Each of these strategies has advantages and drawbacks, and none can guarantee that the behavior of an implementation that is conformant to a prior version of the Unicode Standard will support characters added in a later version of the Unicode Standard in precisely the same way as an implementation that is conformant to the later version. The most that can be hoped for, is that the earlier implementation will behave gracefully in such circumstances.

Default values are temporary: they will be superseded by final assignments, once characters are assigned to a given code point.

8.3 Undetermined Property Values

For many archaic scripts (as well as for not yet fully implemented modern ones) essential characteristics of many characters may not be knowable at the time of their publication. In these cases the proper assignments of property values for newly encoded characters cannot be reliably determined at the time the characters are first added to the Unicode Standard, or for a new property, when the property is first added to the Unicode Character Database. In these cases, and where the property is a required property, it will be given a value of 'undetermined', or 'unknown at time of publication'.

8.4 Preliminary Property Assignments

Sometimes, a determination and assignment of property values can be made, but the information on which it was based may be incomplete or preliminary. In such cases, the property value may be changed when better information becomes available. Currently, there is no machine readable way to provide information about the confidence of a property assignment; however, the text of the Standard or a Technical Report defining the property may provide general indications of preliminary status of property assignments where they are known.

9. Working with properties

[The text in this section is very preliminary].

There are two main issues in working with properties

• Efficient storage and access to the property information
• Interpretation of the character property and implementation of character behavior in context

9.1 Efficient storage and access to property information

The Unicode Standard provides detailed information on character properties (see Chapter 4, Character Properties, and the Unicode Character Database on the accompanying CD-ROM).

These properties can be used by implementers to implement a variety of low-level processes. Fully language-aware and higher-level processes will need additional information.

A two-stage table, as described in Section 5.1, Transcoding to Other Standards, can also be used to handle mapping to character properties or other information indexed by character code. For example, the data from the Unicode Character Database on the accompanying CD-ROM can be represented in memory very efficiently as a set of two-stage tables.

Individual properties are common to large sets of characters and therefore lend themselves to implementations using the shared blocks.

Many popular implementations are influenced by the POSIX model, which provides functions for separate properties, such as isalpha, isdigit, and so on. Implementers of Unicode-based systems and internationalization libraries need to take care to extend these concepts to the full set of Unicode characters correctly.

In Unicode-encoded text, combining characters participate fully. In addition to providing callers with information about which characters have the combining property, implementers and writers of language standards need to provide for the fact that combining characters assume the property of the preceding base character (see also Section 3.5, Combination, and Section 5.16, Identifiers). Other important properties, such as sort weights, may also depend on a character’s context.

Because the Unicode Standard provides such a rich set of properties, implementers will find it useful to allow access to several properties at a time, possibly returning a string of bit-fields, one bit-field per character in the input string.

In the past, many existing standards, such as the C language standard, assumed very minimalist "portable character sets" and geared their functions to operations on such sets. As the Unicode encoding itself is increasingly becoming the portable character set, implementers are advised to distinguish between historical limitations and true requirements when implementing specifications for particular text processes.

Tables require space. Even small character sets often map to characters from several different blocks in the Unicode Standard, and thus may contain up to 64K entries in at least one direction. Several techniques exist to reduce the memory space requirements for mapping tables. Such techniques apply not only to transcoding tables, but also to many other tables needed to implement the Unicode Standard, including character property data, collation tables, and glyph selection tables.

Flat Tables. If diskspace is not at issue, virtual memory architectures yield acceptable working set sizes even for flat tables because frequency of usage among characters differs widely and even small character sets contain many infrequently used characters. In addition, data intended to be mapped into a given character set generally does not contain characters from all blocks of the Unicode Standard (usually, only a few blocks at a time need to be transcoded to a given character set). This situation leaves large sections of the 64K-sized reverse mapping tables (containing the default character, or unmappable character entry) unused—and therefore paged to disk.

Ranges. It may be tempting to "optimize" these tables for space by providing elaborate pro-visions for nested ranges or similar devices. This practice leads to unnecessary performance penalties on modern, highly pipelined processor architectures because of branch penalties.

A faster solution is to use an optimized two-stage table, which can be coded without any test or branch instructions. Hash tables can also be used for space optimization, although they are not as fast as multistage tables.

Two-Stage Tables. Two-stage (high-byte) tables are a commonly employed mechanism to reduce table size (see Figure 5-1). They use an array of 256 pointers and a default value. If a pointer is NULL, the returned value is the default. Otherwise, the pointer references a block of 256 values.

Optimized Two-Stage Table. Wherever any blocks are identical, the pointers just point to the same block. For transcoding tables, this case occurs generally for a block containing only mappings to the "default" or "unmappable" character. Instead of using NULL pointers and a default value, one "shared" block of 256 default entries is created. This block is pointed to by all first-stage table entries, for which no character value can be mapped. By avoiding tests and branches, this strategy provides access time that approaches the simple array access, but at a great savings in storage.

Given an arbitrary 64K table, it is a simple matter to write a small utility that can calculate the optimal number of stages and their width.

[additional guidelines, TBD]

10. Data Management and Distribution

10.1 Versions of the Unicode Character Database

The Unicode Character Database is provided as a collection of flat text files, as described in [UnicodeCharacterDatabase]. Each version of the database has its own, numbered directory on the Unicode ftp site [ftp-http]. In each versioned directory, the filenames of all files carry their version number in the file names. The latest version of the database is replicated in the directory named UNIDATA [Unidata], which uses constant file names (i.e. without version number). It is thus possible to reference both a specific version as well as the latest version of a database file (or the database itself) without need to update the links.

For more information about the versions of the Unicode Standard see [UnicodeVersions].

10.2 File Syntax Conventions

All files need a (formal) syntax description at least as detailed as namelist.txt has - it's probably useful to set up three or four templates for this now, so that we don't get new formats for each proposed property.

The majority of plain text files in the Unicode Character Database use this general format:

LINE := COMMENT
             CODE_RANGE ( ";" VALUE) + { COMMENT }

VALUE := file specific, often a two letter abbreviation or a small number

COMMENT := '#' arbitrary text

CODE_RANGE = CODE { ".."CODE }

CODE := HHHH { { H } H }

H := '0' | '1' | '2' | '3' | '4' | '5' | '6' | '7' | '8' | '9' | 'A' | 'B' | 'C' | 'D' | 'E' | 'F'

By convention, the file starts with a headerI, i.e. several comment lines identifying title, version, date, author, purpose and content of the file, as well as the permissible values and meaning for each field. By convention, following the last field in each line, a comment contains the character name. This comment should be ignored in machine processing of the file.

An XML version of the Unicode Character Database is under preparation.

10.3 Beta Versions of the Character Database

Whenever an update of the database is being developed, a beta version may be released in a directory whose name contains the word 'beta' in the directory name and in all filenames contained therein. This beta directory will be provided solely for review purposes and will not be maintained after the end of the beta period.

11. References

[ArabicShaping]

Data file <ftp://ftp.unicode.org/Public/UNIDATA/ArabicShaping.txt>

[Bidirectional Algorithm]

Mark Davis, Unicode Standard Annex #9: The Bidirectional Algorithm, <http://www.unicode.org/unicode/reports/tr9>

[CaseFolding]

Data file <ftp://ftp.unicode.org/Public/UNIDATA/CaseFolding.txt>

[Case Mapping] 

Mark Davis, Unicode Technical Report #21: Case Mapping, <http://www.unicode.org/unicode/reports/tr21>

[Character Mapping Tables]

Mark Davis, Unicode Technical Report #22: Character Mapping Tables, <http://www.unicode.org/unicode/reports/tr22>

[Collation]

Mark Davis, Unicode Technical Report #10: Collation, <http://www.unicode.org/unicode/reports/tr10>

[EastAsianWidth]

Data file <ftp://ftp.unicode.org/Public/UNIDATA/EastAsianWidth.txt>

[East Asian Width] 

Asmus Freytag, Unicode Standard Annex #11, East Asian Width, <http://www.unicode.org/unicode/reports/tr11>

[Jamo]

Data file <ftp://ftp.unicode.org/Public/UNIDATA/Jamo.txt>

[LineBreak]

Data file <ftp://ftp.unicode.org/Public/UNIDATA/LineBreak.txt>

[Line Breaking]

Asmus Freytag, Unicode Standard Annex #14: Line Breaking Properties, <http://www.unicode.org/unicode/reports/tr14>

[NamesList]

Data file <ftp://ftp.unicode.org/Public/UNIDATA/NamesList.txt>

[NamesList-Format]

Readme file <ftp://ftp.unicode.org/Public/UNIDATA/NamesList.html>

[SpecialCasing]

Data file <ftp://ftp.unicode.org/Public/UNIDATA/SpecialCasing.txt>

[Unicode]

The Unicode Standard, Version 3.0, Addison Wesley Longman, 2000.

[UnicodeCharacterDatabase]

Readme file, <ftp://ftp.unicode.org/Public/UNIDATA/UnicodeCharacterDatabase.html>

[UnicodeData]

Data file <ftp://ftp.unicode.org/Public/UNIDATA/UnicodeData.txt>

[UnicodeData-Format]

Readme file <ftp://ftp.unicode.org/Public/UNIDATA/UnicodeData.html>

[UnicodeVersions]

Versions of the Unicode Standard <http://www.unicode.org/unicode/standard/versions>

Acknowledgements

The author wishes to thank Ken Whistler and Mark Davis for his insightful comments.

Changes from previous drafts

Changes form second working draft:  None, this is the first initial draft submitted to UTC.

Changes from third working draft: Reordered according to UTC feedback. Added information about maintenance, extended discussion on updates.

Changes from fourth working draft: Reworded the definitions based on feedback from Mark Davis. Some other minor changes.

Changes from fifth working draft: Added a few definitions. Cleaned up some of the text.

Changes from sixth working draft: Retargeted according to UTC feedback.

Changes from sevent working draft:

Changes from eighth working draft: incorporating individual feedback

Changes from ninth working draft: removing redundant text, word smithing

Copyright © 2000-2001 Unicode, Inc. All Rights Reserved. The Unicode Consortium makes no expressed or implied warranty of any kind, and assumes no liability for errors or omissions. No liability is assumed for incidental and consequential damages in connection with or arising out of the use of the information or programs contained or accompanying this technical report.

Unicode and the Unicode logo are trademarks of Unicode, Inc., and are registered in some jurisdictions.