[Unicode]  Frequently Asked Questions Home | Site Map | Search

Characters and Combining Marks

Q: Does "text element" mean the same as "combining character sequence"?

A: No, this is a common misperception. A text element just means any sequence of characters that are treated as a unit by some process. A combining character sequence is a base character followed by any number of combining characters. It is one type of a text element, but words and sentences are also examples of text elements. [MD]

Q: So is a combining character sequence the same as a "character"?

A: That depends. For a programmer, a Unicode code value represents a single character (for exceptions, see below). For an end user, it may not. The better word for what end-users think of as characters is grapheme (as defined in the Unicode glossary): a minimally distinctive unit of writing in the context of a particular writing system.

For example, å (A + COMBINING RING or A-RING) is a grapheme in the Danish writing system, while KA + VIRAMA + TA + VOWEL SIGN U is one in the Devanagari writing system. Graphemes are not necessarily combining character sequences, and combining character sequences are not necessarily graphemes. Moreover, there are a number of other cases where a user would not count "characters" the same way as a programmer would: where there are invisible characters such as the RLM used in BIDI, compatibility composites such as "Dz", "ij", or Roman numerals, and so on. [MD]

Q: I would think that the certain characters would have compatibility decompositions. Why don't they?

A: Many characters such as the following are "confusables" rather than compatibility characters.

2044 (FRACTION SLASH) --> 002F (SOLIDUS)
2010 (HYPHEN) --> 002D (HYPHEN-MINUS)
2013 (EN DASH) --> 002D (HYPHEN-MINUS)
2014 (EM DASH) --> 002D 002D (HYPHEN-MINUS, HYPHEN-MINUS)

They are characters that look similar, but do have distinct behavior and generally distinct appearance (whether in length or angle). Consult the Unicode Standard for descriptions of the differences between these characters.

Compatibility characters are really just particular presentation forms of another character (or sequence of characters), encoded to ensure round-trip conversion to legacy encodings. [MD]

Q: Do all the compatibility ideographs have equivalents?

A: No, the ideographs FA0E, FA0F, FA11, FA13, FA14, FA1F, FA21, FA23, FA24, FA27, FA28, and FA29 have no canonical equivalents. These 12 characters are not duplicates and should be treated as a small extension of the set of unified ideographs. In fact, they are derived from industry standards, but are not duplicates of anything. They didn't make it into the main Unihan block because they aren't in any preexisting national standard.  [JC]

Q: Do all the Unicode character set mappings cover control codes?

A: No, the control code mappings are often omitted from the tables on the Unicode site. For the ASCII family of character sets, these are usually one-to-one mappings from the Unicode set based on taking the lower 8 bits of the Unicode character. However, they may differ significantly for other sets, such as EBCDIC.

The correct Unicode mappings for the special graphic characters (01-1F, 7F) of CP437 and other DOS-type code pages are available at ftp://ftp.unicode.org/Public/MAPPINGS/ as well as http://www.unicode.org/Public/MAPPINGS  [JC]

Q: Is the POSIX ctype.h model sufficient for Unicode?

A: POSIX "ctype.h" knows but two cases, whereas Unicode knows three. In POSIX, only European Arabic digits can pass "isdigit", whereas Unicode has many sets of digits, all putatively equal in their status as digits. In POSIX "ctype.h", that which is "alnum" but not "alpha" must be a "digit", but Unicode is aware that not all numbers are digits, nor are all letters alphabetic. Unicode groks spacing and non-spacing marks, but POSIX comprehends them not. [JC]

Q: How should characters (particularly composite characters) be counted, for the purposes of length, substrings, positions in a string, etc.

A: In general, there are 3 different ways to count characters. Each is illustrated with the following sample string. 
"a" + umlaut + greek_alpha + \uE0000.
(the latter is a private use character)

1. Code Units: e.g. how many bytes are in the physical representation of the string. Example:
In UTF-8, the sample has 9 bytes. [61 CC 88 CE B1 F3 A0 80 80]
In UTF-16BE, it has 10 bytes. [00 61 03 08 03 B1 DB 40 DC 00]
In UTF-32BE, it has 16 bytes. [00 00 00 61 00 00 03 08 00 00 03 B1 00 0E 00 00]

2. Codepoints: how may code points are in the string.
The sample has 4 code points. This is equivalent to the UTF-32BE count divided by 4.

3. Graphemes: what end-users consider as characters.
A default grapheme cluster is specified in UAX #29, Unicode Text Segmentation, as well as in UTS #18 Unicode Regular Expressions.

The choice of which one to use depends on the tradeoffs between efficiency and comprehension. For example, Java, Windows and ICU use #1 with UTF-16 for all low-level string operations, and then also supply layers above that provide for #2 and #3 boundaries when circumstances require them. This approach allows for efficient processing, with allowance for higher-level usage. However, for a very high level application, such as word-processing macros, graphemes alone will probably be sufficient. [MD]

Q: Doesn't canonical equivalence mean that no Unicode-conformant process can treat canonically equivalent sequences differently in any way?

A: No. That is too strong a statement about canonical equivalence. Let's take a look at a simple example:

<00C1> a-acute and the sequence <0041 0301> a+combining acute are canonically equivalent sequences.
However, that doesn't mean that "no Unicode-conformant processs should treat them differently in any way." A Unicode-conformant process could declare that it does not interpret combining marks, in which case, for it, <0041 0301> is a sequence of <0041> plus an uninterpreted character. And trivially, a Unicode-conformant process allocating a buffer for character storage clearly has to treat <00C1> and <0041 0301> differently, since the amount of storage required differs.

What canonical equivalence is supposed to mean is that if a Unicode- conformant process interprets all the code points involved in the canonical equivalence, it should not insist on an interpretive difference in the two as constituting some kind of character meaning difference. Thus what is non-conformant would be for Process A to hand Process B <00C1>, i.e. a-acute, for Process B to acknowledge that it got <0041 0301>, i.e. a-acute, and then for Process A to insist that Process B is non-conformant. That insistence would itself be non-conformant, since Process B was within its rights, by virtue of canonical equivalence. [KW]

Q: My language needs a precomposed character, but only the base character and accent are available in Unicode.

See Where Is My Character? and My language needs the digraph "xy". Why is it not encoded as a single character?.

Q: I can't find the diacritical mark I need. Unicode contains one that looks the same but has a different function. Can you add the one I need?

A: Diacritic marks are not encoded by function, and are not specific to language or usage. For example, look at the acute accent. In some languages, it is a diacritic to indicate a distinct letter (with a distinct pronunciation); in other languages it marks a stress, or a quantity; in others it marks a tone. The implications for linguistic processing (including sorting) may be different in each case. Similarly, the U+0308 COMBINING DIAERESIS is to be used for diaeresis, trema, umlaut, as well as other, possibly unrelated uses.

Encoding separate diacritics for each function would have led to confusion as to which was which in each instances, to user inability to chose and enter the correct forms, and similar problems. Moreover, if each function had been encoded, we would have had a legacy problem with interworking with precomposed letters, as for the ISO 8859 family of 8-bit character sets widespread in European implementations. The letter "Ö" is simply encoded as 0xF6 in ISO 8859-1 Latin-1 data, regardless of whether it is being used (in Dutch) as a trema, or (in German, e.g., böse) as an umlaut.

Q: Do I always use U+0323 COMBINING DOT BELOW when I need to put a dot under a character?

A: Some combining marks are intended for use with a specific script. So, for instance, to write a letter in Hindi with a dot below you would use U+0903 DEVANAGARI NUKTA, and to write a pointed letter in Hebrew with a hiriq dot below you would use U+05B4 HEBREW POINT HIRIQ. In other cases, such as Latin characters with a dot below, you would use U+0323 COMBINING DOT BELOW.

Q: Unicode doesn't contain the character I need, which is a Latin letter with a certain diacritical mark. Can you add it?

A: Unicode can already express almost anything you will ever need in any field of study by using a combination of Latin, IPA, or other base letters with the various combining diacritical marks. For example, if you need a highly specialized character such as "Z with stroke, cedilla, and umlaut", you can get this combination by using three existing character codes in combination:

U+01B5 LATIN CAPITAL LETTER Z WITH STROKE
U+0327 COMBINING CEDILLA
U+0308 COMBINING DIAERESIS

With appropriate rendering software, that sequence should produce a glyph combination like this: Z umlaut cedilla

Even if the combination is not available in a particular font, it is unambiguous and Unicode conformant systems should transmit and retain the sequence without distortion, and it may be processed programmatically.

The Navajo-specific question below is also applicable to a wide variety of similar cases.

Q: Unicode doesn't contain some of the precomposed characters needed for Navajo and other indigenous languages of the Americas. Will you add them?

A: The way to encode the various Navajo letters with diacritics is with the use of combining marks. For example, Navajo high-toned nasalized vowels:

a + ogonek + acute = <U+0061, U+0328, U+0301>

and so on for the other vowels.

U+0328 is the combining ogonek, and U+0301 is the combining acute accent. (Navajo orthography uses the ogonek, which is the hook to the right, for nasalization; that is not the same as the cedilla, which is the hook to the left. See the difference between U+0119 e-ogonek, and U+0229 e-cedilla.)

In Unicode Normalization Form C, the a and the ogonek would be replaced by the single code for a-ogonek, producing:

a + ogonek + acute   →   a-ogonek + acute  =  <U+0105, U+0301>
i + ogonek + acute   →   i-ogonek + acute  =  <U+012F, U+0301>

For display and printing, these combinations should just show the whole letters, with both accents placed properly. Up-to-date Microsoft Windows systems (for example) will do that automatically and correctly for you.

See also the web page Where Is My Character? [KW]

Q: Yes, I can represent (for example) X with circumflex by use of X with a combining circumflex: <U+0058, U+0302>. But it doesn't display correctly. The circumflex comes out misplaced, not properly over the "X".

A: Your problem is most likely a limitation of the layout engine and/or font you are using. The real question is what repertoire of base+accent combinations your layout engine and fonts are supporting for display. Fonts that properly support a repertoire with the combination you need should have the correct display.

If the font doesn't support the repertoire, you can end up with various glitches in display. Exactly how things appear in that case will depend on internal details regarding how the font may handle combining marks.

To compare the possible displays of sequences with those that could have resulted if X-circumflex had been encoded as a precomposed character, see the following table.

composite examples

Some fonts, such as the Doulos and Charis fonts, which are freely available for download, contain large repertoires of appropriate precomposed glyphs for use by linguists and writers of minority languages. Try checking out those fonts to see if they might cover your repertoire needs. See also Display Problems.

Q: Just how hard is it for a font designer to support a sequence like X+circumflex, compared to supporting a precomposed character?

A: With modern font technologies, such as OpenType and AAT, the difference is relatively small. For example, in OpenType, it is a matter of adding an entry for the sequence in a ligature table, such as is discussed in the VOLT and InDesign Tutorial. There is no fundamental need for a precomposed character to be encoded in the standard at all in order for the font to have and display the correct precomposed glyph for the combination you need.

The hard work, in either case, is in the design for the precomposed glyph. Conceptually it seems simple enough to add a precomposed glyph to a font — after all, typically the base glyph will be in the font already. But professional font design requires considerable effort. Any time a new accented glyph is added, attention must be paid to design integrity compared to other accented glyphs, kerning issues with all other glyphs, and the possible need for yet other ligatures. Most of this work then has to be repeated for each face of the font: bold, italics, smallcaps, and their combinations. The amount of work for testing the font is multiplied many fold, because not only does the new glyph need testing by itself, but also in interaction with the other glyphs in the font. This is the fundamental reason why commercial fonts are relatively slow to adopt large new collections of precomposed glyphs into their supported repertoires.

Q: Is there a way for font designers to provide flexible support for arbitrary accented combinations?

A: Yes, in  some cases, modern fonts support diacritical anchors, which enable pretty good dynamic display even of completely arbitrary base+accent combinations. For example, in Windows Vista, the core fonts (Arial, Tahoma, Times New Roman, and a few others) already have this feature. Other systems, such as Mac OS X, can provide such dynamic display even in the absence of explicit font support.

Q: Why are new combinations of Latin letters with diacritical marks not suitable for addition to Unicode?

A: There are several reasons. First, Unicode encodes many diacritical marks, and the combinations can already be produced, as noted in the answers to some questions above. If precomposed equivalents were added, the number of multiple spellings would be increased, and decompositions would need to be defined and maintained for them, adding to the complexity of existing decomposition tables in implementations.

Finally, normalization form NFC (the composed form favored for use on the Web) is frozen—no new letter combinations can be added to it. Therefore, the normalized NFC representation of any new precomposed letters would still use decomposed sequences, which can already be expressed by combining character sequences in Unicode. Nothing would be gained by adding the letter with diacritical mark as a precomposed character; on the contrary, adding such a letter would add one or more multiple spellings to be reckoned with, incrementally complicating all Unicode implementations for no net gain. [KW]

Q: Is U+034F COMBINING GRAPHEME JOINER a combining mark?

A: Yes. It is not a format control character, but rather a combining mark. It has the General Category value gc=Mn and the canonical combining class value ccc=0. The presence of a combining grapheme joiner in the midst of a combining character sequence does not interrupt the combining character sequence. [KW]

Q: Does U+034F COMBINING GRAPHEME JOINER affect display of combining character sequences?

A: No. It does not impact cursive joining or ligation (contrast U+200C ZERO WIDTH NON-JOINER and U+200D ZERO WIDTH JOINER). And the CGJ does not have any visible display of its own. Of course, as for any such character in the Unicode Standard with no visible display, it is always possible to use a visible glyph when deliberately showing hidden characters, as for an editor's Show Symbol or Show Hidden mode. [KW]

Q: Does U+034F COMBINING GRAPHEME JOINER join graphemes?

A: No. Despite its name, the combining grapheme joiner neither joins graphemes together in the way punctuation might, nor does it create new graphemes by combinations of other characters. Especially, it cannot be used to construct grapheme clusters out of arbitrary character sequences, or extend the scope of subsequent combining characters. It has no impact on line breaking, except that as for other combining marks, it should be kept with its base when breaking a line. [KW]

Q: What is the function of U+034F COMBINING GRAPHEME JOINER?

A: It has several functions: it is used to affect the collation of adjacent characters for purposes of language-sensitive collation, searching, and matching, and used to distinguish sequences that would otherwise be canonically equivalent.

In collation, the primary function is to prevent contractions from forming. Thus, for example, while “ch” is sorted as a single unit in a tailored Slovak collation, the sequence <c, CGJ, h> will sort as a “c” followed by an “h”. This usage requires no tailoring of either the combining grapheme joiner or the sequence. (It is possible to give sequences of characters which include the combining grapheme joiner special tailored weights; however, such an application of CGJ is not recommended.)

Second, the insertion of a combining grapheme joiner into a sequence of combining marks will block canonical reordering of those combining marks. This can be used in some unusual circumstances where two sequences of combining marks need to be distinguished, but where the different sequences would be neutralized by normalization. For example, the sequence of Hebrew points <hiriq, patah> can be distinguished from the sequence <patah, hiriq> by inserting a combining grapheme joiner: <patah, CGJ, hiriq>. The presence of the CGJ would prevent reordering of that sequence to <hiriq, patah>, thus enabling a reliable distinction to be maintained. Such usage will also cause differences in collation for the affected sequences. [KW] [MD]

Q: Unicode doesn't seem to distinguish between tréma and umlaut, but I need to distinguish. What shall I do?

A. For some purposes, it may be necessary to maintain a distinction between tréma and umlaut, for example, in bibliographic records kept by the German library network. For the Latin script, the Unicode Standard does not distinguish identically appearing diacritical marks with different functions. Doing so would result in confusion in implementations and among users.

The character U+034F COMBINING GRAPHEME JOINER (CGJ) may be used to make the relevant sorting, searching, and data mapping distinctions required for umlaut versus tréma. The semantics of CGJ are such that it should impact only searching and sorting, for systems which have been tailored to distinguish it, while being otherwise ignored in interpretation. The CGJ character was encoded with this purpose in mind.

The sequences <a, umlaut> and <a, CGJ, umlaut> are not canonically equivalent. this means that the distinction will not be normalized away on conversion in and out of bibliographic systems. This eases the interoperability problem. Both sequences will display as they should.

Implementations which need to distinguish the two for searching and sorting may systematically maintain weighting distinctions. <a, umlaut> = <ä> can be treated as equivalent to <a, e> for sorting purposes, while the tréma <a, CGJ, umlaut> can be weighted as a secondary variant of <a> thus resulting in the desired behavior for such systems. Existing collations which do not distinguish tréma and umlaut in their data will continue to work exactly as they currently do, since in default collation tables CGJ is ignored in weighting.

Existing collation, searching, and matching based on the Unicode Collation Algorithm will continue to behave as originally specified: they will not distinguish tréma and umlaut in German data. Only collation tables that add new weights for the sequence <CGJ, umlaut> will distinguish between that and a plain umlaut. [KW]

Q: Is it possible to apply a diacritic or combining enclosing mark to a sequence of more than one (non-combining) character?

A: No, with the exception of the "double diacritics" deliberately designed to be applied onto a two letter sequence, e.g. U+035D COMBINING DOUBLE BREVE. Neither ZWJ (U+200D ZERO WITDH JOINER) nor CGJ (U+034F COMBINING GRAPHEME JOINER) "glue" characters together in a way that the scope of any following combining character would be affected. To get a character sequence like "Esc" into something like the U+20E3 COMBINING ENCLOSING KEYCAP, you must resort to higher-level protocols. [KP]

Q: I’ve looked through the Unicode codecharts and don’t see the combination I need for transliterating Egyptological yod, which is made up of the letter i (/I) plus a half-ring-looking diacritic above it. What should I do?

A: Because the combination of the letter i or I and diacritic is already covered by characters in Unicode, no precomposed characters for Egyptological yod were separately encoded. (See Where Is My Character? My language needs the digraph "xy". Why is it not encoded as a single character? and other questions above.)

For the diacritic, three choices are available: U+0313 COMBINING COMMA ABOVE, U+0357 COMBINING RIGHT HALF RING ABOVE, or U+0486 COMBINING CYRILLIC PSILI PNEUMATA. The placement of the diacritic is up to the font-designer and rendering engine, so you should test out available fonts with the preferred diacritic. (For further information on the set of comma-form and half-ring diacritics in Unicode and their relationships, see Unicode Technical Note #32.)

Q: Will this method always work?

A: Typing the letter Latin i/I and the combining diacritic should work, as long as you have a font with the proper glyphs and a recent computer whose rendering engine can display it. Note that the display may work better with some fonts and on certain platforms. If the display doesn’t work on a webpage, see http://www.unicode.org/help/display_problems.html. To help other users get the best results when viewing your webpages, it may be advisable to also include a note on your webpage, identifying which fonts and which browsers provide the best results.

Q: Should I use the combining diacritic over U+0131 LATIN SMALL LETTER DOTLESS I, or should I use U+0069 LATIN SMALL LETTER I?

A: Rendering of diacritics over i automatically accounts for the removal of the dot on i, so the proper choice is to use LATIN SMALL LETTER I (U+0069).