| Revision | 4 |
| Authors | Mark Davis (mark@unicode.org) |
| Date | 1999-04-30 |
| This Version | http://www.unicode.org/unicode/reports/tr9/tr9-4.html |
| Previous Version | http://www.unicode.org/unicode/reports/tr9/tr9-3.html |
| Latest Version | http://www.unicode.org/unicode/reports/tr9/ |
| Unicode Technical Reports | http://www.unicode.org/unicode/techreports.html |
Summary
This document presents the revision of the BIDI algorithm which is part of Version 3.0 of The Unicode Standard. The main changes from Version 2.1 are to clarify the precise results of the algorithm. In addition, there have been some normative changes that will affect the results of the algorithm in edge cases. These normative changes have been approved by the Unicode Technical Committee. Additional editorial changes will be made before the text appears in the publication of Version 3.0.
Unlike most Technical Reports, this document is in the same format as the book text. In the future, this document should be accompanied by a reference implementation.
Status of this document
This document has been considered and approved by the Unicode Technical Committee for publication as a Draft Technical Report. At the current time, the specifications in this technical report are provided as information and guidance to implementers of the Unicode Standard, but do not form part of the standard itself. The Unicode Technical Committee plans to incorporate all of the material in this technical report into Version 3.0 Unicode Standard, either as informative or as normative specification. Please mail corrigenda and other comments to Unicore@Unicode.org.
The Unicode Standard prescribes a memory representation order known as logical order. When text is presented in horizontal lines, most scripts display characters from left to right. However, there are several scripts (such as Arabic or Hebrew) where the natural ordering of horizontal text in display is from right to left. If all of the text has the same horizontal direction, then the ordering of the display text is unambiguous. However, when bidirectional text (a mixture of left-to-right and right-to-left horizontal text) is present, some ambiguities can arise in determining the ordering of the displayed characters.
This section describes the algorithm used to determine the directionality for bidirectional Unicode text. The algorithm extends the implicit model currently employed by a number of existing implementations and adds explicit format codes for special circumstances. In most cases, there is no need to include additional information with the text to obtain correct display ordering. However, when necessary, additional information can be included in the text by means of this small set of directional formatting codes.
In general, the Unicode Standard does not supply formatting codes; formatting is left up to higher-level protocols. However, in the case of bidirectional text, there are circumstances where an implicit bidirectional ordering is not sufficient to produce comprehensible text. To deal with these cases, a minimal set of directional formatting codes is defined to control the ordering of characters when rendered. This allows exact control of the display ordering for legible interchange and also ensures that plain text used for simple items like filenames or labels can always be correctly ordered for display.
The directional formatting codes are used only to influence the display ordering of text. In all other respects they should be ignored--they have no effect on the comparison of text, nor on word breaks, parsing, or numeric analysis. The characters are still interpreted in logical order--only the display is affected. The ordering of bidirectional text depends upon the directional properties of the characters in the text. Section 4.3, Directionality lists the ranges of characters that have each particular directional character type.
Two types of explicit codes are used to modify the standard implicit Unicode bidirectional algorithm. In addition, there are implicit ordering codes, the right-to-left and left-to-right marks. All of these codes are limited to the current paragraph; thus their effects are terminated by a paragraph separator. The directional types left-to-right and right-to-left are called strong types, and characters of those types are called strong directional characters. The directional types associated with numbers are called weak types, and characters of those types are called weak directional characters.
Although the term embedding is used for some explicit codes, the text within the scope of the codes is not independent of the surrounding text. Characters within an embedding can affect the ordering of characters outside, and vice versa. The algorithm is designed so that the use of explicit codes can be equivalently represented by out-of-line information, such as stylesheet information. However, any alternative representation will be defined by reference to the behavior of the explicit codes in this algorithm.
The following codes signal that a piece of text is to be treated as embedded. For example, an English quotation in the middle of an Arabic sentence could be marked as being embedded left-to-right text. If there were a Hebrew phrase in the middle of the English quotation, then that phrase could be marked as being embedded right-to-left. These codes allow for nested embeddings.
| RLE | Right-to-Left Embedding | Treat the following text as embedded right-to-left. |
|---|---|---|
| LRE | Left-to-Right Embedding | Treat the following text as embedded left-to-right. |
The precise meaning of these codes will be made clear in the discussion of the algorithm. The effect of right-left line direction, for example, can be accomplished by simply embedding the text with RLE...PDF.
The following codes allow the bidirectional character types to be overridden when required for special cases, such as for part numbers. These codes allow for nested directional overrides.
| RLO | Right-to-Left Override | Force following characters to be treated as strong right-to-left characters. |
|---|---|---|
| LRO | Left-to-Right Override | Force following characters to be treated as strong left-to-right characters. |
The precise meaning of these codes will be made clear in the discussion of the algorithm. The right-to-left override, for example, can be used to force a part number made of mixed English, digits and Hebrew letters to be written from right to left.
The following code terminates the effects of the last explicit code (either embedding or override) and restores the bidirectional state to what it was before that code was encountered.
| Pop Directional Format | Restore the bidirectional state to what it was before the last LRE, RLE, RLO, LRO. |
These characters are very light-weight codes. They act exactly like right-to-left or left-to-right characters, except that they do not display (or have any other semantic effect). Their use is generally more convenient than the explicit embeddings or overrides, since their scope is much more local (as will be made clear in the specification to follow).
RLM |
Right-to-Left Mark | Right-to-left zero-width character |
|---|---|---|
LRM |
Left-to-Right Mark | Left-to-right zero-width character |
There is no special mention of the implicit directional marks in the following algorithm. That is because their effect on bidirectional ordering is exactly the same as a corresponding strong directional character; the only difference is that they do not appear in the display.
The Bidirectional Algorithm takes a stream of text as input, and proceeds in three main phases:
The algorithm only reorders text within a paragraph; characters in one paragraph have no effect on characters in a different paragraph. Paragraphs are divided by the Paragraph Separator or appropriate Newline Function (see Section 4.3, Directionality and UTR #13 Newline Guidelines on the handling of CR, LF, and CRLF). Paragraphs may also be determined by higher-level protocols: for example, the text in two different cells of a table will be in different paragraphs.
Combining characters always attach to the preceding base character in the memory representation. Even after reordering for display and performing character shaping, the glyph representing a combining character will attach to the glyph representing its base character in memory. Depending on the line orientation and the placement direction of base letterform glyphs, it may, for example, attach to the glyph on the left, or on the right, or above.
In the following text, the normative definitions and rules are distinguished by the following numbering:
Numbering |
Section |
|---|---|
| BDn | Definitions |
| Pn | Paragraph levels |
| Xn | Explicit levels and directions |
| Wn | Weak types |
| Nn | Neutral types |
| In | Implicit levels |
| Ln | Resolved levels |
BD1. The bidirectional characters types are values assigned to each Unicode character, including unassigned characters.
BD2. Embedding levels are numbers that indicate how deeply the text is nested, and the default direction of text on that level. The minimum embedding level of text is zero, and the maximum explicit depth is level 61.
Embedding levels are explicitly set by both override format codes and by embedding format codes; higher numbers mean the text is more deeply nested. The reason for having a limitation is to provide a precise stack limit for implementations to guarantee the same results. Sixty-one levels is far more than sufficient for ordering, even with mechanically generated formatting; the display becomes rather muddied with more than a small number of embeddings.
BD3. The default direction of the current embedding level (for a character in question) is called the embedding direction. It is L if the embedding level is even, and R if the embedding level is odd.
For example, in a particular piece of text, Level 0 is plain English text, Level 1 is plain Arabic text, possibly embedded within English level 0 text. Level 2 is English text, possibly embedded within Arabic level 1 text, and so on. Unless their direction is overridden, English text and numbers will always be an even level; Arabic text (excluding numbers) will always be an odd level. The exact meaning of the embedding level will become clear when the reordering algorithm is discussed, but the following provides an example of how the algorithm works.
BD4. The paragraph embedding level is the embedding level that determines the default bidirectional orientation of the text in that paragraph.
BD5. The direction of the paragraph embedding level is called the paragraph direction.
In some contexts the paragraph direction is also known as the base direction.
BD6. The directional override status determines whether the bidirectional type of characters is to be reset with explicit directional controls. This status has three states:
Status |
Interpretation |
|---|---|
neutral |
no override is currently active |
right-to-left |
characters are to be reset to R |
left-to-right |
characters are to be reset to L |
In the following examples, case is used to indicate different implicit character types for those unfamiliar with right-to-left letters. Uppercase letters stand for right-to-left characters (such as Arabic or Hebrew), while lowercase letters stand for left-to-right characters (such as English or Russian).
Memory: car is THE CAR in arabic Character types: LLL-LL-RRR-RRR-LL-LLLLLL Resolved levels: 000000011111110000000000
Notice that the neutral character (space) between THE and CAR gets the level of the surrounding characters. This is how the implicit directional marks have an effect. By inserting appropriate directional marks around neutral characters, the level of the neutral characters can be changed.
The normative bidirectional character types for each character are specified in Chapter 4, Character Properties, and updated in the Unicode Character Properties files on the Unicode web site (ftp://ftp.unicode.org/Public/UNIDATA/).
The bidirectional character types are summarized in Table 3-7.
|
|
The term European digits is used to refer to decimal forms common in Europe and elsewhere, and Arabic-Indic digits to refer to the native Arabic forms. (See the Section 8.2, Arabic, for more details on naming digits.) |
|
Unassigned characters are given strong types in the algorithm. This is an explicit exception to the general Unicode conformance requirements with respect to unassigned characters. As characters become assigned in the future, these BIDI types may change. |
|
Private use characters can be assigned different values by a conformant implementation. |
|
For the purpose of the BIDI algorithm, inline objects (such as graphics) are treated as if they are an OBJECT REPLACEMENT CHARACTER (U+FFFC). |
Table 3-8 lists additional abbreviations used in the examples and internal character types used in the algorithm.
| Symbol | Description |
|---|---|
| N | Neutral or Separator (B, S, WS, ON) |
| e | The text ordering type (L or R) that matches the embedding level direction (even or odd) |
| SOP | Start of paragraph. Treated as e. |
| EOP | End of paragraph. Treated as e. |
In the Unicode 3.0 character database, new BIDI character types are introduced to make the body of the algorithm depend only on the types of characters, and not on the character values. The changes from the 2.0 BIDI types are listed in Table 3-9:
| Characters | New BIDI type |
|---|---|
| All characters with general category Me, Mn | NSM |
| All characters of type R in the Arabic ranges (0600-06FF, FB50-FDFF,
FE70-FEFE) [Letters in the Unicode 3.0 Thana and Syriac ranges will also get this value.] |
AL |
| The explicit embedding characters: LRO, RLO, LRE, RLE, PDF | LRO, RLO, LRE, RLE, PDF, resp. |
| Formatting characters and controls (general category Cf and Cc) that are of BIDI type ON | BN |
| Zero Width Space | BN |
Implementations that use older property tables can adjust to the modifications in the BIDI algorithm by algorithmically remapping the above characters to the new types.
The body of the bidirectional algorithm uses character types and explicit codes to produce a list of resolved levels. This resolution process consists of five steps: (1) determining the paragraph level; (2) determining explicit embedding levels and directions; (3) resolving weak types; (4) resolving neutral types; and (5) resolving implicit embedding levels.
P1. Split the text into separate paragraphs. Paragraph separators are kept with the previous paragraph. Within each paragraph, apply all the other rules of this algorithm.
P2. In each paragraph, find the first character that either is a strong directional character (L, AL, R), or is a directional code (RLE, LRE, RLO, LRO) not immediately followed by PDF.
Because paragraph separators delimit text in this algorithm, this will generally be the first strong character after a paragraph separator or at the very beginning of the text.
P3. If a character is found in P2 and it is of type AL, R, RLE, or RLO, then set the paragraph embedding level to one; otherwise, set it to zero.
All explicit embedding levels are determined from the embedding and override codes, by applying the explicit level rules X1 through X9. These rules are applied as part of the same logical pass over the input.
X1. Begin by setting the current embedding level to the paragraph embedding level. Set the directional override status to neutral. Process each character iteratively, applying rules X2 through X8. Only embedding levels from 0 to 61 are valid in this phase.
In the resolution of levels in rules I1 and I2, the maximum embedding level of 62 can be reached.
X2. With each RLE, compute the least greater odd embedding level.
a. If this new level would be valid, then this embedding code is valid. Remember (push) the current embedding level and override status. Reset the current level to this new level, and reset the override status to neutral. The RLE and subsequent characters are set to the new current level. Set the type of the RLE to R.
b. If the new level would not be valid, then this code is invalid. Don't change the current level or override status. Set the type of the RLE to BN. Until the matching PDF is reached, do the same to any other embedding or override code.
For example, level 0 => 1; levels 1, 2 => 3; levels 3, 4 => 5; ...59,60 => 61; above 60, no change (don’t change levels with RLE if the new level would be invalid).
X3. With each LRE, compute the least greater even embedding level.
a. If this new level would be valid, then this embedding code is valid. Remember (push) the current embedding level and override status. Reset the current level to this new level, and reset the override status to neutral. The LRE and subsequent characters are set to the new current level. Set the type of the LRE to L.
b. If the new level would not be valid, then this code is invalid. Set the type of the LRE to BN. Don't change the current level or override status. Until the matching PDF is reached, do the same to any other embedding or override code.
For example, levels 0, 1 => 2; levels 2, 3 => 4; levels 4, 5 => 6; ...58, 59 => 60; above 59, no change (don’t change levels with LRE if the new level would be invalid).
An explicit directional override sets the embedding level in the same way the explicit embedding codes do, but also changes the directional character type of affected characters to the override direction.
X4. With each RLO, compute the least greater odd embedding level.
a. If this new level would be valid, then this embedding code is valid. Remember (push) the current embedding level and override status. Reset the current level to this new level, and reset the override status to right-to-left. The RLO and subsequent characters are set to the new current level. Set the type of the RLO to R.
b. If the new level would not be valid, then this code is invalid. Set the type of the RLO to BN. Don't change the current level or override status. Until the matching PDF is reached, do the same to any other embedding or override code.
X5. With each LRO, compute the least greater even embedding level.
a. If this new level would be valid, then this embedding code is valid. Remember (push) the current embedding level and override status. Reset the current level to this new level, and reset the override status to left-to-right. The LRO and subsequent characters are set to the new current level. Set the type of the LRO to L.
b. If the new level would not be valid, then this code is invalid. Set the type of the LRO to BN. Don't change the current level or override status. Until the matching PDF is reached, do the same to any other embedding or override code.
X6. Whenever the directional override status is not neutral, reset the current character type to the directional override status.
If the directional override status is neutral, then all intermediate characters retain their normal types: Arabic characters stay AL, Latin characters stay L, neutrals stay N, and so on. If the directional override status is R, then all characters become R. If the directional override status is L, then all characters become L.
There is a single code to terminate the scope of the current explicit code, whether an embedding or a directional override. All codes and pushed states are completely popped at the end of paragraphs.
X7. With each PDF, determine the matching embedding or override code.
a. If there was a valid matching code, set the level of the PDF to that code's embedding level and type. Then restore (pop) the last remembered (pushed) embedding level and directional override.
b. If there was no valid matching code, set the PDF to BN.
The following is an example of the levels and types set by the embedding codes:
Memory: L R L <RLE> L R L <LRO> R L R <PDF> R <PDF> L R Levels: 0 0 0 1 1 1 1 2 2 2 2 2 1 1 0 0 Types: L R L R L R L L L L L L R R L R
X8. All explicit directional embeddings and overrides are completely terminated at the end of each paragraph, even if there are no matching PDF codes. Any unterminated embedding codes are still valid and interpreted. Paragraph separators are not included in the embedding.
Memory: L R L <RLE> L R L <LRO> R L R B Levels: 0 0 0 1 1 1 1 2 2 2 2 0 Types: L R L R L R L L L L L N
Because the embedding codes have strong types, all overrides and resolution of weak types and neutrals take effect within the bounds of an embedding; that is, nothing within an embedding or override will affect the character direction of codes outside of that embedding, and vice versa. (Although character directions are not affected across embedding blocks, the final display ordering may be changed because of the way that levels are reordered in the final phase of the algorithm.) The two exceptions are in empty embeddings, and in adjacent embeddings of the same type:
X9. If the character after a PDF is the same as the matching code for that PDF, set the PDF and that next character to BN. Otherwise, if the PDF is immediately preceded by an embedding code, set that previous character and the PDF to BN.
RLE ... PDF RLE ... PDF => R ...BN BN ... R RLE PDF => BN BN
|
This rule allows implementations with merging style runs, such as found in most word processors, to achieve the same effect as using embedding codes. If a non-merged display order is needed, that can be achieved by inserting a zero-width space or zero-width no-break space between the two codes. |
Non-spacing marks are now resolved based on the previous characters.
W1. Search backwards from each non-spacing mark (NSM) to the first character which is not a boundary neutral (BN). If there is no such character, change the type of the NSM to ON. Otherwise change the type of the NSM to the type of that character.
AL NSM NSM => AL AL AL R BN NSM => R BN R L BN NSM => L BN L SOP NSM => SOP ON
The text is next parsed for numbers. This pass will change the directional types European Number Separator, European Number Terminator, and Common Number Separator to be European Number text, Arabic Number text, or Other Neutral text. The text to be scanned may have already had its type altered by directional overrides. If so, then it will not parse as numeric.
W2. Search backwards from each instance of a European number until the first strong character (R, L, AL) or paragraph boundary is found. If a character is found, and it is AL, change the type of the European number to Arabic number.
AL EN => AL AN AL N EN => AL N AN SOP N EN => SOP N EN L N EN => L N EN R N EN => R N EN
Because embeddings are bounded by strong types, the backwards search will stop at an embedding boundary.
W3. Change all ALs to R.
W4. Change any sequence of Boundary Neutrals adjacent to an European Number to a European Number; change any remaining sequence of Boundary Neutrals adjacent to an European Terminator to a European Terminator; change any remaining sequence of Boundary Neutrals adjacent to an Arabic Number to an Arabic Number.
EN BN => EN EN BN EN => EN EN AN BN => AN AN BN AN => AN AN AN BN EN => AN EN EN EN BN AN => EN EN AN ET BN EN => ET EN EN AN BN ET => AN ET ET
W5. A single European separator between two European numbers changes to an European number. A single common separator between two numbers of the same type changes to that type:
EN ES EN => EN EN EN EN CS EN => EN EN EN AN CS AN => AN AN AN
W6. A sequence of European terminators adjacent to European numbers changes to all European numbers:
ET ET EN => EN EN EN EN ET ET => EN EN EN AN ET EN => AN EN EN
W7. Otherwise, separators, terminators and Boundary Neutrals change to Other Neutral:
AN ET => AN ON L ES EN => L ON EN EN CS AN => EN ON AN ET AN => ON AN BN => ON
W8. Search backwards from each instance of a European number until the first strong character (R, L) or paragraph boundary is found. If an L is found, or if no strong character is found and the paragraph direction is L, then the change the type of the European number to L.
L N EN => L N L R N EN => R N EN
The next phase resolves the direction of the neutrals. The results of this phase are that all neutrals become either R or L. Generally, neutrals take on the direction of the surrounding text. In case of a conflict, they take on the embedding direction. End-of-paragraph and start-of-paragraph are treated as if there were a character of the paragraph direction at that position.
N1. A sequence of neutrals takes the direction of the surrounding strong text. European and Arabic numbers are treated as though they were R.
R N R => R R R L N L => L L L R N AN => R R AN AN N R => AN R R R N EN => R R EN EN N R => EN R R
N2. Any remaining neutrals take the embedding direction.
N => e
Since end-of-paragraph (EOP) and start-of-paragraph (SOP) are treated as if they were characters of the paragraph direction, the following examples are covered by these rules:
L N EOP => L e EOP R N EOP => R e EOP SOP N L => SOP e L SOP N R => SOP e R
Examples. A list of numbers separated by neutrals and embedded in a directional run will come out in the run’s order.
Storage: he said "THE VALUES ARE 123, 456, 789, OK". Display: he said "KO ,789 ,456 ,123 ERA SEULAV EHT".
In this case, both the comma and the space between the numbers take on the direction of the surrounding text (uppercase = right-to-left), ignoring the numbers. The commas are not considered part of the number since they are not surrounded on both sides (see number parsing). However, if there is an adjacent left-to-right sequence, then European numbers will adopt that direction:
Storage: he said "IT IS A bmw 500, OK." Display: he said ".KO ,bmw 500 A SI TI"
In the final phase, the embedding level of text may be increased, based upon the resolved character type. Right-to-left text will always end up with an odd level, and left-to-right and numeric text will always end up with an even level. In addition, numeric text will always end up with a higher level than the paragraph level. (Note that it is possible for text to end up at levels higher than 61 as a result of this process.) This results in the following rules:
I1. For all characters with an even (left-to-right) embedding direction, those of type R go up one level and those of type AN or EN go up two levels.
I2. For all characters with an odd (right-to-left) embedding direction, those of type L, EN or AN go up one level.
Table 3-10 summarizes the results of the implicit algorithm.
| Type | Embedding Level | |
|---|---|---|
| Even | Odd | |
| L | EL | EL+1 |
| R | EL+1 | EL |
| AN | EL+2 | EL+1 |
| EN | EL+2 | EL+1 |
The following algorithm describes the logical process of finding the correct display order. As described before, this logical process is not necessarily the actual implementation, which may diverge for efficiency as long as it produces the same results. As opposed to resolution phases, this algorithm acts on a per-line basis, and is applied after any line wrapping is applied to the paragraph.
The process of breaking a paragraph into one or more lines that fit within particular bounds is outside the scope of the bidirectional algorithm. Where character shaping is involved, it can be somewhat more complicated (see pages 6-22 through 6-32). Logically there are the following steps:
L1. On each line, reset the embedding level of the following characters to the paragraph embedding level:
|
The types of characters used here are the original types, not those modified by the previous phase. |
|
Since a Paragraph Separator breaks lines, there will be at most one per line, at the end of that line. |
In combination with the following rule, this means that trailing white space will appear at the visual end of the line (in the paragraph direction). Tabulation will always have a consistent direction within a paragraph.
L2. From the highest level found in the text to the lowest odd level on each line, reverse any sequence of characters that are at that level or higher.
This reverses a progressively larger series of substrings. The following four examples illustrate this.
Memory: car means CAR. Resolved levels: 00000000001110 Reverse level 1: car means RAC. Memory: car MEANS CAR. Resolved levels: 22211111111111 Reverse level 2: rac MEANS CAR. Reverse levels 1-2: .RAC SNAEM car Memory: he said "car MEANS CAR." Resolved levels: 000000000222111111111100 Reverse level 2: he said "rac MEANS CAR." Reverse levels 1-2: he said "RAC SNAEM car." Memory: DID YOU SAY ‘he said "car MEANS CAR"’? Resolved levels: 11111111111112222222224443333333333211 Reverse level 4: DID YOU SAY ‘he said "rac MEANS CAR"’? Reverse levels 3-4: DID YOU SAY ‘he said "RAC SNAEM car"’? Reverse levels 2-4: DID YOU SAY ‘"rac MEANS CAR" dias eh’? Reverse levels 1-4: ?‘he said "RAC SNAEM car"’ YAS UOY DID
L3. Combining marks applied to a right-to-left base character will at this point precede their base character. If the rendering engine expects them to follow the base characters in the final display process, then the ordering of the marks and the base character must be reversed.
Many font designers provide default metrics for combining marks that support rendering by simple overhang. Because of the reordering for right-to-left characters, it is common practice to make the glyphs for most combining characters overhang to the left (thus assuming the characters will be applied to left-to-right base characters) and make the glyphs for combining characters in right-to-left scripts overhang to the right (thus assuming that the characters will be applied to right-to-left base characters). With such fonts, the display ordering of the marks and base glyphs may need to be adjusted when combining marks are applied to "unmatching" base characters. See Section 5.12, Rendering Non-Spacing Marks for more information.
L4. A character that possesses the mirrored property as specified by Section 4.7, Mirrored, must be depicted by a mirrored glyph if the resolved directionality of that character is R.
For example, U+0028 left parenthesis—which is interpreted in the Unicode Standard as an opening parenthesis—appears as "(" when its resolved level is even, and as the mirrored glyph ")" when its resolved level is odd.
The bidirectional algorithm specifies part of the intrinsic semantics of right-to-left characters. In the absence of a higher-level protocol that specifically supersedes the interpretation of directionality, systems that interpret these characters must achieve results identical to the implicit bidirectional algorithm when rendering.
Conformance: A process that displays text containing supported right-to-left characters or embedding codes shall display all visible representations of characters (excluding format characters) in the same order as if the Bidirectional Algorithm had been applied to the text, in the absence of higher level protocols.
The goal in marking a format or control character as BN is that it have no effect on the rest of the algorithm. Since the precise ordering of format characters with respect to others is not required for conformance, implementations are free to handle them in different ways for efficiency as long as the ordering of the other characters is preserved.
As with any Unicode characters, systems do not have to support any particular explicit directional formatting code (although it is not generally useful to include a terminating code without including the initiator). Generally, conforming systems will fall into three classes:
The following are concrete examples of how systems may apply higher-level protocols to the ordering of bidirectional text.
When text using a higher-level protocol is to be converted to Unicode plain text, formatting codes should be inserted to ensure that the order matches that of the higher-level protocol, or (as in the last example) the appropriate characters can be substituted.
In the case of vertical line orientation, the bidirectional algorithm is still used to determine the levels of the text. However, these levels are not used to reorder the text, since the characters are usually ordered uniformly from top to bottom. Instead, the levels are used to determine the rotation of the text. Sometimes vertical lines follow a vertical baseline in which each character is oriented as normal (with no rotation), with characters ordered from top to bottom whether they are Hebrew, numbers, or Latin. When setting text using the Arabic script in vertical lines, it is more common to employ a horizontal baseline that is rotated by 90° counterclockwise so that the characters are ordered from top to bottom. Latin text and numbers may be rotated 90° clockwise so that the characters are also ordered from top to bottom.
The bidirectional algorithm also comes into effect when some characters are ordered from bottom to top. For example, this happens with a mixture of Arabic and Latin glyphs when all the glyphs are rotated uniformly 90° clockwise. (The choice of whether text is to be presented horizontally or vertically, or whether text is to be rotated, is not specified by the Unicode Standard, and is left up to higher-level protocols.)
Because of the implicit character types and the heuristics for resolving neutral and numeric directional behavior, the implicit bidirectional ordering will generally produce the correct display without any further work. However, problematic cases may occur when a right-to-left paragraph begins with left-to-right characters, or there are nested segments of different-direction text, or there are weak characters on directional boundaries. In these cases, embeddings or directional marks may be required to get the right display. Part numbers may also require directional overrides.
The most common problematic case is that of neutrals on the boundary of an embedded language. This can be addressed by setting the level of the embedded text correctly. For example, with all the text at level 0 the following occurs:
Memory: he said "I NEED WATER!", and expired. Display: he said "RETAW DEEN I!", and expired.
If the exclamation mark is to be part of the Arabic quotation, then the user can select the text I NEED WATER! and explicitly mark it as embedded Arabic, which produces the following result:
Memory: he said "<RLE>I NEED WATER!<PDF>", and expired. Display: he said "!RETAW DEEN I", and expired.
A simpler method of doing this is to place a right directional mark (RLM) after the exclamation mark. Since the exclamation mark is now not on a directional boundary, this produces the correct result.
Memory: he said "I NEED WATER!<RLM>", and expired. Display: he said "!RETAW DEEN I", and expired.
This latter approach is preferred since it does not make use of the stateful format codes, which can easily get out of sync if not fully supported by editors and other string manipulation. The stateful format codes are generally only needed for more complex (and rare) cases such as double embeddings, as in the following:
Memory: DID YOU SAY ‘<LRE>he said "I NEED WATER!<RLM>", and expired.<PDF>’? Display: ?‘he said "!RETAW DEEN I", and expired.’ YAS UOY DID
Copyright © 1998-1999 Unicode, Inc.. All Rights Reserved. The Unicode Consortium makes no expressed or implied warranty of any kind, and assumes no liability for errors or omissions. No liability is assumed for incidental and consequential damages in connection with or arising out of the use of the information or programs contained or accompanying this technical report.
Unicode and the Unicode logo are trademarks of Unicode, Inc., and are registered in some jurisdictions.