This report presents the specifications of UTF-EBCDIC - EBCID-Friendly UCS Transformation Format.

Status of this document

Scope

To address the use of Unicode character data in byte-oriented ASCII-based systems, the Unicode Standard (see section A.2 of the Unicode Standard) (also ISO/IEC 10646 –1, Amendment no. 2) has defined UTF-8. Use of UTF-8 permits existing ASCII-based systems that have hard-coded dependency on the encoding of the ASCII repertoire of characters to safely process the corresponding Unicode characters. There is a similar requirement to transform Unicode characters to a form that is safe for EBCDIC systems for the control characters and invariant characters.

This Technical Report defines the EBCDIC-friendly UCS transformation format, UTF-EBCDIC.

Neither UTF-EBCDIC nor its intermediate form called UTF-8M in this technical report, are intended to be used in open interchange environments. It is useful in homogeneous EBCDIC systems and networks.

Description

Conversion of the Unicode values to a variable length byte sequence called I8-sequence (intermediate 8-bit sequence) by applying a modified UTF-8 transformation (UTF-8M), enabling the preservation of 65 control characters as single bytes.

• The bytes in the I8-sequence are then converted to the UTF-EBCDIC byte sequence by using a single-byte to single-byte reversible conversion.

Note: The following notation is used in this Technical Report: X'nn .. mm' represent hexadecimal values; <bb…bb> represent values in bit notation; U+abcd represents a Unicode character.

Definition

Step 1: UTF-8M

When these are converted to the UTF-EBCDIC form, the corresponding 65 EBCDIC control characters and 95 EBCDIC graphic characters are preserved as single bytes in the UTF-EBCDIC byte sequence. The 95 EBCDIC graphic characters include 82 invariant (occupy the same code position) characters (including SPACE) across most EBCDIC single-byte code pages and 13 variant ASCII graphic characters (occupy varying code positions). Positions assigned to controls, the invariant graphic characters and the variant graphics are shown in Table B.1.

Furthermore, the values X'00'...X'9F' do not appear in any byte of an I8-sequence except as the direct representation of U+0000 to U+009F. Each Unicode value that is not a part of a valid surrogate pair is represented in an I8-sequence by 1, 2, 3 or 4 bytes, depending on the Unicode value. A valid surrogate pair maps into either 4 bytes or 5 bytes, depending on the Unicode scalar values represented by the surrogate pair.

UTF-8M is intended to be used only as an intermediate step in arriving at UTF-EBCDIC. It is not intended to be used elsewhere.

The I8-sequence is a variable length encoding of Unicode characters using 8-bit sequences, where the high bits indicate which part of the sequence a byte belongs to. Table 1 shows how the bits in a Unicode value (or a valid surrogate pair) are distributed among the bytes in the I8-sequence. The corresponding ranges of Unicode scalar values are also shown.
 
 

where ^a uuuu = 000pp -1, or ^b uuuu = qpppp -1
(to account for addition of 10000₁₆ as in Section 3.7, Surrogates, in the Unicode Standard 2.0)

When converting from I8-sequences to Unicode values, however, implementations do not need to check that the shortest number of bytes is being used, which simplifies the conversion algorithm.

As in the case of UTF-8, any byte-serialized Unicode characters of the "little-endian" form (byte-swapped) must be converted to its corresponding "big-endian" form before converting to the I8-sequence.

Characteristics of I8-sequence

• Unicode characters from U+0000 to U+007E (ASCII repertoire) map to single-byte I8-sequence values X'00' to X'7E' (ASCII values X'00' to X'7E').

• Unicode characters from U+0080 to U+009F (C1 controls) map to single-byte I8-sequence values X'80' to X'9F' (IS 4873 values X'80'to X'9F').

• ASCII values or IS 4873 values do not otherwise occur in an I8-sequence. This paves the way for transforming these into corresponding single-byte EBCDIC controls and graphics in the second step of UTF-EBCDIC transform.

• It is very simple and efficient to convert to and from Unicode text.

 
• The first byte indicates the number of bytes to follow in a multi-byte sequence. This allows for efficient forward parsing.

It is efficient to find the start of a character string from an arbitrary location in a byte stream. You need to search at most five bytes (seven if full range of 31 bits of ISO/IEC 10646 is to be considered) backwards, and it is simple to recognize an initial byte. For example, after converting an UTF-EBCDIC byte back into I8-sequence, in C
 
isInitialByte = ( (byte & 0xE0) != 0xA0);

The search can also be done directly on UTF-EBCDIC byte by utilizing a shadow vector (see Table 4 described later)
 
• The I8-sequence is reasonably compact in terms of number of bytes used for encoding.

Step 2: Byte Conversion

The 64 control characters (U+0000 to U+003F, U+0080 to U+009F), the ASCII 'DEL' character (U+007F), the 95 ASCII graphic characters (including the SPACE character) (U+0020 to U+007E) are mapped respecting EBCDIC conventions, as defined in IBM Character Data Representation Architecture, CDRA, with one exception -- the pairing of EBCDIC Line Feed and New Line control characters are swapped from their CDRA default pairings to ISO 6429 Line Feed (U+000A) and Next Line (U+0085) control characters (to be in line IBM OS/390 UNIX Services, or Open MVS practice and preference, stemming from the hard-coding of X'0A' as the New Line in most ASCII-C compilers.).

(Note: This is a significant change not discussed before at UTC meetings. This was brought to light by the implementers of UTF-EBCDIC in IBM -- that in the MVS Open environment the mapping to and from Unicode maps the EBCDIC New Line to Unicode (U+000A) and Unicode (U+0085) to EBCDIC Line Feed. This draft incorporates this change and requires approval by UTC members / reviewers.)

The map preserves the invariance for a set of 82 graphic characters (including SPACE) (known as the IBM Syntactic Graphic Character set), and maintains consistency with IBM MVS Open Systems Code page (CPGID 1047) for the variant characters from within the ASCII repertoire.

The remaining 96 bytes of EBCDIC 8-bit structure are allocated to X'A0' to X'FF' -- the trailing bytes and leading bytes of the I8-sequence.(from Table 1). The minimum criterion for allocation of these bytes is that it provides for a reversible map.

The trailing and leading bytes (X'A0' to X'FF' of the I8-sequence) are paired with the unassigned UTF-EBCDIC bytes in increasing order. Table 2 and table 3 show the byte maps between the I8-sequence bytes and UTF-EBCDIC bytes in the forward and reverse directions respectively. The resulting UTF-EBCDIC multi-byte sequences will be in the same order as their corresponding Unicode scalar values (when the sequences are zero-filled to equal number of bytes and compared with each other). Please note that the UTF-EBCDIC single-byte values, however, will not be in the same order as their corresponding Unicode values.

(Note: At UTC 79 meeting in San Jose, there was a request for reallocating these bytes to permit the multi-byte EBCDIC-UTF sequences to be in the same order as their corresponding Unicode values, as with UTF-8 sequences. This was not accepted at the meeting itself on the basis of the perceived benefit being not strong enough to possibly break existing implementations. After the meeting, it has been ascertained that it is not too late for implementations to change, to be able to derive the perceived benefits of the above ordering. This draft incorporates this change and requires approval by UTC members / reviewers.)

The resulting UTF-EBCDIC byte sequence can be transparently processed in most EBCDIC systems. It also retains all the characteristics (see the section "Characteristics of I8-sequence" above) of I8-sequence mentioned earlier. Since EBCDIC code page definitions have 13 variants (and only 82 invariants) the choice of the above byte map for the graphic characters has been made to accommodate the MVS Open Systems environment for standardization purposes.
 

Shadow flags

(Note: The categories of bytes in the shadow flag table has changed to reflect the changes in Tables 3 and 4 above.  This draft incorporates this change and requires approval by UTC members / reviewers.)
 

Signature

(Note: The signature sequence of UTF-EBCDIC has changed to reflect the changes in Tables 3 and 4 above. This draft incorporates this change and requires approval by UTC members / reviewers.)

Where to use UTF-EBCDIC?

Bibliography

---

ANNEX A: Intellectual Property Related

IBM LOGO

International Business Machines Corporation Route 100

Somers, NY 10589
 
 

June 2, 1998
 
 
 
 

The Chair, Unicode Technical Committee
 
 

Subject: Disclosure of IBM Technology - EBCDIC-Friendly UCS Transformation Format (EF-UTF)
 
 

The attached document entitled "EBCDIC-Friendly UCS Transformation Format (EF-UTF)" contains IBM technology that has been filed for application for Canadian Patent. However, IBM believes that the technology could be beneficial to the EBCDIC community at large; allowing the community to derive the enormous benefits provided by UCS (ISO/IEC 10646 and Unicode).
 
 

This letter is to inform you that IBM is pleased to make the attached documentation, and the associated technology that has been filed for patent, freely available to anyone concerned towards making the transformation format as part of the UCS standards.
 
 
 
 

Sincerely
 
 

SIGNED
 
 

Elizabeth G. Nichols
Director of National Language Support
and Information Development
 
 

EGN:ghs

Attachment
 
 
 

(Note: The term EF-UTF has been changed to UTF-EBCDIC at the suggestion of UTC meting 78 -- V.S. Umamaheswaran)

ANNEX B: Additional Information

Positions of controls, variants and invariants in EBCDIC

A comparison of UTF-EBCDIC and UTF-8

Special nature of UCS values FEFF, FFFE and FFFF

• X'FE' X'FF', X'FF' X'FE' and X'FF' X'FF' in the I8-sequence

The X'FE' and X'FF' are lead octets of seven-byte I8-sequence (assuming values from all the planes of UCS-4). They will be surrounded (in a properly formed I8-sequence) by a value less than X'C0'. None of the sequences X'FF' X'FF', X'FE' X'FF', and X'FF' X'FE' can appear in a well-formed I8-sequence.
 
• X'FE' X'FF', X'FF'X'FE' and X'FF' X'FF' in the UTF-EBCDIC sequence

The I8-sequence to UTF-EBCDIC byte mappings are: X'FE' to X'FD', and X'FF' to X'FE' (see Table 2). The values X'FE' and X'FF' can be generated in a UTF-EBCDIC byte sequence from I8-sequence values by mapping X'FF' to X'FE' and X'9F' to X'FF' (from Table 2).

X'FF' is the lead byte of a seven-byte I8 sequence and must be followed by six trailing bytes in the range X'A0' to X'BF', which does not include X'9F'. So the X'FE' X'FF' sequence cannot appear in UTF-EBCDIC.

The X'9F' is assigned to the control character -- Application Program Command (APC) -- in ISO-8 C1. According to ISO/IEC 6429, the APC is followed by a parameter string using bit combinations from 0/8 to 0/13 (X'08' to X'0D') and 2/0 to 7/14 (X'20' to X'7E') and terminated by the control function String Terminator (ST) (coded at X'9C' in C1). Therefore, the sequence X'FF' X'FF', the equivalent of two APC controls without intervening parameters or ST-s, also should not appear in UTF-EBCDIC sequence. None of the valid parameter bit combinations can generate a 7-byte I8 sequence that starts with X'FF'. So the sequence X'FF' X'FE' also cannot appear in a UTF-EBCDIC sequence.

Normalization to fixed width

However, this would be possible only if processing is tolerant to native Unicode encoding. If transparency to EBCDIC invariance and controls is needed also in the normalized form, then Unicode cannot be directly used for normalization. It can be seen from Table 1 that the last code position in the BMP (U+FFFF) requires a four-bytes in the I8-sequence and in the corresponding UTF-EBCDIC sequence. A 32-bit integer can be used for normalization of up to four-byte UTF-EBCDIC sequences.

The maximum Unicode scalar value that a four-byte I8-sequence or UTF-EBCDIC sequence can represent is:

On the mapping of bytes in step 2

Similarly the pairing of I8-sequence bytes and UTF-EBCDIC sequence bytes could be done in multiple ways. The simplest requirement on this byte-pairing is that it should be unique and reversible. The pairing adopted in this version of the UTR is based on the request from Oracle Corporation's representative Mr. Jianping Yang -- to be able to maintain the order of the UTF-EBCDIC multi-byte sequences the same as the order of the corresponding Unicode scalar values.

On the ordering of UTF-EBCDIC sequences

trailing bytes << Leading bytes of 2-byte-sequence <<  .. .. .. <<
        Leading bytes of 7-byte sequences

for (i = 0; i < n; ++i) {
    byte1 = source1[i];
    byte2 = source2[i];
    if (byte1 == byte2) continue; // fast path

    // check for the single bytes vs multibytes
    if (shadow[byte1] < 2) {
        if (shadow[byte2] > 2) return - 1; // single bytes less than multi
    } else {
        if (shadow[byte2] < 2) return 1; // multibyte greater than single
    }

    // now the shadows are of the same type, so just compare the bytes
    if (byte1 < byte 2) return - 1;
    return 1;
}

However, if the desired order is to be the same order as Unicode values for all the characters, both the single-byte and the multi-byte characters, the intermediate I8-sequence bytes should be compared.  This approach also makes the comparison immune to any local customization of the mapping (see  On the Mapping .. ) and provides a consistent Unicode value order.  The following is a sample for the comparison code.

for (i = 0; i < n; ++i) {
    byte1 = source1[i];
    byte2 = source2[i];
    if (byte1 == byte2) continue; // fast path

    // compare the I8-sequence counterparts
    // take advantage of the ability of I8-sequence bytes being similar
    // to UTF-8 byte to preserve the same order as Unicode values
    // ebtoi8 is the reverse mapping vector from UTF-EBCDIC to I8 bytes
    if (ebtoi8[byte1] < ebtoi8[byte2]) return - 1;
    return 1;
}

Credits

Changes from previous revisions

Copyright