UTF-EBCDIC

Summary

Status of this document

The content of all technical reports must be understood in the context of the appropriate version of the Unicode Standard. References in this technical report to sections of the Unicode Standard refer to the Unicode Standard, Version 3.0. See http://www.unicode.org/unicode/standard/versions for more information.

Contents

• 1 Scope
• 2 Description
• 3 Definitions
 
• 3.1 Step 1: UTF-8-Mod
• 3.2 Characteristics of the I8-sequence
• 3.3 Step 2: Byte Conversion
• 3.4 Shadow Flags
• 3.5 Signature
• 3.6 Where to Use UTF-EBCDIC?
• 4 Bibliography
• 5 Annex A: Intellectual Property Related
• 6 Annex B: Additional Information
 
• 6.1 Controls, Variants, and Invariants in EBCDIC
• 6.2 A Comparison of UTF-EBCDIC and UTF-8
• 6.3 FEFF, FFFE, and FFFF in UTF-EBCDIC
• 6.4 Normalization to Fixed Width
• 6.5 Mapping of Bytes in Step 2
• 6.6 Ordering of UTF-EBCDIC Sequences
• 7 Acknowledgements
• 8 Revisions

1 Scope

To address the use of Unicode character data in byte-oriented ASCII-based systems, the Unicode Standard (see section A.2 of the Unicode Standard) (also ISO/IEC 10646 -1, Amendment no. 2) has defined UTF-8. Use of UTF-8 permits existing ASCII-based systems that have hard-coded dependency on the encoding of the ASCII repertoire of characters to safely process the corresponding Unicode characters. There is a similar requirement to transform Unicode characters to a form that is safe for EBCDIC systems for the control characters and invariant characters.

This Technical Report defines the UTF-EBCDIC.

Neither UTF-EBCDIC nor its intermediate form called UTF-8-Mod in this technical report, are intended to be used in open interchange environments. It is useful in homogeneous EBCDIC systems and networks.

2 Description

• Conversion of the Unicode scalar values to a variable length byte sequence called I8-sequence (intermediate 8-bit sequence) by applying a modified UTF-8 transformation (UTF-8-Mod), enabling the preservation of 65 control characters as single bytes.


Valid pairs of surrogates (see Section 3.7, Surrogates, in the Unicode Standard 2.0) must be converted first to their corresponding Unicode scalar values by applying the UTF-16 transformation. Unicode scalar values in the range X'10000' to X'10FFFF' obtained from I8-sequences, are transformed into the corresponding surrogate pairs using the UTF-16 transformation.
 

• The bytes in the I8-sequence are then converted to the UTF-EBCDIC byte sequence by using a single-byte to single-byte reversible conversion.

Note: The following notation is used in this Technical Report: X'nn .. mm' represent hexadecimal values; <bb...bb> represent values in bit notation; U+abcd represents a Unicode character.

3 Definition

3.1 Step 1: UTF-8-Mod

When these I8-sequence bytes are converted to the UTF-EBCDIC form, the corresponding 65 EBCDIC control characters and 95 EBCDIC graphic characters are preserved as single bytes in the UTF-EBCDIC byte sequence. The 95 EBCDIC graphic characters include 82 invariant (occupy the same code position) characters (including SPACE) across most EBCDIC single-byte code pages and 13 variant ASCII graphic characters (occupy varying code positions). Positions assigned to EBCDIC controls, the invariant graphic characters and the variant graphics are shown in Table B.1.

Furthermore, the values X'00'...X'9F' do not appear in any byte of an I8-sequence except as the direct representation of U+0000 to U+009F. Each Unicode scalar value that is not a part of a valid surrogate pair is represented in an I8-sequence by 1, 2, 3 or 4 bytes, depending on the value. A valid surrogate pair is first converted to its corresponding Unicode scalar value, which then maps into either 4 bytes or 5 bytes, depending on the value.

The UTF-8-Mod transformation is intended to be used only as an intermediate step in arriving at UTF-EBCDIC. It is not intended to be used elsewhere.

The I8-sequence is a variable length encoding of Unicode characters as 8-bit byte sequences, where the high bits of each byte indicate which part of the sequence a byte belongs to. Table 1 shows how the bits in a Unicode scalar value (or a valid surrogate pair) are distributed among the bytes in the I8-sequence. I8-sequence corresponding to a valid surrogate pair is also shown, including the UTF-16 transformation to convert the valid pair to the corresponding Unicode scalar value.
 

A valid surrogate pair -- a high surrogate from the range X'D800' to X'DBFF' followed by a low surrogate from the range X'DC00' to X'DFFF' -- must be converted to its corresponding Unicode scalar value in the range X'10000' to X'10FFFF', using the UTF-16 transformation. The following table shows the correspondence between the bit patterns of the surrogate pairs and the corresponding I8-sequence bytes.
 

where ^a uuuu = 000pp -1, or ^b uuuu = qpppp -1
(to account for addition of 10000₁₆ as in Section 3.7, Surrogates, in the Unicode Standard 2.0)

When converting from I8-sequences to Unicode scalar values, however, implementations do not need to check that the shortest number of bytes is being used, which simplifies the conversion algorithm.

3.2 Characteristics of the I8-sequence

• Some of the important characteristics of I8-sequence are:

• Unicode characters from U+0000 to U+009F (ASCII repertoire, C0 and C1 controls) map to single-byte I8-sequence values X'00' to X'9F' (ASCII values X'00' to X'7F' and ISO/IEC 4873 values X'80' to X'9F'). ASCII values or ISO/IEC 4873 control values do not otherwise occur in an I8-sequence. This paves the way for transforming these into corresponding single-byte EBCDIC controls and graphics in the second step of UTF-EBCDIC transform.
• The I8-sequence is reasonably compact in terms of number of bytes used for encoding. It is very simple and efficient to convert to and from Unicode text.
• The first byte indicates the number of bytes to follow in a multi-byte sequence. This allows for efficient forward parsing. It is also efficient to find the start of a character string from an arbitrary location in a byte stream. You need to search at most five bytes (seven bytes, if the full range of 31 bits of ISO/IEC 10646 is considered) backwards, and it is simple to recognize an initial byte. For example, after converting a UTF-EBCDIC byte back into I8-sequence, in C

isInitialByte = ( (byte & 0xE0) != 0xA0);

The search for initial or trailing bytes can also be done directly on UTF-EBCDIC byte by utilizing a shadow vector (see Table 4 described later).

3.3 Step 2: Byte Conversion

The 64 control characters (U+0000 to U+001F, U+0080 to U+009F), the ASCII DELETE character (U+007F), the 95 ASCII graphic characters (including the SPACE character) (U+0020 to U+007E) are mapped respecting EBCDIC conventions, as defined in IBM Character Data Representation Architecture, CDRA, with one exception -- the pairing of EBCDIC Line Feed and New Line control characters are swapped from their CDRA default pairings to ISO/IEC 6429 Line Feed (U+000A) and Next Line (U+0085) control characters (to be in line with IBM OS/390 UNIX Services, or Open MVS practice and preference, stemming from the hard-coding of X'0A' as the New Line in most ASCII-C compilers.).

The map preserves the invariance for a set of 82 graphic characters (including SPACE) (known as the IBM Syntactic Graphic Character set), and maintains consistency with the IBM MVS Open Systems Code page (CPGID 1047) for the variant characters from within the ASCII repertoire.

The remaining 96 bytes of EBCDIC 8-bit structure are allocated to X'A0' to X'FF' -- the trailing bytes and leading bytes of the I8-sequence.(from Table 1). The minimum criterion for allocation of these bytes is that it provides for a reversible map.

The trailing and leading bytes (X'A0' to X'FF' of the I8-sequence) are paired with the unassigned UTF-EBCDIC bytes in increasing order. Table 2 and Table 3 show the byte maps between the I8-sequence bytes and UTF-EBCDIC bytes in the forward and reverse directions respectively. The resulting UTF-EBCDIC multi-byte sequences will be in the same lexical (numerical) order as their corresponding Unicode scalar values (when the sequences are zero-filled to equal number of bytes and compared with each other). Please note that the UTF-EBCDIC single-byte values, however, will not be in the same order as their corresponding Unicode scalar values.

The resulting UTF-EBCDIC byte sequence can be transparently processed in most EBCDIC systems. It also retains all the characteristics (see the section 3.2 Characteristics of the I8-sequence above) of I8-sequence mentioned earlier. Since EBCDIC code page definitions have 13 variants (and only 82 invariants) the choice of the above byte map for the graphic characters has been made to accommodate the MVS Open Systems environment for standardization purposes.
 

3.4 Shadow Flags

3.5 Signature

3.6 Where to Use UTF-EBCDIC?

4 Bibliography

• The Unicode Standard Version 2.0: The Unicode Consortium ISBN 0-201-48345-9, Addison Wesley Developers Press, July 1996.
• CDRA: IBM - Character Data Representation Architecture - Reference and Registry, SC09-2190-00, December 1996.
• ISO/IEC 10646-1: 1993(E): Information Processing - Universal Coded Character Set (UCS):Part 1, Basic Multilingual Plane
• Amendment 1 to ISO/IEC 10646-1: Transformation Format for 16 Planes of Group 00 (UTF-16); 1996
• Amendment 2 to ISO/IEC 10646-1: Transformation Format 8 (UTF-8)
• ISO/IEC 646: Information Processing - 7-Bit Coded Character Set for Information Interchange
• ASCII - ANSI Standard X3.4; also the International Referenc Version of ISO/IEC 646 - 1993
• ISO/IEC 4873: Information Processing - 8-Bit Code for Information Interchange - Structure and Rules for implementation
• ISO/IEC 6429: Information Processing - 7-Bit and 8-Bit Coded Character Sets - Control Functions for Coded Character Sets
• ISO/IEC 8859-xx: Information Processing - 8-Bit Single-Byte Coded Graphic Character Sets (several parts)
• SHARE Report SSD No. 366: ASCII and EBCDIC Character Set and Code Issues in Systems Application Architecture, The ASCII/EBCDIC Character Set Task Force. Edited by Edwin Hart, The Johns Hopkins University, Applied Physics Laboratory, Laurel, Maryland, USA; published by Share Inc., 111 East Wacker Drive, Chicago, Illinois, USA 60601; June 1989

5 Annex A: Intellectual Property Related

IBM LOGO

International Business Machines Corporation Route 100

Somers, NY 10589
 

June 2, 1998
 
 
 

The Chair, Unicode Technical Committee
 

Subject: Disclosure of IBM Technology - EBCDIC-Friendly UCS Transformation Format (EF-UTF)
 

The attached document entitled "EBCDIC-Friendly UCS Transformation Format (EF-UTF)" contains IBM technology that has been filed for application for Canadian Patent. However, IBM believes that the technology could be beneficial to the EBCDIC community at large; allowing the community to derive the enormous benefits provided by UCS (ISO/IEC 10646 and Unicode).
 

This letter is to inform you that IBM is pleased to make the attached documentation, and the associated technology that has been filed for patent, freely available to anyone concerned towards making the transformation format as part of the UCS standards.
 
 
 

Sincerely
 

SIGNED
 

Elizabeth G. Nichols
Director of National Language Support
and Information Development
 

EGN:ghs

Attachment
 
 

(Note: The term EF-UTF has been changed to UTF-EBCDIC at the suggestion of UTC meting 78 -- V.S. Umamaheswaran)

6 Annex B: Additional Information

6.1 Controls, Variants, and Invariants in EBCDIC

6.2 A comparison of UTF-EBCDIC and UTF-8

6.3 FEFF, FFFE, and FFFF in UTF-EBCDIC

• X'FE' X'FF', X'FF' X'FE' and X'FF' X'FF' in the I8-sequence

• X'FE' X'FF', X'FF'X'FE' and X'FF' X'FF' in the UTF-EBCDIC sequence

The I8-sequence to UTF-EBCDIC byte mappings are: X'FE' to X'FD', and X'FF' to X'FE' (see Table 2). The values X'FE' and X'FF' can be generated in a UTF-EBCDIC byte sequence from I8-sequence values by mapping X'FF' to X'FE' and X'9F' to X'FF' from Table 2).

The X'9F' is assigned to the control character -- Application Program Command (APC) -- in ISO-8 C1. According to ISO/IEC 6429, the APC is followed by a parameter string using bit combinations from 0/8 to 0/13 (X'08' to X'0D') and 2/0 to 7/14 (X'20' to X'7E') and terminated by the control function String Terminator (ST) (coded at X'9C' in C1). Therefore, the sequence X'FF' X'FF', the equivalent of two APC controls without intervening parameters or ST-s, also should not appear in UTF-EBCDIC sequence. None of the valid parameter bit combinations can generate a 7-byte I8 sequence that starts with X'FF'. So the sequence X'FF' X'FE' also cannot appear in a UTF-EBCDIC sequence.

6.4 Normalization to Fixed Width

However, this would be possible only if processing is tolerant to native Unicode encoding. If transparency to EBCDIC invariance and controls is needed also in the normalized form, then Unicode cannot be directly used for normalization. It can be seen from Table 1 that the last code position in the BMP (U+FFFF) requires four bytes in the I8-sequence and in the corresponding UTF-EBCDIC sequence. A 32-bit integer can be used for normalization of up to four-byte UTF-EBCDIC sequences.

The maximum Unicode scalar value that a four-byte I8-sequence or UTF-EBCDIC sequence can represent is:
 

corresponding to the end of plane 3 in group 0. Using UTF-16 to represent planes 1 to 16, the surrogate characters in the BMP can be used. By treating the surrogate characters as any other BMP characters, up to plane 16 can be encoded using the 16-bit form, and hence can be contained within the 32-bit normalized form of UTF-EBCDIC. Care has to be taken to correctly process the corresponding UTF-EBCDIC sequence corresponding to the surrogate pairs, similar to dealing with combination sequences. When it is desirable to convert valid surrogate pairs into corresponding Unicode scalar value and then apply UTF-EBCDIC, only up to plane 3 can be contained within the 32-bit normalized value. For all values beyond group 0, plane 3 of UCS, the UTF-EBCDIC will contain more than four octets. The normalization for these cases will need 64 bits (assuming nothing between 32 and 64 bits is practical).

6.5 Mapping of Bytes in Step 2

Similarly the pairing of I8-sequence bytes and UTF-EBCDIC sequence bytes could be done in multiple ways. The simplest requirement on this byte-pairing is that it should be unique and reversible. The pairing adopted in this version of the UTR is based on the request from Oracle Corporation's representative Mr. Jianping Yang -- to be able to maintain the order of the UTF-EBCDIC multi-byte sequences the same as the order of the corresponding Unicode scalar values.

6.6 Ordering of UTF-EBCDIC Sequences

The byte values within each set are ordered in increasing order. Note that the UTF-EBCDIC single-bytes do not have this property - either among themselves or between themselves and the bytes of the multi-byte UTF-EBCDIC sequences. The single-bytes are ordered according to their CP1047 order. So doing a "binary comparison" of the text would look like:
 

The resulting order is a mix of EBCDIC CP1047 order for the single bytes and Unicode order for the multi-byte UTF-EBCDIC characters.

However, if the desired order is to be the same order as Unicode scalar values for all the characters, both the single-byte and the multi-byte characters, the intermediate I8-sequence bytes should be compared. This approach also makes the comparison immune to any local customization of the mapping (see Mapping of Bytes in Step 2) and provides a consistent Unicode value order. The following is a sample for the comparison code.
 

If the desire is to preserve the EBCDIC order for the single-bytes (the ASCII repertoire) or the traditional order of the multi-byte sequences (such as for EBCDIC-Japanese, EBCDIC-Cyrillic, EBCDIC-Arabic etc.) localization resources such as a weight look up table in locales should be employed.

6 Acknowledgments

7 Revisions

Copyright © 1999 Unicode, Inc.. All Rights Reserved. The Unicode Consortium makes no expressed or implied warranty of any kind, and assumes no liability for errors or omissions. No liability is assumed for incidental and consequential damages in connection with or arising out of the use of the information or programs contained or accompanying this technical report.