Unicode Technical Report #13

Unicode Newline Guidelines

Summary

This document describes guidelines for how to handle different characters used to represent CRLF and other representations of new lines on different platforms.

Status

This document contains informative material which has been considered and approved by the Unicode Technical Committee for publication as a Technical Report and as part of the Unicode Standard, Version 3.0. Any reference to version 3.0 of the Unicode Standard automatically includes this technical report. Please mail corrigenda and other comments to the author.

The content of all technical reports must be understood in the context of the appropriate version of the Unicode Standard. References in this technical report to sections of the Unicode Standard refer to the Unicode Standard, Version 3.0. See http://www.unicode.org/unicode/standard/versions for more information.

Contents

• 1 Introduction
• 2 Definitions
• 3 Background
• 4 Recommendations
  • 4.1 Converting from other character code sets
  • 4.2 Interpreting characters in text
  • 4.3 Converting to other code sets
  • 4.4 Input and Output
  • 4.5 Page Separator

1 Introduction

Newlines are represented on different platforms by carriage return (CR), line feed (LF), CRLF, or next line (NEL). Unfortunately, not only are newlines represented by different characters on different platforms, they also have ambiguous behavior even on the same platform. Especially with the advent of the web, where text on a single machine can arise from many sources, this causes a significant problem.

Unfortunately, these characters are often transcoded directly into the corresponding Unicode codes when a character set is transcoded; this means that even programs handling pure Unicode have to deal with the problems. For information on handling newlines in regular expressions, see UTR #18: Unicode Regular Expression Guidelines.

2 Definitions

The following table provides hexadecimal values for the acronyms used in the text. The Unicode Standard does not formally assign control characters, instead it provides the 65 code values for use as in the 7 and 8-bit standards. See The Unicode Standard, Version 2.0, Section 2.6 Controls and Control Sequences.

• There are two mappings of LF and NEL used by EBCDIC systems. The first EBCDIC column shows the MVS Open Edition (including CP1047) mapping of these characters, while the second column shows the CDRA mapping. This difference arises from the use of LF character as 'New Line' in ASCII-based Unix environments and in some data transfer protocols that use the Unix assumptions. The second column is based on the standardized definitions — both in ASCII and EBCDIC of LF.
• NEL is not actually defined in ASCII: it is defined in ISO 6429 as a C1 control.

For clarity, when referring to the function that a particular character has, we will use lowercase (e.g., paragraph separator); when referring to the specific characters that represent those functions, we will use titlecase or an acronym (e.g., Paragraph Separator or PS).]

The term NLF (new line function) stands for different characters depending on the platform; that is, any of CR, LF, CRLF, or NEL.

3 Background

A paragraph separator is used to indicate a separation between paragraphs, while a line separator indicates where a line break alone should occur, typically within a paragraph. For example:

This is a paragraph with a line separator at this point,
causing the word "causing" to appear on a different line, but not causing the typical paragraph indentation, sentence-breaking, line spacing, or change in flush (right, center or left paragraphs).

For comparison, line separators basically correspond to HTML <BR>, and paragraph separators to older usage of HTML <P> (modern HTML delimits paragraphs by enclosing them in <P>...</P>). In word processors, paragraph separators are usually entered using a keyboard RETURN or ENTER; line separators are usually entered using a modified RETURN or ENTER, such as SHIFT-ENTER.

A record separator is used to separate records. For example, when exchanging tabular data, a common format is to tab-separate the cells, and use a CRLF at the end of a line of cells. This function is not precisely the same as line separation, but the same characters are often used.

Traditionally, NLF started out as a line separator (and sometimes record separator). It is still used as a line separator in simple text editors such as program editors. As platforms and programs started to handle word processing with automatic line-wrap, these characters were reinterpreted to stand for paragraph separators. For example, even such simple programs as the Windows Notepad program or the Mac SimpleText program interpret their platform's NLF as a paragraph separator, not a line separator.

Once NLF was reinterpreted to stand for a paragraph separator, in some cases some other control character was impressed into service as a line separator. For example, vertical tabulation VT is used in Microsoft Word. However, the choice of character for line separator is even less standardized than the choice of character for NLF.

Yet, many internet protocols and a lot of existing text treats NLF as a line separator, so you can't just simply treat NLF as a paragraph separator in all circumstances. 

4 Recommendations

The Unicode Standard defines two unambiguous separator characters, Paragraph Separator (PS = 2029₁₆) and Line Separator (LS = 2028₁₆). In Unicode text, the PS and LS characters should be used wherever the desired function is unambiguous. Otherwise, the following specifies how to cope with an NLF when converting from other character sets to Unicode, when interpreting characters in text, and when converting from Unicode to other character sets.

Note: Even if you know which characters represents NLF on your particular platform, on input and in interpretation, treat CR, LF, CRLF, and NEL the same. Only on output do you need to distinguish between them.

4.1 Converting from other character code sets

1. If you do know the exact usage of any NLF, then convert it to LS or PS.
2. If you don't know the exact usage of any NLF, remap it to your platform NLF. (This doesn't really help you in interpreting Unicode text unless you are the only source of that text, since someone else may have left in LF, CR, CRLF, or NEL.)

4.2 Interpreting characters in text

1. Always interpret PS as paragraph separator and LS as line separator.
2. In word processing, interpret any NLF the same as PS.
3. In simple text editors, interpret any NLF the same as LS.
4. In parsing, choose the safest interpretation. For example, if you are dealing with sentence-break heuristics, you would reason in the following way that it is safer to interpret any NLF as a LS:
   • Suppose you misinterpret an NLF as LS, when it was meant to be PS. Since most paragraphs are terminated with punctuation anyway, in only a few cases would this cause misidentification of sentence boundaries.
   • Suppose you misinterpret an NLF as PS, when it was meant to be LS. In this case, line breaks would cause sentence breaks, which would mess up the sentence break heuristics significantly.

4.3 Converting to other character code sets

1. If you know the intended target, map NLF, LS, and PS appropriately, depending on the target conventions. For example, when mapping to Microsoft Word's internal conventions for Windows documents you would map LS to VT, and PS and any NLF to CRLF.
2. If you don't know the intended target, map NLF, LS, and PS to the platform newline convention (CR, LF, CRLF, or NEL). In Java, for example, this is done by mapping to a string nlf, defined as:
   String nlf = System.getProperties("line.separator");

4.4 Input and Output

1. A readline function should stop at NLF, LS, FF, or PS. In the typical implementation it does not include the NLF, LS, PS, or FF that caused it to stop. Note that since the separator is lost, the use of readline is limited to text processing, where there is no difference among the flavors of separators.
2. A writeline (or newline) function should convert NLF, LS, and PS according to the conventions in §4.3 Converting to other character code sets.
3. In C, gets is defined to terminate at a newline and replaces the newline with '\0', while fgets is defined to terminate at a newline and includes the newline in the array it copies the data into. C implementations interpret '\n' either as LF or as the underlying platform newline NLF depending on where it occurs. EBCDIC C compilers substitute the relevant codes, based on the EBCDIC execution set.

4.5 Page Separator

FF is commonly used as a page separator, and it should be interpreted that way in text. When displaying on the screen, it causes the text after the separator to be forced to the next page. It should be independent of paragraph separation: a paragraph can start on one page and continue on the next page. Except when displaying on pages, in most parsing and in readline it is interpreted in the same way as a LS.

Copyright © 1998-1999 Unicode, Inc. All Rights Reserved. The Unicode Consortium makes no expressed or implied warranty of any kind, and assumes no liability for errors or omissions. No liability is assumed for incidental and consequential damages in connection with or arising out of the use of the information or programs contained or accompanying this technical report.

Unicode and the Unicode logo are trademarks of Unicode, Inc., and are registered in some jurisdictions.