| Version: | 6.2.0 (build 261) |
|---|---|
| Revised: | Aug. 28, 2012 |
Overview
This file contains a description of the syntax for Nameslist (*.lst) files supported by the Unibook Character Browser. A syntax compatible, but not textually identical version of this document is available from Unicode, Inc. under the UCD Terms of Use. The version number of the current file is that of the most recent version of Unibook at the time this file was updated. That version may differ slightly from the corresponding version of the Unicode Standard.
1.0 Introduction
The Unicode name list file NamesList.txt (also NamesList.lst) is a plain text file used to drive the layout of the character code charts in the Unicode Standard. The information in this file is a combination of several fields from the UnicodeData.txt and Blocks.txt files, together with additional annotations for many characters.
This document describes the syntax rules for the file format, but also gives brief information on how each construct is rendered when laid out for the code charts. Some of the syntax elements are used only in preparation of the drafts of the code charts and are not present in the final, released form of the NamesList.txt file.
Over time, the syntax has been extended by adding new features. The syntax for formal aliases and index tabs was introduced with Unicode 5.0. The syntax for marginal sidebar comments is utilized extensively in draft versions of the NamesList.txt file. The support for UTF-8 encoded files and the syntax for the UTF-8 charset declaration in a comment at the head of the file were introduced after Unicode 6.1.0 was published, as was the syntax for the specification of variation sequences and alternate glyphs and their respective summaries.
The same input file can be used for the preparation of drafts and final editions for ISO/IEC 10646. Earlier versions of that standard used a different style, referred below as ISO-style. That style necessitated the presence of some information in the name list file that is not needed (and in fact removed during parsing) for the Unicode code charts.
With access to the layout program (Unibook) it is a simple matter of creating name lists for the purpose of formatting working drafts or other documents containing proposed characters.
The content of the NamesList.txt file is optimized for code chart creation. Some information that can be inferred by the reader from context has been suppressed to make the code charts more readable. See the chapter on Code Charts in the Unicode Standard.
1.1 NamesList File Overview
The NamesList files are plain text files which in their most simple form look like this:
@@<tab>0020<tab>BASIC LATIN<tab>007F
; this is a file comment (ignored)
0020<tab>SPACE
0021<tab>EXCLAMATION MARK
0022<tab>QUOTATION MARK
. . .
007F<tab>DELETE
The semicolon (as first character), @ and <tab> characters are used by the file syntax and must be provided as shown. Hexadecimal digits must be in UPPERCASE. A double @@ introduces a block header, with the title, and start and ending code of the block provided as shown.
For a minimal name list, only the NAME_LINE and BLOCKHEADER and their constituent syntax elements are needed.
The full syntax with all the options is provided in the following sections.
2.0 NamesList File Structure
This section defines the overall file structure
NAMELIST: TITLE_PAGE* EXTENDED_BLOCK* TITLE_PAGE: TITLE | TITLE_PAGE SUBTITLE | TITLE_PAGE SUBHEADER | TITLE_PAGE IGNORED_LINE | TITLE_PAGE EMPTY_LINE | TITLE_PAGE NOTICE_LINE | TITLE_PAGE COMMENT_LINE | TITLE_PAGE PAGEBREAK | TITLE_PAGE FILE_COMMENT | FILE_COMMENT EXTENDED_BLOCK: BLOCK | BLOCK SUMMARY BLOCK: BLOCKHEADER | BLOCKHEADER INDEX_TAB | BLOCK CHAR_ENTRY | BLOCK SUBHEADER | BLOCK NOTICE_LINE | BLOCK EMPTY_LINE | BLOCK IGNORED_LINE | BLOCK SIDEBAR_LINE | BLOCK PAGEBREAK | BLOCK FILE_COMMENT | BLOCK CROSS_REF CHAR_ENTRY: NAME_LINE | RESERVED_LINE | CHAR_ENTRY ALIAS_LINE | CHAR_ENTRY FORMALALIAS_LINE | CHAR_ENTRY COMMENT_LINE | CHAR_ENTRY CROSS_REF | CHAR_ENTRY DECOMPOSITION | CHAR_ENTRY COMPAT_MAPPING | CHAR_ENTRY IGNORED_LINE | CHAR_ENTRY EMPTY_LINE | CHAR_ENTRY NOTICE_LINE | CHAR_ENTRY FILE_COMMENT | CHAR_ENTRY VARIATION_LINE
In other words:
Neither TITLE nor SUBTITLE may occur after the first BLOCKHEADER.
Only TITLE, SUBTITLE, SUBHEADER, PAGEBREAK, COMMENT_LINE, NOTICE_LINE, EMPTY_LINE, IGNORED_LINE and FILE_COMMENT may occur before the first BLOCKHEADER.
- CROSS_REF, DECOMPOSITION, COMPAT_MAPPING, VARIATION_LINE, ALIAS and FORMALALIAS_LINE lines occurring before the first block header are treated as if they were COMMENT_LINEs.
Directly following either a NAME_LINE or a RESERVED_LINE an uninterrupted sequence of the following lines may occur (in any order and repeated as often as needed): ALIAS_LINE, CROSS_REF, DECOMPOSITION, COMPAT_MAPPING, FORMALALIAS_LINE, NOTICE_LINE, EMPTY_LINE, IGNORED_LINE, VARIATION_LINE and FILE_COMMENT.
- The conventional order of elements in a char entry: NAME_LINE, FORMALALIAS_LINE, ALIAS, COMMENT_LINE or NOTICE_LINE, CROSS_REFs, VARIATION_LINE, and optionally ending in either DECOMPOSITION or COMPAT_MAPPING is not enforced by the layout program (Unibook).
Except for CROSS_REF, NOTICE_LINE, SIDEBAR_LINE, EMPTY_LINE, IGNORED_LINE and FILE_COMMENT, none of these lines may occur in any other place.
- A NOTICE_LINE or CROSS_REF displays differently depending on whether it follows a header or title or is part of a CHAR_ENTRY
A PAGEBREAK may appear anywhere, except the middle of a CHARACTER_ENTRY. A PAGEBREAK before the file title lines may not be supported. INDEX_TABs may appear after any block header.
If the first line a of a file is a file comment, it may contain a UTF-8 charset declaration (see below). Alternatively, or in addition, a BOM may be present at the very beginning of the file, forcing the encoding to be interpreted as UTF-16 (little-endian only) or UTF-8.
Several of these elements, while part of the formal definition of the file format, do not occur in final published versions of the names list.
Blocks followed by Summaries
A block may be extended by a summary of standard variation sequences or selected alternate glyphs defined for characters in the block:
SUMMARY: ALTGLYPH_SUMMARY | VARIATION SUMMARY | ALTGLYPH_SUMMARY VARIATION_SUMMARY ALTGLYPH_SUMMARY: ALTGLYPH_SUBHEAD | ALTGLYPH_SUMMARY SUBHEAD | ALTGLYPH_SUMMARY NOTICE_LINE | ALTGLYPH_SUMMARY FILE_COMMENT | ALTGLYPH_SUMMARY EMPTY_LINE VARIATION_SUMMARY: VARIATION_SUBHEAD | VARIATION_SUMMARY SUBHEAD | VARIATION_SUMMARY NOTICE_LINE | VARIATION_SUMMARY FILE_COMMENT | VARIATION_SUMMARY EMPTY_LINE
When formatted for display, each summary will recap the information presented in the VARIATION_LINE elements of the preceding block, grouped by alternate glyph variants and standardized variation sequences, and preceded by the corresponding subhead. Additional SUBHEAD and NOTICE lines, if provided, immediately follow the ALTGLYPH_SUBHEAD or VARIATION_SUBHEAD. There is no provision to provide subheads that are interspersed between items in the summary.
These syntax constructs are entirely optional. If the ALTGLYPH_SUBHEAD or VARIATION_SUBHEAD are omitted from the names list, but the preceding block nevertheless contains VARIATION_LINE elements as described below, Unibook will automatically generate any required summaries using a default format for the headers.
Thus, the main purpose for providing ALTGLYPH_SUBHEAD or VARIATION_SUBHEAD elements would be to provide specific contents for these summary titles as well as allow the ability to add additional information via SUBHEAD and NOTICE elements. The final published version of the Unicode names list is machine generated and will always explicitly provide any summary subheads.
2.1 NamesList File Elements
This section provides the details of the syntax for the individual elements.
ELEMENT SYNTAX // How rendered
NAME_LINE: CHAR TAB NAME LF
// The CHAR and the corresponding image are echoed,
// followed by the name as given in NAME
| CHAR TAB "<" LCNAME ">" LF
// Control and noncharacters use this form of
// lowercase, bracketed pseudo character name
| CHAR TAB NAME SP COMMENT LF
// Names may have a comment, which is stripped off
// unless the file is parsed for an ISO style list
| CHAR TAB "<" LCNAME ">" SP COMMENT LF
// Control and noncharacters may also have comments
RESERVED_LINE: CHAR TAB "<reserved>" LF
// The CHAR is echoed followed by an icon for the
// reserved character and a fixed string e.g. "<reserved>"
COMMENT_LINE: TAB "*" SP EXPAND_LINE
// * is replaced by BULLET, output line as comment
| TAB EXPAND_LINE
// Output line as comment
ALIAS_LINE: TAB "=" SP LINE
// Replace = by itself, output line as alias
FORMALALIAS_LINE:
TAB "%" SP NAME LF
// Replace % by U+203B, output line as formal alias
CROSS_REF: TAB "x" SP CHAR SP LCNAME LF
| TAB "x" SP CHAR SP "<" LCNAME ">" LF
// x is replaced by a right arrow
| TAB "x" SP "(" LCNAME SP "-" SP CHAR ")" LF
| TAB "x" SP "(" "<" LCNAME ">" SP "-" SP CHAR ")" LF
// x is replaced by a right arrow;
// (second type as used for control and noncharacters)
// In the forms with parentheses the "(","-" and ")" are removed
// and the order of CHAR and LCNAME is reversed;
// i.e. all inputs result in the same order of output
| TAB "x" SP CHAR LF
// x is replaced by a right arrow
// (this type is the only one without LCNAME
// and is used for ideographs)
VARIATION_LINE: TAB "~" SP CHAR VARSEL SP LABEL LF
| TAB "~" SP CHAR VARSEL SP LABEL "(" LCNAME ")"LF
// output standardized variation sequence or simply the char code in case of alternate
// glyphs, followed by the alternate glyph or variation glyph and the label and context
FILE_COMMENT: ";" LINE
EMPTY_LINE: LF
// Empty and ignored lines as well as
// file comments are ignored
IGNORED_LINE: TAB ";" EXPAND_LINE
// Skip ';' character, ignore text
SIDEBAR_LINE: ";;" LINE
// Skip ';;' characters, output line
// as marginal note
DECOMPOSITION: TAB ":" SP EXPAND_LINE
// Replace ':' by EQUIV, expand line into
// decomposition
COMPAT_MAPPING: TAB "#" SP EXPAND_LINE
| TAB "#" SP "<" LCTAG ">" SP EXPAND_LINE
// Replace '#' by APPROX, output line as mapping;
// check the <tag> for balanced < >
NOTICE_LINE: "@+" TAB LINE
// Skip '@+', output text as notice
| "@+" TAB * SP LINE
// Skip '@', output text as notice
// "*" expands to a bullet character
// Notices following a character code apply to the
// character and are indented. Notices not following
// a character code apply to the page/block/column
// and are italicized, but not indented
SUBTITLE: "@@@+" TAB LINE
// Skip "@@@+", output text as subtitle
SUBHEADER: "@" TAB LINE
// Skip '@', output line as text as column header
VARIATION_SUBHEAD:
"@~" TAB LINE
// Skip '@', output line as text as column header
| "@~"
// Output a default standard variation sequences summary subheader
ALTGLYPH_SUBHEAD:
"@@~" TAB LINE
// Skip '@', output line as text as column header
| "@@~"
// Output a default alternate glyph summary subheader
BLOCKHEADER: "@@" TAB BLOCKSTART TAB BLOCKNAME TAB BLOCKEND LF
// Skip "@@", cause a page break and optional
// blank page, then output one or more charts
// followed by the list of character names.
// Use BLOCKSTART and BLOCKEND to define
// what characters belong to a block.
// Use BLOCKNAME in page and table headers
BLOCKNAME: LABEL
| LABEL SP "(" LABEL ")"
// If an alternate label is present it replaces
// the BLOCKNAME when an ISO-style names list is
// laid out; it is ignored in the Unicode charts
BLOCKSTART: CHAR // First character position in block
BLOCKEND: CHAR // Last character position in block
PAGEBREAK: "@@" // Insert a (column) break
INDEX_TAB: "@@+" // Start a new index tab at latest BLOCKSTART
TITLE: "@@@" TAB LINE
// Skip "@@@", output line as text
// Title is used in page headers
EXPAND_LINE: {ESC_CHAR | CHAR | STRING | ESC+}+ LF
// Instances of CHAR (see Notes) are replaced by
// CHAR NBSP x NBSP where x is the single Unicode
// character corresponding to CHAR.
// If character is combining, it is replaced with
// CHAR NBSP <circ> x NBSP where <circ> is the
// dotted circle
Notes:- Blocks must be aligned on 16-code point boundary and contain an integer multiple of 16-code point columns. The exception to that rule is for blocks of ideographs, etc., for which no names are listed in the file. The BLOCKEND for such blocks must correspond to the last assigned character, and not the actual end of the block.
- Blocks must be non-overlapping and in ascending order. NAME_LINEs must be in ascending order and follow the block header for the block to which they belong.
- Reserved entries are optional, and will normally be supplied automatically. They are required whenever followed by ALIAS_LINE, COMMENT_LINE, NOTICE_LINE or CROSS_REF.
- The French version of the names list uses French rules, which allow apostrophe and accented letters in character names.
- An empty alternative glyph summary subheader expression will result in default header "Selected Alternative Glyphs"
- An empty standard variation subheader expression will result in the default header "Standardized Variation Sequences"
- When displaying a VARIATION_LINE for alternate glyphs, the "ALTn" selector is not displayed.
- If a glyph is unavailable for the variant glyph in a VARIATION_LINE it is replaced by the glyph for LIGHT SCREEN.
2.2 NamesList File Primitives
The following are the primitives and terminals for the NamesList syntax.
LINE: STRING LF
COMMENT: "(" LABEL ")"
| "(" LABEL ")" SP "*"
| "*"
NAME: <sequence of uppercase ASCII letters, digits, space and hyphen>
LCNAME: <sequence of lowercase ASCII letters, digits space and hyphen>
| LCNAME "-" CHAR
LCTAG: <sequence of ASCII letters>
STRING: <sequence of Latin-1 characters, except controls>
LABEL: <sequence of Latin-1 characters, except controls, "(" or ")">
VARSEL: CHAR
| ALT ( "1"|"2"|"3"|"4"|"5"|"6"|"7"|"8"|"9" )
CHAR: X X X X
| X X X X X
| X X X X X X
X: "0"|"1"|"2"|"3"|"4"|"5"|"6"|"7"|"8"|"9"|"A"|"B"|"C"|"D"|"E"|"F"
ESC_CHAR: ESC CHAR
ESC: "\"
// Special semantics of backslash (\) are supported
// only in EXPAND_LINE.
TAB: <sequence of one or more ASCII tab characters 0x09>
SP: <ASCII 20>
LF: <any sequence of ASCII 0A and 0D>
Notes:
- Multiple or leading spaces, multiple or leading hyphens, as well as word-initial digits in NAMEs or LCNAMEs are illegal.
- When names containing code points are lowercased to make them LCNAMEs, the code point values remain uppercase. Such code points by convention follow a hyphen and are the last element in the name.
- Special lookahead logic prevents a 4 digit number for a standard, such as ISO 9999 from being misinterpreted as ISO CHAR. Currently recognized are "ISO", "DIN", "IEC" and "S X" as well as "S C" for the JIS X and JIS C series of standards. For other standards, or for four-digit years in a comment, use a NOTICE_LINE instead, which prevents expansion, or use '\" to escape the digits.
- Single and double straight quotes in an EXPAND_LINE are replaced by curly quotes using English rules. Smart apostrophes are supported, but nested quotes are not. Single quotes can only be applied around a single word.
- A CHAR inside ' or " is expanded, but only its glyph image is printed, the code value is not echoed.
- Inside an EXPAND_LINE, backslash is treated as an escape character that removes the special meaning of any literal character and also prevents the following digit sequence from being expanded. A backslash character in isolation is never displayed. A sequence of two backslash characters results in display of a single backslash, but has no effect on the interpretation of following characters.
- The hyphen in a character range CHAR-CHAR is replaced by an EN DASH on output.
- The NamesList.txt file is encoded in UTF-8 if the first line is a FILE_COMMENT containing the declaration "UTF-8" or any casemap variation thereof. Otherwise the file is encoded in Latin-1 (older versions). Beyond detecting the charset declaration (typically: "; charset=utf-8") the remainder of that comment is ignored. Even when the file is encoded as UTF-8, the character repertoire for running text (anything other than CHAR) is effectively restricted to the repertoire of Latin-1.
- The code chart layout program (Unibook) can accept files in several other formats. These include little-endian UTF-16, prefixed with a BOM, or UTF-8 prefixed with the UTF-8 BOM.
- While the format allows multiple <tab> characters, by convention the actual number of tabs is always one or two, chosen to provide the best layout of the plaintext file.
- Earlier published versions of the NamesList file may contain trailing or otherwise extraneous spaces or tab characters; while these are errors in the files, they are not being corrected, to retain stability of the published versions. Anyone writing a parser for older versions of this file may need to be prepared to handle such exceptions.
- The final LF in the file must be present.
Modifications
Changes from the 2012-05-13 version:
- Edited the variation syntax definitions, description and corresponding notes for wording.
- Minor tweaks to the layout of BNF syntax, mostly adding tabs and | characters as needed.
- Fixed some typographical errors and minor inconsistencies.
Changes from the 2012-03-31 version:
- Added syntax for elements required by variation sequence and alternate glyph summaries.
- Edited and reformatted some notes for readability.
- Documented the permitted presence of CROSS_REF outside character entries within blocks. Such CROSS_REFS have been present in published names lists, but that information was missing in the syntax description. For an example see the Currency Symbols block in the code charts.
Changes from the 2010-06-23 version:
- Added description of UTF-8 charset declaration and file encoding.
- Removed constraint that LCTAG consist only of lowercase letters, because of the existence of the "noBreak" tag.
- Added definitions for ESC_CHAR and ESC primitives.
- Clarified interpretation of backslash escapes in EXPAND_LINE.
Changes from the 2007-04-03 version:
- Better aligned the rules section with the actual published files and behavior of existing parsers. This included fixing some obvious typos and clarifying some notes as well as the following changes, which are listed individually.
- Replaced instances of <tab> by TAB throughout.
- NAME_LINE for special names may have trailing COMMENTs including COMMENTs consisting entirely of "*".
- In CROSS_REF added the form without LCNAME, fixed the literal to the correct lowercase "x" and noted that LCNAME may have "<" and ">" around it in the data. Also added missing LF in the rules.
- Removed a redundant rule for BLOCKHEADER.
- Changed FORMALALIAS_LINE from LINE to NAME to match actual restriction on contents.
- Extended the documentation of lookahead logic for CHAR.
- Accounted for FILE_COMMENT in overall file structure.
Changes from the 2006-02-16 version:
- Decomp and xref lines must contain a SP followed by a CHAR, < > for compat tags must be balanced, invalid xrefs treated as comments. Xrefs must not contain uppercase characters, except following a "-" (code point part of CJK)
- Relaxed the restriction on lines starting with #, :, %, x and = on the TITLE_PAGE. These are now treated as comments.
Changes from the 2006-07-13 version:
- Added SIDEBAR_LINE.
- Noted that comments in NAME_LINEs must be preceded by SP.
- Provided additional information on allowable characters in names.
- Noted that CROSS_REF must contain a SP and CHAR, and that COMPAT_MAPPING must contain a SP and may contain a <tag>.
- Noted that LCNAME may contain upper case characters under exceptional circumstances.
Changes from the 1999-07-05 version:
- Added FORMALALIAS_LINE and INDEX_TAB to syntax.
- Fixed the list of lines that may appear before a blockheader by adding NOTICE_LINE.
- Minor fixes to the wording of several syntax definitions.
Changes from earlier versions:
- Fixed syntax to better reflect restrictions on characters in character and block names.
- Better document treatment of comments in block names, plus French name rules.
- Use of 4-6 digit hex notation is now supported.
Copyright © 1995-2012 ASMUS, Inc. All Rights Reserved. This version of Unibook is distributed by the Unicode Consortium under a license from ASMUS, Inc., subject to the end user license agreement shown during startup and viewable via the Help/About/License.. command. This documentation file may not be republished in full or in part, except for the purpose of reviewing Unibook. Unibook and ASMUS are trademarks of ASMUS, Inc. Unicode is a registered trademark of the Unicode Consortium. Microsoft Windows and Microsoft Office are trademarks of Microsoft Corporation. Other terms may be trademarks of their respective trademark owners, whether identified or not.