Characters recommended in both UTS#39 and MSR but excluded from the Root Zone or Reference LGR

This document has been submitted as a UTC document. For convenience in documenting the character list it is presented using an LGR template format. A few minor details of the boilerplate in that template may not be applicable in this context and should be disregarded.

The collection comprises 274 characters included in the Maximal Starting Repertoire [MSR] for the DNS Root Zone that are also recommended in UTS#39 but are not part of the Second Level Reference LGR [RefLGR], as well as the uppercase equivalents for 88 of them (Latin), plus 70 decimal digits excluded from the Reference LGR, for a total of 432 characters.

Recommendation

• Proposed:Uncommon_Use (415)—The majority of these 432 characters are proposed for reclassification to Uncommon_Use, based on the fact that the expert teams charged with reviewing them for the ICANN Root Zone LGR and Reference LGR for the Second Level could not come up with evidence that they are used in common everyday writing, even for minority languages in reasonably widespread use. Consequently, they declined to include them in the respective LGRs.
• Proposed:Technical (1)—In a few cases, a different type, such as Technical, is proposed for the character.
   
• Recommended (5)—In review, some of the characters are found to have documented use compatible with retaining their Identifier_Type of Recommended. These characters are tagged with Identifier_Type Recommended and no change is proposed.
• Review_Needed (11)—Some characters need additional review before a recommendation can be proposed, these are tagged as Review_Needed.

The table in Section 2, “Repertoire” lists the characters with their proposed or existing Identifier_Type values in the “Tags” column, together with references to the source of their classification, as well as any other information. Because this document only includes characters not added to and therefore not listed in [RefLGR], this source is not documented on the character level.

Background

There are over a thousand non-Han characters with Identifier_Type Recommended that are proposed for consideration of reclassification because they appear to fail reasonable criteria for being needed in identifiers. They come in two sets. For one set, an independent analysis [MSR] has found indications that they should have been considered Uncommon_Use, Obsolete or Technical based on information available at the time of encoding, including their uppercase equivalents and any native digits that can be considered obsolete. That set is discussed in another document. The second set contains characters that were tentatively retained as Recommended in the [MSR] but upon further review by local expert teams from the [RZ-LGR] project were found to not be needed for any language or minority language in reasonably widespread use. That second set is discussed here.

The initial analysis was carried out for the purposes of defining the allowed repertoire for IDN Top Level Domain names for the DNS Root Zone. There are some restrictions that are specific to the Root Zone, such as a prohibition on digits, so a follow-on effort determined how to relax these restrictions in a manner appropriate for the needs of Second-Level Domains. This resulted in the Second-Level Reference Label Generation Rules [RefLGR]. The characters listed in this document are those that were not added to the [RefLGR], for lack of evidence of their use in everyday common writing for any language or minority language in vigorous and reasonably widespread use.

The set of characters discussed in this document starts with all characters of Identifier_Type Recommended, subtracting any character disallowed in [MSR] and then subtracting any character included in [RefLGR]. Added to the set are any uppercase equivalents as well as any native digits in that were not added to the [RefLGR] due to not being in common use.

The implication here is that any character not included in the Reference LGR for lack of documented or identifiable usage should be considered Uncommon_Use for Unicode's default identifiers—until such time as independent evidence to the contrary is produced, or a different Identifier_Type is proposed as a better fit. Until then, in lack of a demonstrated use case, it seems not helpful to continue to suggest that these characters should be supported as recommended. This also applies to any sets of native digits for which local experts considered them uncommon_use for the purpose of identifiers.

Arriving at a precise cutoff for Uncommon_Use is difficult because there is no single source or perfect information on the use of writing systems, and the details of such use are changing over time. In the research cited, that determination started with the [EGIDS] classification as a proxy for the likely level of modern use of the writing system, but made further adjustments in expert review. Accordingly, this document suggests that the UTC should consider the published results of the cited research as one of the better sources of information available and only deviate from it on the basis of even better information.

All decisions for the classification of characters in [MSR], or inclusion in [RZ-LGR] and [RefLGR] are documented and sourced on the character level; the same is not true for Unicode's classification, so it is not easily possible to verify any of the decisions that underlie the classification published in UTS39. By first making the alignment proposed here, and then carefully documenting deviations, a positive side effect might be that the classification overall becomes more transparent and reviewable.

For further background on specific disposition for characters in the DNS Root Zone [RZ-LGR] and Second-Level Reference LGR [RefLGR] see the cited references and links therein.

Additional Notes

• U+0931 ऱ DEVANAGARI LETTER RRA is part of the Root Zone and Reference LGR via sequence (does not occur standalone)
• U+09BC  ়  BENGALI SIGN NUKTA is part of the Root Zone and Reference LGR via sequence (does not occur standalone)
• U+0DA6 ඦ SINHALA LETTER SANYAKA JAYANNA is part of the Root Zone and Reference LGR via sequence (does not occur standalone)
• U+0E45 ๅ THAI CHARACTER LAKKHANGYAO is part of the Root Zone and Reference LGR via sequence (does not occur standalone)
• U+1063  ၣ  MYANMAR TONE MARK SGAW KAREN HATHI is part of the Root Zone and Reference LGR via sequence (does not occur standalone)

Although they are not listed as standalone, they are part of the set of code points used in the repertoire of Root Zone and Reference LGR and therefore no change in their existing Identifier_Type is proposed, and they are not listed below.

Discussion and Review

Domain names are an important and deliberately conservative set of identifiers. That said, there may be other classes of identifiers that don't require the same level of restrictions, so this proposal should not be understood to suggest that default Identifiers must be restricted to only those characters that are being recommended for IDNs. Rather, the purpose is to bring the facts discovered during the development of the IDN repertoire for the DNS Root Zone and the [RefLGR] to the attention of the Unicode Technical Committee, so that characters that were classified Recommended can be given additional scrutiny before confirming their status.

Arabic Script

As review progresses, a number of characters have been identified with usage as documented in comments and references. For some, the nature of the use should be adjusted:

• U+0671 ٱ ARABIC LETTER ALEF WASLA - this letter is considered to be “an important Quranic character” (which would make it Technical, but not Uncommon_Use). It is also claimed to be used with a newly invented orthography Luri language in Iran.
• U+067C ټ ARABIC LETTER TEH WITH RING - this letter was considered by [Proposal-Arabic] only for Kirghiz and Azerbaijani, but it is used in Kazakh and is part of the name of the Uyghur language “ئۇيغۇر”.

In “Proposal for Arabic Script Root Zone LGR”, [Proposal-Arabic] the repertoires needed for the Arabic Alphabets used for Southern Azerbaijani in Iran [AZ], and for Uyghur [UG], Kazakh [KK], and Kyrgyz [KY] in China, the latter with their own published standards, do not appear to have been included in that document. A more detailed review based on documented repertoires for these writing systems suggests a few characters that should be attested after all.

Accordingly, the proposal was changed to Proposed:Technical for U+0671 ٱ and to retain as unchanged the assignment to Recommended for the others.

Combining Marks

Needed for NFD:—Where combining marks (such as U+0654  ٔ ) are excluded, but needed for decompositions of recommended characters, it was proposed to focus on the NFC format for Identifier_Type but documenting in UTS#39 that combining characters may be marked as Uncommon_Use even when they are in the NFD version of a modern language's exemplar characters.

The following text is proposed for UTS#39:

Where combining marks, such as U+3099  ゙  COMBINING KATAKANA-HIRAGANA VOICED SOUND MARK are only needed for NFD of recommended characters, they have been given Identifier_Type Uncommon_Use. Identifier systems that work with unnormalized text, or text in NDF may wish to add the full set of characters required in canonical decompositions.

Arabic Script combining marks:—Arabic combining marks are categorically excluded from domain names, see also RFC5564. The Internet Architecture Board [IAB] has issued a statement referencing this issue. Please also see the “Proposal for Arabic Script Root Zone LGR”, [Proposal-Arabic]. In consequence, their current assignment of Recommended should be specifically reviewed by the UTC. If it is felt that a change to Uncommon_Use is not the best classification, then perhaps Inclusion or Technical might be more appropriate.

In the context of this document, this affects the following:

• The major Arabic script combining marks U+064B  ً ..U+0652  ْ , U+0654  ٔ ..U+0655  ٕ , U+0657 (Note: of this set, the current document does not mention U+0657  ٗ ). A blanket exclusion may not be applicable to non-domain name identifiers and this set should be discussed explicitly before any changes in Identifier_Type are agreed. Marked as "needing review". (See also the discussion for "Combining Marks" below.)

Review note: If no agreement can be reached, these may be separated into their own document.

Uppercase Characters

Uppercase characters must share the Identifier_Type of their lowercase equivalent. This is to ensure that default identifiers do not change validity if case mapped. Proposed Identifier_Type values included here match that of their lowercase equivalents. In addition, it is recommended that an invariant test be created to automatically verify the relation between Identifier_Type for lowercase and uppercase characters. Because uppercase characters were not independently evaluated, their table entries do not cite a reference containing source information attesting use.

Native Digits

Not all communities use the native digits encoded for their scripts equally in everyday situations. Where their use is not preferred in that context, native digit sets excluded from the [RefLGR]. They are tagged with Identifier_Type Proposed:Uncommon_Use and a comment indicating their lack of common use. In cases where native digits are clearly historical or obsolete, Proposed:Obsolete might be more appropriate. However, no such determination has been made here.

Tibetan Script

No attempt has been made to ascertain modern use for specific characters of the Tibetan script. The script is considered by ICANN as eligible for the Root Zone in principle, but work on defining the label generation rules has faced some difficulties and has not yet commenced. It might be reasonable to reflect that uncertainty by also not giving these characters Identifier_Type Recommended until some body, project, or group has created a definite analysis of this script for identifier purposes. (Tibetan characters have been excluded from the list of characters in this document).

Character Classes

The table in Section 4.1, “Character Classes” presents information about a number of sets collecting characters with different attributes or properties. For each set, a count of members is given. An optional arrow (→) indicates that only a smaller subset of the given set is actually found in this document. For example the notation 2332→0 for the set of characters tagged with RefLGR indicates that none of the characters in this document are part of the Reference LGR, which is as expected, as this document only discusses characters that are not included there.

Contributors

The excerpt that this proposal is based on was prepared by Asmus Freytag, based on published data found in [RefLGR] and reference information from [MSR]. For details on the process and contributors to those projects, see [RefLGR-Overview], in particular, Section 1, “Overview” and Section 6, “Contributors”. Mark Davis, Michel Suignard and Roozbeh Pournader have contributed feedback to this proposal.

Appendix: Unicode sets

The following are listings of the sets of characters for each proposed Identifier_Type value. These can be used to import the values into property data or to convert them to Unicode set notations for additional comparisons. Elements of the sets are space separated and consist either of bare hex codes for single characters or a pair of hex codes separated by HYPHEN-MINUS to indicate a range.

Review_Needed(064B-0652 0654-0655 0670)
Recommended(0674 06C5 06C7-06C9)
Proposed:Technical(0671)
Proposed:Uncommon_Use(0114 0115 012C 012D 014E 014F 0156 0157 0162 0163 01D5 01D6 01D7 01D8 01D9 01DA 01DB 01DC 01DE 01DF 01E0 01E1 01E2 01E3 01EA 01EB 01EC 01ED 01F0 01F4 01F5 01F8 01F9 01FA 01FB 01FC 01FD 01FE 01FF 021E 021F 0226 0227 0228 0229 022A 022B 022C 022D 022E 022F 0230 0231 0232 0233 0400 040D 0450 045D 04C1 04C2 04CB 04CC 04DA 04DB 04EA 04EB 04EC 04ED 05B4 05F0-05F2 0682 0690 0692 0694 069B-069E 06A1 06A3 06A5 06B2 06B4 06B6-06B7 06B8-06B9 06BF 06CA 06D3 06EE-06EF 06FA-06FC 06FF 0750 0753-0755 0757-075F 0761 0764-0765 0769 076B-076D 0772-077D 08A1 08AA-08AC 0904 090C 0929 0934 0944 0979-097A 098C 09D7 0A03 0A66-0A6F 0A72-0A73 0A81 0B0C 0B35 0B57 0B66-0B6F 0BD7 0BE6-0BEF 0C0C 0C31 0C55-0C56 0C66-0C6F 0C8C 0CB1 0CBC 0CC4 0CD5-0CD6 0D0C 0D29 0D66-0D6F 0D8E 0D9E 0DE6-0DEF 0E4E 0EDE-0EDF 108B-108D 1090-1099 10F7-10F8 1207 1287 12AF 12F8-12FF 130F 131F 1347 135A 135D-135E 135F 179D-179E 17A9 17B2 17D7 1E02 1E03 1E04 1E05 1E06 1E07 1E08 1E09 1E0A 1E0B 1E0C 1E0D 1E0E 1E0F 1E10 1E11 1E14 1E15 1E16 1E17 1E1C 1E1D 1E1E 1E1F 1E22 1E23 1E24 1E25 1E26 1E27 1E28 1E29 1E2E 1E2F 1E30 1E31 1E32 1E33 1E34 1E35 1E38 1E39 1E3A 1E3B 1E3E 1E3F 1E40 1E41 1E4C 1E4D 1E4E 1E4F 1E50 1E51 1E52 1E53 1E54 1E55 1E56 1E57 1E58 1E59 1E5A 1E5B 1E5C 1E5D 1E5E 1E5F 1E60 1E61 1E64 1E65 1E66 1E67 1E68 1E69 1E6A 1E6B 1E6E 1E6F 1E78 1E79 1E7A 1E7B 1E7C 1E7D 1E7E 1E7F 1E80 1E81 1E82 1E83 1E84 1E85 1E86 1E87 1E88 1E89 1E8A 1E8B 1E8E 1E8F 1E90 1E91 1E92 1E93 1E94 1E95-1E99 2D80-2D96 A7B9 AB01-AB06 AB09-AB0E)