Characters excluded from both MSR and Reference LGR but allowed in UTS#39

This document has been submitted as a UTC document. For convenience in documenting the character list, it is presented using an LGR template format. A few minor details of the boilerplate in that template may not be applicable in this context and should be disregarded.

The collection includes 713 characters that are not part of the Maximal Starting Repertoire [MSR] because they were found to be in limited or otherwise restricted use and that are also excluded from Reference LGR [RefLGR]. In addition, the set includes the uppercase equivalents for 125 of them (Latin, Cyrillic and Greek), for a total of 769 characters. These characters are listed as Recommended or Inclusion in UTS#39 as of Unicode Version 16.0.0.

In review, some characters were found to be in active general use after all, at a level consistent with an Identifier_Type of Recommended. Such usage has been annotated, and a reference documenting that use has been cited. For those characters, no change to the Identifier_Type is proposed here.

Recommendation

Except for exceptions noted in review, these 769 characters should not be considered Recommended, based on an independent analysis [MSR] that has found indications that they should have been considered Uncommon_Use, Obsolete or Technical, or Exclusion or Inclusion. These indications were based on information available at the time of encoding, such as character proposal documents correlated with available information on the use of the writing systems these characters were intended for. Where they are already assigned Identifier_Type Inclusion, no change is recommended.

More specifically the proposal is to update the Identifier_Types as follows, based on the classification provided and as amended in review:

• Proposed:Technical (155) for members of the sets identified as “technical”, “poetic”, “religious_use”, “modifier” or “symbol” and “Uppercase” (where lowercase is “technical”)
• Proposed:Obsolete (336) for members of the sets identified as “obsolete”, “historic” or “polytoniko” and “Uppercase” (where lowercase is “obsolete”)
• Proposed:Uncommon_Use (281) for members of the sets identified as “uncommon_use”, “limited_use” and “Uppercase” (where lowercase is “uncommon_use”)
• Proposed:Exclusion (2) for members of the sets identified as “duplicate”
• Proposed:Inclusion (1) for a subset of set identified as “punctuation”
• Review_Needed (0) for tags of “numeric”, “context-other”, “punctuation” and no classification in [MSR] (this is a placeholder, as discussion proceeds characters will be moved out of this category into another.)

The following tags mark existing Identifier_Type values; if no proposed value is also present, the proposal is to not change the Identifier_Type:

• Inclusion (11) for characters currently assigned Inclusion; usually, because [MSR] classifies them as “punctuation” and Inclusion was reaffirmed in review.
• Recommended (2) for characters currently assigned Recommended; usually, because this was reaffirmed in review.

Review note: the proposal lists multiple Identifier_Type values in 13 cases where both Uncommon_Use and Obsolete apply, 3 where Uncommon_Use and Technical apply, and 3 cases where Technical and Obsolete apply.

The table in Section 2, “Repertoire” lists the characters with their proposed or existing Identifier_Type values in the “Tags” column, together with references to the source of their classification, as well as other information, such as any known languages.

Review Needed

A few characters have been given no classification in [MSR]. These are for the most part not in IDNA2008, or they represent punctuation characters with Identifier_Type Inclusion, which require no change. Some characters are classified as “context*” which makes them ineligible for the [MSR] and [RZ-LGR], but not the [RefLGR], but the decision of including them in the default identifiers should be reaffirmed in review.

The best matching Identifier_Type for characters tagged only as “punctuation” may need review; there's a strong bias against intra-word punctuation in the Reference LGRs and a complete prohibition in the DNS Root Zone (“letter principle”). Arguably, most intra-word punctuation characters not generic enough for use in IDNs might perhaps best be considered for optional Inclusion instead of Recommended, unless there's an indication that the use is uncommon or obsolete.

Characters that were only identified as “symbol” in [MSR] have been proposed here as Proposed:Technical as that seems the most likely Identifier_Type, unless additional evidence should suggest a different classification.

Background

There are over a thousand non-Han characters with Identifier_Type Recommended that are proposed for consideration of reclassification because they appear to fail reasonable criteria for being needed in identifiers. They come in two sets. For the set documented here, an independent analysis [MSR] has found indications that they should have been considered Uncommon_Use, Obsolete or Technical based on information available at the time of encoding, including their uppercase equivalents and any native digits that can be considered obsolete. The second set contains characters that were tentatively retained as Recommended in the [MSR] but upon further review by local expert teams from the [RZ-LGR] or [RefLGR] projects were found to not be needed after all for any language or minority language in reasonably widespread use. That set is discussed in another document.

The analysis on which the recommendations in this document are based was originally carried out for the purposes of defining the repertoire for IDN Top Level Domain names for the DNS Root Zone. There are some restrictions that are specific to the Root Zone, such as a prohibition on digits, so a follow-on effort determined how to relax these restrictions in a manner appropriate for the needs of Second-Level Domains. This resulted in the Second-Level Reference Label Generation Rules [RefLGR]. The characters listed here are those that were not included in either the [MSR] or the [RefLGR].

Because the original analysis [MSR] found specific indications why the characters in this document should not be recommended, they have been broken out separately and are listed in this document with the rationale for that exclusion provided. Notably, the review for purposes of the [RefLGR] did find that a few characters tentatively excluded in the [MSR] were indeed appropriate for domain names on the Second Level, while most were not.

The implication here is that any character not included in this set should be classified with some value other than Recommended for Unicode's default identifiers—until such time as independent evidence to the contrary is produced.

Arriving at a precise cutoff for the various Identifier_Type categories is difficult because there is no single source or perfect information on the use of writing systems and the de facto general-purpose subset thereof. In addition, the details of such use may change over time. For determining when to suggest “uncommon_use”, the [EGIDS] classification was used as a proxy for the likely level of modern use for the writing system(s) for which the character was encoded, but further adjustments were made in expert review. Where the status of a character was in doubt, for the purposes of [MSR], the decision was made to refrain from rejecting a character, but to defer the decision to expert review in the [RefLGR] project. Accordingly, this document suggests that the UTC should consider the published results of the cited research as one of the better sources of information available and only deviate from it on the basis of even better information.

The latest published result of the [MSR] project has a cutoff of Unicode Version 11.0. The current document includes characters added as Recommended between versions 12.0 and 16.0, for which a tentative analysis following the same criteria as in [MSR] is presented here. These should be treated as “tentatively analyzed” and have been identified accordingly. At this point there are no indications that any of these would be included in a future update of the [MSR] as allowed, and inclusion in either [RZ-LGR] or [RefLGR] in the near term also does not appear likely.

All decisions for the classification of characters in [MSR], or inclusion in [RZ-LGR] and [RefLGR] are documented and sourced on the character level; the same is not true for Unicode's classification, so it is not easily possible to verify any of the decisions that underlie the classification published in UTS39. By first making the alignment proposed here and then carefully documenting deviations, a positive side effect might be that the classification overall becomes more transparent and reviewable.

For further background on specific dispositions for characters in the DNS Root Zone [RZ-LGR] and Second-Level Reference LGR [RefLGR], see the cited references and links therein.

Discussion and Review

Domain names are an important and deliberately conservative set of identifiers. That said, there may be other classes of identifiers that don't require the same level of restrictions, so this proposal should not be understood to suggest that Default Identifiers must be restricted to only those characters that are being recommended for IDNs. Rather, the purpose is to bring the facts discovered during the development of the IDN repertoire for the DNS Root Zone and the [RefLGR] to the attention of the Unicode Technical Committee, so that characters that were classified Recommended can be given additional scrutiny before confirming their status.

Indic Scripts

In review, the following character has been identified that may well have documented use:

• U+093D ऽ DEVANAGARI SIGN AVAGRAHA - this sign is claimed to be in use in modern Hindi for writing foreign sounds in informal settings (cool: कूऽल) or stretched native ones (माँऽऽऽ!). See [Avagraha]. The suggested usage seems tenuous for the purposes of default identifiers and the shape, at least in Devanagari, is confusable with ASCII. Those factors argue against retaining this character as Recommended and it has been proposed as Obsolete.

Review note: We might benefit from an affirmative decision on whether this type of usage meets our cutoff for default identifier, and if not, whether to assign it Obsolete, as proposed here, or Uncommon_Use or Technical instead.

Greek Script

There is some evidence that Greek polytoniko orthography is considered to be of limited utility for domain names. The [Greek-IDN-Case-Study] report considers it obsolete for Greek and not useful to modern users, citing difficulty in accessing it from modern keyboards. This decision makes sense in an identifier context, particularly for the Root Zone. [Proposal-Greek] which only contains monotonic Greek, suggests in an aside that there might be a use case for polytonic Greek for the Second Level, particularly for certain traditionalist or religious organizations. However, the [RefLGR] project has not received a request to add a reference LGR for the second level supporting that orthography, thus limiting its availability for contracted domains. This makes sense given the ratio of monotonic vs. polytonic orthographies in public life. The [Greek-ccTLD] is an example of a registry that does allow polytonic Greek.

To complicate the matter, much of the precomposed uppercase Greek letters used for polytonic Greek are not PVALID in IDNA2008 due to complexities with uppercasing. These complexities make supporting the orthography complicated in cases where identifiers are not limited to lowercase letters as in IDN2008. While any identifier implementation and, within the restrictions of IDNA2008, any domain name registry is free to support polytonic Greek, it appears a poor choice for default identifiers. Accordingly, the proposal is to treat the affected characters as Obsolete; this affects all characters in the Greek Extended block and the YPOGEGRAMMENI.

Arabic Script

The following issues have been raised in review with respect to the Arabic script.

• U+063D ؽ ARABIC LETTER FARSI YEH WITH INVERTED V - this letter can be found to be in use with Azerbaijani and should therefore not be considered “obsolete, historic”. For documentation, see [AZ] and [Azerbaijani-Encoding-Proposal]. It is proposed to retain this character as Recommended.

Tibetan Script

No definite recommendations can be made for the Tibetan script at this time. It is considered by ICANN as eligible for the Root Zone in principle, but work on defining the label generation rules has faced some obstacles and has not commenced. Nevertheless, Tibetan was covered in the MSR analysis, and for some characters a suggested adjustment of Identifier_Type can be provided; two are called out as being homoglyph digraphs, constituting a “duplicate” of a two-character sequence; these have been proposed for Exclusion.

Beyond applying that limited reassessment, it might be reasonable to reflect the preliminary nature of the understanding of the likely use of Tibetan in identifiers by also not giving any of the Tibetan characters an Identifier_Type Recommended until some technical body, project, or group has created a definite analysis of this script for identifier purposes. (Only Tibetan characters with proposed exclusion from the [MSR] have been included in this document).

Bopomofo Script

The [MSR] and [RefLGR] exclude the Bopomofo script, considering the entire script special use as it tends to be used almost exclusively in education. A separate proposal exists to change the status of the script in UAX31 to Limited_Use. If the proposal to reclassify the script is accepted, it would take precedence. Therefore, Bopomofo characters U+3105 ㄅ..U+312D ㄭ, U+312F ㄯ and U+31A0 ㆠ..U+31BA ㆺ have been removed from this document.

Uppercase Characters

Uppercase characters must share the Identifier_Type of their lowercase equivalent. This is to ensure that default identifiers do not change validity if case mapped. In assembling this data set, a few uppercase characters were spotted that are currently Recommended, but for which the lowercase equivalents were already not Recommended. For these, Identifier_Type values matching their lowercase equivalents are proposed here. In addition, it is recommended that an invariant test be created to automatically verify the relation between Identifier_Type for lowercase and uppercase characters. Because uppercase characters are not independently evaluated, their table entries do not cite a reference containing source information leading to the proposed Identifier_Type value.

Digits

If a writing system shares a script covered in the [MSR] but extensions for it were excluded, any native digits for that writing system are given proposed Identifier_Type values matching those of the letters.

Intra-Word or Word-Related Punctuation

A number of these have been made Inclusion in the existing Identifier_Type. However, some are Recommended.

• U+005F _ LOW LINE is currently Recommended, making it a bit of an outlier.

It might be useful to ask whether it wouldn't be cleaner for default identifiers to leave the entire set of these for explicit inclusion. That would separate the design of an identifier syntax from the default repertoire.

Combining Marks

Needed for NFD:—Where combining marks (such as U+3099  ゙ ) are only needed for decompositions of recommended characters, it was proposed to focus on the NFC format for Identifier_Type but document in UTS#39 that combining characters may be marked as Uncommon_Use even when they are in the NFD version of a modern language's exemplar characters.

The following text is proposed for UTS#39:

Where combining marks, such as U+3099  ゙  COMBINING KATAKANA-HIRAGANA VOICED SOUND MARK are only needed for NFD of recommended characters, they have been given Identifier_Type Uncommon_Use. Identifier systems that work with unnormalized text, or text in NDF may wish to add the full set of characters required in canonical decompositions.

High Risk Characters

Certain characters are particularly risky for security-relevant identifiers, such as IDNs. These include characters that look like punctuation marks. The usual mitigation mechanisms for visually equivalent characters apply to situations where both partners are in the repertoire. By definition, that is not the case for these, as the confusability is with delimiting punctuation, which would falsely suggest part of the string is outside the label. They should therefore never be recommended for default identifiers but given Identifier_Type Exclusion or Inclusion. Further analysis should be carried out in the context of a separate proposal.

Defaults for Newly Encoded Characters

Characters newly encoded for existing scripts in Unicode 17.0 and after should not be given Identifier_Type Recommended by default, as most often, scripts are supplemented with additional Technical or Obsolete characters. For modern characters, unless they are intended for support of a particularly vigorous language with well established orthography in widespread use, they should be considered Uncommon_Use by default, unless they are clearly Technical in nature. (Note: characters encoded since the cutoff for the last MSR version have been tentatively analyzed here using the same principles as in [MSR].)

Presentation of Data

The table in Section 2, “Repertoire" lists the characters affected by this proposal.

• All characters have tags indicating their existing or proposed Identifier_Type based on the classification in [MSR] and other available evidence, including that received in review.
• The classification of characters excluded from the MSR is a bit more fine-grained than the current values for Identifier Type, which means that the proposed Identifier_Type values correspond to a combination of MSR classifications. In some cases, this leads to two values for the proposed Identifier_Type.
• Uppercase equivalents are noted as Uppercase in the comments. As uppercase characters are not used in IDNA2008, they are not independently evaluated here but share the proposed Identifier_Type of their lowercase equivalent.
• Appropriate Identifier_Type values have been proposed for any other characters not in IDNA2008.
• Comments on characters give further information, such as the name of a language for which a character can be considered in limited, technical or historical use, or a UTC document reference.

Character Classes

The table in Section 4.1, “Character Classes” presents information about a number of sets collecting characters with different attributes or properties. For each set, a count of members is given. An optional arrow (→) indicates that only a smaller subset of the given set is actually found in this document. For example, the notation 5271→0 for the set of characters tagged with Identifier_Type Limited_Use indicates that none of the characters here currently have that Identifier_Type value, which is as expected, as this document only discusses characters that are currently Recommended or Inclusion.

Contributors

The excerpt that this proposal is based on was prepared by Asmus Freytag, based on published data found in [RefLGR] and reference information from [MSR]. For details on the process and contributors to those projects, see [RefLGR-Overview], in particular, Section 1, “Overview” and Section 6, “Contributors”. Mark Davis, Michel Suignard and Roozbeh Pournader have contributed feedback and/or additional classifications to this proposal.

Notable Changes and Updates

• U+063D ؽ ARABIC LETTER FARSI YEH WITH INVERTED V — The type for this character is retained as Recommended; it is anticipated that this change will also be reflected in a future update of [MSR]. The analysis for several recently encoded characters up to Unicode Version 16.0 has been added.
• U+02BB ʻ MODIFIER LETTER TURNED COMMA — The proposed type for this character has changed to Inclusion; this reflects the fact that this character is confusable with some syntax characters.
• U+0375 ͵ GREEK LOWER NUMERAL SIGN — The proposed type for this character has been changed from Inclusion to Technical; this reflects the fact that it is claimed to be in current use to mark text as numeric.
• U+0F3E  ༾  TIBETAN SIGN YAR TSHES and U+0F3F  ༿  TIBETAN SIGN MAR TSHES — The proposed type for these characters has changed to Technical; this reflects their use in converting numbers to astrological signs.
• U+3099  ゙  COMBINING KATAKANA-HIRAGANA VOICED SOUND MARK and U+309A  ゚  COMBINING KATAKANA-HIRAGANA SEMI-VOICED SOUND MARK — The proposed type for these characters has been changed to Uncommon_Use; this reflects that their use is only required in NFD.

Appendix: Unicode Sets

The following are listings of the sets of characters for each proposed Identifier_Type value. These can be used to import the values into property data or to convert them to Unicode set notations for additional comparisons. Elements of the sets are space-separated and consist either of bare hex codes for single characters or a pair of hex codes separated by HYPHEN-MINUS to indicate a range.

Review_Needed()
Proposed:Inclusion(02BB)
Proposed:Exclusion(0F7B 0F7D)
Proposed:Technical(0200-0217 02BC 02EC 030F-0311 0313-0314 0324-0325 032D 032E 0330 0335 0338 0339 0342 0375 03FC 0559 0653 06E5-06E6 0870-0888 0950 097D 0A74 0AD0 0B82 0BD0 0EAF 0F00 0F35 0F37 0F3E-0F3F 0FC6 10F9 10FA 1E00-1E01 1E18-1E1B 1E2A-1E2D 1E72-1E77 1FB0-1FB1 2D27 2D2D A717-A71F A788 A7C5-A7C6 1DF00-1DF1E 1DF25-1DF2A)
Proposed:Obsolete(0138 0345 037B-037D 03FC-03FF 049C-049D 04A6-04A7 04B8-04B9 0514-0523 0526-0529 052E-052F 063B-063C 063E-063F 06AC 077E-077F 08B5 093D 0960-0963 0971 09BD 09E0-09E3 0ABD 0AE0-0AE3 0B3D 0B60-0B61 0C3D 0C60-0C61 0CBD 0CE0-0CE3 0CF1-0CF2 0D3A 0D3D 0D4C 0D4E 0D60-0D61 0F6A 0F82-0F83 0F86-0F8F 1050-1059 17DC 1F00-1F15 1F18-1F1D 1F20-1F45 1F48-1F4D 1F50-1F57 1F59 1F5B 1F5D 1F5F-1F70 1F72 1F74 1F76 1F78 1F7A 1F7C 1F80-1FB4 1FB6-1FBA 1FBC 1FC2-1FC4 1FC6-1FC8 1FCA 1FCC 1FD0-1FD2 1FD6-1FDB 1FE0-1FE2 1FE4-1FEB 1FF2-1FF4 1FF6-1FF8 1FFA 1FFC A67F A7C0-A7C1 A7C4 A7C7-A7CA A7D0-A7D1 A7D3 A7D5-A7D9 1B11F-1B122 1B132 1B150-1B152 1B155 1B164-1B167 1E08F)
Proposed:Uncommon_Use(048A-048F 04C3-04CA 04CD-04CE 04F6-04F7 04FA-04FF 0510-0513 05EF 0889-088E 08B2 08B6-08BA 08C3-08C8 09FE 0A01 0AFA-0AFF 0B55 0C01 0C04 0C3C 0C5D 0C80 0CDD 0CF3 0D00 0D54-0D56 0E86 0E89 0E8C 0E8E-0E93 0E98 0EA0 0EA8-0EA9 0EAC 0EBA 0ECE 0F6A 0F6B-0F6C 0F82-0F83 0FAE-0FAF 0FB0 0FC6 1050-1059 1065-106D 106E-1070 1071 1072-1074 108E 109A-109B 109C-109D 10F9 10FA 10FD 10FE-10FF 1380-138F 2DA0-2DA6 2DA8-2DAE 2DB0-2DB6 2DB8-2DBE 2DC0-2DC6 2DC8-2DCE 2DD0-2DD6 2DD8-2DDE A792-A793 A7C2-A7C3 A9E7-A9EF A9F0-A9FE AA60-AA76 AA7A AA7C-AA7D AA7E-AA7F AB11-AB16 AB20-AB26 AB28-AB2E AB66-AB67 1133B)