Characters excluded from both MSR and Reference LGR but allowed in UTS#39

This document has been submitted as a UTC document. For convenience in documenting the character list it is presented using an LGR template format. A few minor details of the boilerplate in that template may not be applicable in this context and should be disregarded.

The collection comprises 696 characters are not part of the Reference LGR [RefLGR] and also excluded from the MSR [MSR], as well as the uppercase equivalents for 128 of them (Latin, Cyrillic and Greek), for a total of 824 characters. These characters are listed as Recommended or Inclusion in UTS#39.

Recommendation

These 824 characters should not be considered Recommended, based on an independent analysis [MSR] that has found indications that they should have been considered Uncommon_Use, Obsolete or Technical based on information available at the time of encoding, such as character proposal documents, correlated with available information on the use of the writing systems these were intended for. Where they are already assigned Identifier_Type Inclusion, no change is recommended.

More specifically the Identifier_Types should be updated, based on the tags provided:

• Technical for "technical", "poetic", "religious_use", "modifier" or "phonetic"
• Obsolete for "obsolete" or "historic" and "(Greek)Uppercase" (where lowercase is "obsolete" or "historic")
• Uncommon_Use "uncommon_use", "limited_use" and "Uppercase" (where lowercase is "uncommon_use")

A few characters have been given no classification in the tags. These are for the most part not in IDNA2008 or they represent punctuation with Identifier_Type Inclusion, which seems fine. Some characters are classified as "context*" which makes them ineligible for the [MSR] and [RZ-LGR], but not the [RefLGR].

The best matching Identifier_Type for characters tagged only as "punctuation" and "symbol" is unclear; there's a strong bias against intra-word punctuation in the Reference LGRs and a complete prohibition in the DNS Root Zone ("letter principle"). Arguably most intra-word punctuation not generic enough for use in IDNs might perhaps best be considered for optional "Inclusion" instead of "Recommended", unless there's an indication that the use is uncommon or obsolete.

The latest published result of the [MSR] project has a cutoff of Unicode, Version 11.0. The current document includes characters added as Recommended between versions 12.0 and 16.0 for which analysis is preliminary. These should be treated as "tentatively analyzed" and have been tagged accordingly. Unless their need for identifiers, or use in a reasonably widely used general orthography for any language can be established affirmatively from other sources, they should NOT retain Identifier_Type Recommended. Instead, they should be considered Uncommon_Use, unless a they can be identified as Obsolete or Technical. There are no indications that any of these would be included in a future update of the [MSR], and inclusion in either [RZ-LGR] or [RefLGR] in the near term would seem even less likely.

Background

There are about a thousand non-Han characters with Identifier_Type Recommended that are proposed for consideration of reclassification because they appear to fail reasonable criteria for being needed in identifiers. They come in two sets. For the set documented here, an independent analysis [MSR] has found indications that they should have been considered Uncommon_Use, Obsolete or Technical based on information available at the time of encoding. The second set contains characters that were tentatively retained as Recommended in the [MSR] but upon further review by local expert teams from the [RZ-LGR] project were found to not be needed after all for any language or minority language in reasonably widespread use. That set is discussed in another document.

The analysis on which the recommendations in this document is based was originally carried out for the purposes of defining the repertoire for IDN Top Level Domain names for the DNS Root Zone. There are some restrictions that are specific to the Root Zone, such as a prohibition on digits, so a follow-on effort determined how to relax these restrictions in a manner appropriate for the needs of Second-Level Domains. This resulted in the Second-Level Reference Label Generation Rules [RefLGR]. The characters listed here are those that were not included in either the [MSR] or the [RefLGR].

Because the original analysis [MSR] found specific indications why the characters in this document should not be recommended, they have been broken out separately and are listed in this document with the rationale for that exclusion provided. Notably, the review for purposes of the [RefLGR] did find that a few characters tentatively excluded in the [MSR] were indeed appropriate for domain names on the Second Level, while most were not.

The implication here is that any character not included in this set should be classified with some value other than Recommended for Unicode's default identifiers—until such time as independent evidence to the contrary is produced.

Arriving at a precise cutoff for the various Identifier_Type categories is difficult because there is no single source or perfect information on the use of writing systems and the de-facto general purpose subset thereof. In addition, the details of such use may change over time. Accordingly, this document suggests that the UTC should consider the published results of the cited research as one of the better sources of information available and only deviate from it on the basis of even better information.

All decisions for the classification of characters in [MSR], or inclusion in [RZ-LGR] and [RefLGR] are documented and sourced on the character level; the same is not true for Unicode's classification, so it is not easily possible to verify any of the decisions that underlie the classification published in UTS39. By first making the alignment proposed here, and then carefully documenting deviations, a positive side effect might be that the classification overall becomes more transparent and reviewable.

Special Considerations

The [MSR] and [RefLGR] exclude the Bopomofo script, considering the entire script special use as it tends to be used almost exclusively in education. This could be addressed by either changing the status of the script in UAX31 to Limited_Use or to mark the entire set of Bopomofo characters as Technical.

The domain name registry community appears to have settled on regarding the Greek polytoniko orthography as obsolete for Greek. This should be reflected in Unicode's default identifiers.

Defaults for characters newly assigned for Unicode 17.0.0 and after. Newly encoded characters for existing scripts should not be given Identifier_Type Recommended by default, as most often, scripts are supplemented with additional Technical or Obsolete characters. For modern characters, unless they are intended for support of a particularly vigorous language with well established orthography in widespread use, they should be considered Uncommon_Use by default. The same should be retroactively applied to the few characters added to the recommended scripts since the cutoff for the [MSR] work (as documented in the latest MSR version).

High risk characters. Certain characters are risky for security relevant identifiers, such as IDNs. These include characters that look like punctuation marks. The usual mitigation mechanism for visually equivalent characters apply to situations where both partners are in the repertoire. By definition, that is not the case for these, as the confusability is with delimiting punctuation which whould falsely suggest part of the string is outside the label. They should therefore not be recommended for default identifiers.

No definite recommendations can be made for the Tibetan script at this time. It is considered by ICANN as eligible for the Root Zone in principle, but work on defining the label generation rules has faced some difficulties and has not commenced. Nevertheless, Tibetan was covered in the MSR analysis, and for some characters a suggested adjustment of Identifier_Type can be provided. Beyond that, it might be reasonable to reflect the preliminary nature of the understanding of the likely use of Tibetan in identifiers by also not giving any of the Tibetan characters an Identifier_Type Recommended until some body, project, or group has created a definite analysis of this script for identifier purposes. (Only Tibetan characters with proposed exclusion from the [MSR] have been included in this document).

Root Zone and Reference LGRs

For further background on the DNS Root Zone and Second-Level Reference LGR see the cited references and links therein.

Additional Notes

All characters have tags matching indicating a recommended classification based on available evidence. This classification was used by experts and reviewers in making an initial cutoff for the MSR and has been slightly amended here.

The classification of characters excluded from the MSR is a bit more fine-grained than the current values for Identifier Type, see the recommended mappings above.

Uppercase equivalents are tagged as Uppercase or GreekUppercase only, their classification is understood to match that of their lowercase equivalent. As uppercase characters are used in IDNA2008, they are not independently evaluated here. Any other characters not in IDNA2008 are not classified, but indicated in the comments.

For other characters, comments give further information, such as the name of a language that can be considered limited in use, a Unicode version or a UTC document reference. Code points slated for encoding in 17.0 may not display with other than code point value in the table as the tool formatting the template cannot retrieve the character name or script for unreleased versions.

Discussion and Review

Domain names are an important, and deliberately conservative set of identifiers. That said, there may be other classes of identifiers that don't require the same level of restrictions, so this proposal should not be understood to suggest that default Identifiers must be restricted to only those characters that are being recommended for IDNs. Rather, the purpose is to bring the facts discovered during the development of the IDN repertoire for the DNS Root Zone and the [RefLGR] to the attention of the Unicode Technical Committee, so that characters that were classified Recommended can be given additional scrutiny before confirming their status.

As review progresses, a number of characters have been identified that may well have documented use:

• U+063D ؽ ARABIC LETTER FARSI YEH WITH INVERTED V - this letter is claimed to be in use with Azerbaijani, and should not be considered "obsolete, historic".
• U+093D ऽ DEVANAGARI SING AVAGRAHA - this sign is claimed to be in use in modern Hindi for writing foreign sounds (cool: कूऽल) or stretched native ones (माँऽऽऽ!). This may be a matter of which criteria are to be used, UTC should make an affirmative decision.

The repertoires needed for Uyghur, Kazakh, and Kyrgyz, all written in Arabic in China with their own published standards are claimed to be missing in the cited review. However, they are mentioned in “Proposal for Arabic Script Root Zone LGR”, [Proposal-Arabic]. It might be necessary to make a more detailed review based on the full claimed repertoire for these.

Contributors

This excerpt was prepared by Asmus Freytag, based on published data found in [RefLGR] and reference information from [MSR]. For details on the process and contributors to those projects, see [RefLGR-Overview], in particular, Section 1, “Overview” and Section 6, “Contributors”. Mark Davis, Michel Suignard and Roozbeh Pournader have contributed feedback.