L2/09-232R

From: Mark Davis
Date: 2009-08-14 (revised)

Proposal

1. In U5.2, add a section to UAX#29 describing that the following characters are common candidates for tailoring to add to MidLetter.
   
   • [\-\u058A\u1806\u2010\u2011\u2E17\u30A0\uFE63\uFF0D][\u058A\u0F0B\u30A0\u30FB]
     
   • U+002D ( - ) HYPHEN-MINUS
     U+058A ( ֊ ) ARMENIAN HYPHEN
     U+0F0B ( ་ ) TIBETAN MARK INTERSYLLABIC TSHEG
     U+1806 ( ᠆ ) MONGOLIAN TODO SOFT HYPHEN
     U+2010 ( ‐ ) HYPHEN
     U+2011 ( ‑ ) NON-BREAKING HYPHEN
     U+2E17 ( ⸗ ) DOUBLE OBLIQUE HYPHEN
     U+30A0 ( ゠ ) KATAKANA-HIRAGANA DOUBLE HYPHEN
     U+30FB ( ・ ) KATAKANA MIDDLE DOT
     U+FE63 ( ﹣ ) SMALL HYPHEN-MINUS
     U+FF0D ( － ) FULLWIDTH HYPHEN-MINUS
2. In U5.2, add a section to UAX#29 discussing name validation characters, and giving guidelines for usage adding characters to the word characters allowed above.
3. In U5.2, add pointers in UAX#31 and #29 wordbreak and text indicating the relationship between identifiers and words (and that the character sets are not the same).
   
4. Add a test for consistency between the WB properties and Table 3 (with the known exceptions) to the invariant tests.
   

====

For entry field validation, implementations sometimes need to know which characters can occur in personal names. While it is a bit fuzzy exactly what this means, they want to distinguish between characters like those in "James Smith-Faley, Jr." and those in "!#@♥≠". Note that it is important to be reasonably lenient: it is extremely annoying for people not to be able to add legitimate names, like "di Silva", because those names have characters like space.

Typically, these personal name validations should not be language-specific; I might be using a website in a language other than the one for my name, for example. While a more sophisticated validation might use context among characters, a basic validation just wants to know "what characters can be part of names?". The text should explain that:

1. It is only a guideline, and may need tailoring for different environments
2. It is a lenient, non-language-specific set - for language-specific characters one should see CLDR.
3. Mention characters:
   
   • [,.[:whitespace:]]
   • U+002C ( , ) COMMA
     U+002E ( . ) FULL STOP
     [:whitespace:]
4. It includes characters that may not be appropriate for identifiers, and those that would not be parts of words.
5. It does not include contextual tests
6. Additional tests may be needed in cases where security is at issue.
7. The set can be narrowed if name fields are split out. For example, "," may not be necessary if titles are split out; if titles are not allowed, "." may not be necessary.
   
8. Word characters contains some characters that may be part of words in a broad sense, such as "c:a" in Swedish or a dictionary word containing hyphenation points, that might not normally be part of names.
9. Explain the use of NFKC in name validation
   

---

Background Information

Characters added by Word Boundaries

Basic Latin - ASCII punctuation and symbols

Latin 1 Supplement - Latin-1 punctuation and symbols

Greek And Coptic - Punctuation

Hebrew - Additional punctuation

General Punctuation - General punctuation

Vertical Forms - Glyphs for vertical variants

CJK Compatibility Forms - Glyphs for vertical variants

CJK Compatibility Forms - Overscores and underscores

Small Form Variants - Small form variants

Halfwidth And Fullwidth Forms - Fullwidth ASCII variants

0027 (') APOSTROPHE

002D (-) HYPHEN-MINUS
002E (.) FULL STOP
003A (:) COLON
00B7 (·) MIDDLE DOT
058A (֊) ARMENIAN HYPHEN
05F3 (׳) HEBREW PUNCTUATION GERESH
05F4 (״) HEBREW PUNCTUATION GERSHAYIM
0F0B ( ་ ) TIBETAN MARK INTERSYLLABIC TSHEG
200C () ZERO WIDTH NON-JOINER*
200D () ZERO WIDTH JOINER*
2010 (‐) HYPHEN
2019 (’) RIGHT SINGLE QUOTATION MARK
2027 (‧) HYPHENATION POINT
30A0 (=) KATAKANA-HIRAGANA DOUBLE HYPHEN
30FB ( ・ ) KATAKANA MIDDLE DOT

 [\p{alpha}\p{WB=Extend}\p{WB=FO}\p{WB=LE}\p{WB=ML}\p{WB=MB}\p{WB=EX}-\p{cf}]

Here are Word characters minus Identifier characters.

Basic Latin - ASCII punctuation and symbols

Greek And Coptic - Punctuation

Cyrillic - Historic miscellaneous

Arabic - Koranic annotation signs

General Punctuation - General punctuation

Combining Diacritical Marks For Symbols - Enclosing diacritics

Combining Diacritical Marks For Symbols - Additional enclosing diacritics

Enclosed Alphanumerics - Circled Latin letters

Supplemental Punctuation - Medievalist punctuation

Cyrillic Extended B - Combining numeric signs

Vertical Forms - Glyphs for vertical variants

Small Form Variants - Small form variants

Halfwidth And Fullwidth Forms - Fullwidth ASCII variants

And the Identifier Characters minus the Word Characters

Armenian - Punctuation

Tibetan - Marks and signs

Katakana - Katakana punctuation

Katakana - Conjunction and length marks