L2/09-232

From: Mark Davis
Date: 2009-07-07

Background

For entry field validation, implementations sometimes need to know which characters can occur in personal names. While it is a bit fuzzy exactly what this means, they want to distinguish between characters like those in "James Smith-Faley, Jr." and those in "!#@♥≠". Note that it is important to be reasonably lenient: it is extremely annoying for people not to be able to add legitimate names, like "di Silva", because those names have characters like space.

Typically, these personal name validations should not be language-specific; I might be using a website in a language other than the one for my name, for example. While a more sophisticated validation might use context among characters, a basic validation just wants to know "what characters can be part of names?".

Much of this can be derived (with a bit of work) from http://www.unicode.org/reports/tr29/#Default_Word_Boundaries. The basic characterization of characters that can be in words is from the combination of properties:


Word boundaries adds 27 characters: http://unicode.org/cldr/utility/list-unicodeset.jsp... that also should be included, at least prima facie (see list at the end of this document). That brings us to:


But we also include other characters in http://www.unicode.org/reports/tr31/tr31-10.html#Specific_Character_Adjustments Table 3 that are suitable for identifiers -- and words. (That set is http://unicode.org/cldr/utility/list-unicodeset.jsp.... Most of those are included in the above, but the following are not:

Missing Characters for Word Break (from Table 3)

Basic Latin - ASCII punctuation and symbols

Armenian - Punctuation

Tibetan - Marks and signs

General Punctuation - Dashes

Katakana - Katakana punctuation

Katakana - Conjunction and length marks

Some of these clearly need to be allowed in names. There are seven other characters with "HYPHEN" in their names. Of these, all but the BULLET one probably qualify.

Hyphen-Named Characters

Proposal

Based on this, there are a few items I recommend. First is to reconconcile some gratuitious differences between word characters and identifier characters that were uncovered in this process. That is, do the following (after review to catch exceptions):

1. Add the missing characters from Table 3. Candidate Characters for Inclusion in Identifiers in TR31 (also listed in Missing Characters from Word Break above) to \p{Word_Break=MidLetter}
2. Add the missing characters from \p{Word_Break=MidLetter} -- those that are isNFKC -- to Table 3.
3. Add the hyphen characters above, excluding the hyphen bullet, to \p{Word_Break=MidLetter} and those that are isNFKC to Table 3).
   
4. Add an invariant test for consistency between the WB properties and Table 3.

Second is to have a list of "name validation" characters that people can use. That is, make it easier for people to get a set of name-validation characters by at providing a list of the exceptional characters in a new section of TR29. Aside from the above characters, this list would also include ".," (and their variants) and space (and its variants). The text in that section needs have a number of caveats to make it clear what the limitations on the use of the list are. In particular:

1. It is only a guideline, and may need tailoring for different environments
2. It is a lenient, non-language-specific set - for language-specific characters one should see CLDR.
   
3. It includes characters that may not be appropriate for identifiers, and those that would not be parts of words.
4. It does not include contextual tests
5. Additional tests may be needed in cases where security is at issue.
6. The set can be narrowed if name fields are split out. For example, "," may not be necessary if titles are split out; if titles are not allowed, "." may not be necessary.
   
7. It contains some other characters that may be part of words in a broad sense, such as "c:a" in Swedish or a dictionary word containing hyphenation points, that might not normally be part of names.
   

---

Background Information

Characters added by Word Boundaries

Basic Latin - ASCII punctuation and symbols

Latin 1 Supplement - Latin-1 punctuation and symbols

Greek And Coptic - Punctuation

Hebrew - Additional punctuation

General Punctuation - General punctuation

Vertical Forms - Glyphs for vertical variants

CJK Compatibility Forms - Glyphs for vertical variants

CJK Compatibility Forms - Overscores and underscores

Small Form Variants - Small form variants

Halfwidth And Fullwidth Forms - Fullwidth ASCII variants

0027 (') APOSTROPHE

002D (-) HYPHEN-MINUS
002E (.) FULL STOP
003A (:) COLON
00B7 (·) MIDDLE DOT
058A (֊) ARMENIAN HYPHEN
05F3 (׳) HEBREW PUNCTUATION GERESH
05F4 (״) HEBREW PUNCTUATION GERSHAYIM
0F0B ( ་ ) TIBETAN MARK INTERSYLLABIC TSHEG
200C () ZERO WIDTH NON-JOINER*
200D () ZERO WIDTH JOINER*
2010 (‐) HYPHEN
2019 (’) RIGHT SINGLE QUOTATION MARK
2027 (‧) HYPHENATION POINT
30A0 (=) KATAKANA-HIRAGANA DOUBLE HYPHEN
30FB ( ・ ) KATAKANA MIDDLE DOT

 [\p{alpha}\p{WB=Extend}\p{WB=FO}\p{WB=LE}\p{WB=ML}\p{WB=MB}\p{WB=EX}-\p{cf}]

Here are Word characters minus Identifier characters.

Basic Latin - ASCII punctuation and symbols

Greek And Coptic - Punctuation

Cyrillic - Historic miscellaneous

Arabic - Koranic annotation signs

General Punctuation - General punctuation

Combining Diacritical Marks For Symbols - Enclosing diacritics

Combining Diacritical Marks For Symbols - Additional enclosing diacritics

Enclosed Alphanumerics - Circled Latin letters

Supplemental Punctuation - Medievalist punctuation

Cyrillic Extended B - Combining numeric signs

Vertical Forms - Glyphs for vertical variants

Small Form Variants - Small form variants

Halfwidth And Fullwidth Forms - Fullwidth ASCII variants

And the Identifier Characters minus the Word Characters

Armenian - Punctuation

Tibetan - Marks and signs

Katakana - Katakana punctuation

Katakana - Conjunction and length marks