L2/13-203
Re: | #36 & #39 Recommendations |
To: | UTC |
From: | Mark Davis |
Live: |
While doing the latest version of UTR #36 & UTS #39 and associated data I uncovered a few issues, for which I have the following recommendations. Some of these could be implemented in the upcoming release of #36 & #39, while others are probably more appropriate for the version built on UCD 7.0.
4. Be specific about NFC in IdMod
5. Mechanical derivation of IdMod
See also http://unicode-inc.blogspot.ch/2013/10/proposed-updates-for-utr-36-and-uts-39.html
Idempotence of a normalization/canonicalization function is a crucial invariant to maintain, and it would be worth adding a section about that, and transitivity.
There is very nicely written article at http://labs.spotify.com/2013/06/18/creative-usernames/ that details how bad idempotence caused a security breach. We could just have a short section that points to that detailed explanation.
Another useful discussion is on the transitivity of comparison functions for Unicode. These are easy to get wrong, and sorting behaves bizarrely when faced with non-transitive comparisons, and can also cause unexpected breaches.
In each PRI for #36 we should also solicit pointers to other useful articles.
Each of the versions of the data files for UTS #39 is for a specific version of the UCD. However, the versioning of the data files for UTS #39 does not reflect that. I recommend that we switch the versioning to use the version of the UCD that it was built for.
This is definitely easier for users to understand and manage in their implementations. For comparison, here are the versions so far:
Server folder | release date | Version | Unicode version | Date |
revision-02/ | 2006-08-15 | Version 1 | 5.1.0 | 2008-04 |
revision-03/ | 2006-08-11 | draft only! | n/a | n/a |
revision-04/ | 2010-08-05 | Version 2 | 6.0.0 | 2010-10 |
revision-05/ | 2012-07-23 | Version 3 | 6.1.0 | 2012-01 |
revision-06/ | 2013-xx-x? | Version 4* | 6.3.0 | 2013-09 |
If for some reason we had to issue a modified version between releases, we could use the 3rd field, eg 6.3.1.
Now, there are a few ways we could manage using the same version number.
The data files are gratuitously different from the UCD data files, yet many people will want to use the same parser. We could make the following changes to make it more consistent.
Use values for Idmod Status and Type that are more like UCD, eg Limited_Use instead of limited-use.
Change the format for confusables.txt to be more easily parsed in the same way as the UCD files.
01C3 ; 0021 ; SL #...
=>
01C3 ; Confusable_SL ; 0021 #...
In the current algorithm using idmod data, it doesn’t say which normalization form the text is presumed to be in, if any. However, an underlying presumption is that the text is in NFC, and we should make that clear in the text.
In theory, there are complications because of canonical equivalence. I mention them below for completeness, although I don’t think that they present a problem in practice at this point.
a. Consider the case of x + under_dot (as on http://www.unicode.org/standard/where/). Because we base our data on single code points, not sequences, to allow <U+0078 U+0323> requires that we mark U+0323 as recommended (thus with any base). To be more fine-grained, we’d have to extend the data file to also allow a sequence of code points, like the following. However, that doesn’t seem worth the effort for the few cases where this arises.
0078 0323 ; allowed ; recommended # LATIN SMALL LETTER X + COMBINING DOT BELOW
b. Consider the NFC combining character sequence <a-dot-below, umlaut>, where:
Because this sequence is canonically equivalent to <a-umlaut, dot-below>, it should also be allowed. Again, however, complicating an algorithm to handle this case doesn’t seem worth the effort for the few cases (if any!) where this would arise.
If in the future we want to handle (a) and (b), we could extend the algorithm and data at that time.
Replace historic, technical in curated list with those that match UAX 31 or are derived from cldr: Aspirational, Limited_Use, Exclusion, Not_Cldr.
1. We currently derive most of historic and limited-use from UAX 31, but the terms are not the same. Aligning them would make the tables and the derivation clearer.
2. For technical and some of historic and limited-use, the original data was put together before CLDR was as well-developed, and depended on review of information in the UTC plus curated contributions. We now have more resources at our disposal, however, and can base the majority of the data on the CLDR exemplar characters. There still will be need, however, for an exception file, but it should be much smaller and easier to maintain.
My suggested algorithm is to collect all the characters for locales in the latest release of CLDR where the following conditions are true. We’d modify Step 7 in http://unicode.org/draft/reports/tr39/tr39.html#IDMOD_Data_Collection so that any character outside of this list would be marked as ‘Not_Cldr’.
Clause C is a heuristic, because the languages that are below that level don’t get as much attention. As we improve the coverage and quality, this clause can be refined. Esperanto is added to B simply because it is the only “non-living” language with sizable usage (however, its addition doesn’t really make a difference.
If we applied the CLDR-based methodology, we’d get add the following characters (compared to the previous version). Note that the uppercase equivalents would also be included; they are just omitted from this list for brevity.
[ƒƙƴɓɔɖɗɛɣɲʋạảấầẩẫậắ ằẳẵặẹẻẽềểễệỉịọỏốồổỗộớ ờởỡợụủứừửữựỳỵỷỹ]
Most of these are Vietnamese characters, perhaps previously excluded by being covered by NFD (see above).
If we applied the CLDR-based methodology, we’d remove the following characters (compared to the previous version). Note that the uppercase equivalents would also be included; they are just omitted from this list for brevity.
NOTE: because this would be a more significant change, I’d suggest that we just add the additions above in the version for UCD 6.3. For the beta files for the version built on UCD 7.0, we could switch over to using CLDR; that would allow time for people to review the Subtractions to make sure that we can make adjustments where necessary.
Latin+Greek+Cyrillic: 324 Code Points
Arabic + Ethiopic: 294 Code Points
Remainder: 458 Code Points
In addition, we’d remove the CJK ideographs that are not in CLDR. Note that they would still be a superset of the IDNA characters accepted by the CJK NICs.