IDNA Preprocessing

1. Introduction

Because of the transformation step in IDNA2003, not only can we type "Bücher.de", but we can also type any of "http://Bücher.de", "HTTP://BÜCHER.DE", or "HTTP://BU¨CHER.DE" (where the ¨ represents a U+0308 ( ̈ ) COMBINING DIAERESIS), or many other variations.

There is a projected update of IDNA2003 which is called either IDNAbis or IDNA2008 (the latter by those who think it could be adopted this year). There are quite a number of changes between these two versions: the one relevant to this document is that IDNAbis no longer has a casefolding and normalizing step. Instead, IDNAbis disallows any string that is not already case-folded and normalized. This means that strict adherence to IDNAbis, without any other action, would cause any of the above variant strings to fail. Thus typing "http://Bücher.de" would fail. (Domain names containing all ASCII characters, such as "Bucher.de", would continue to work even with case variations.)

However, when using the IDNAbis specification, many user agents such as browsers will have a requirement to interoperate compatibly with the prior IDNA2003 specification and/or operate in an environment that needs to allow lenient typing or parsing of internationalized domain names. In particular, there are many cases where an internationalized domain name is not formally allowed according to the relevant specifications, but there is widespread de facto use. For example, here is a chart showing the behavior of some major browsers with links containing IDNs.

Note that #6 is not formally provided for in IDNA2003, because the transformation is handled there on a label-by-label basis. However, this form is commonly supported (probably because it is just simpler to apply the transformation to the whole host name rather than to each label individually).

Firefox and IE7 both accept all of these forms (except #4, which is coming), and interpret them as equivalent. Because of that, and because of the substantial number of existing (and future) web pages that contain these formats, implementations will have little choice but to support a preprocessing step that allows for all of the forms, into the indefinite future. Note that this is not simply a typing or UI issue; these are in existing HTML pages. These forms are also parsed out of plain text; for example, most email clients parse for URLs (or IRIs, the internationalized version) and add links to them. So an occurrence of "http://Bücher.de" in the middle of plain text will often be transformed as well.

IDNAbis (current draft) allows for preprocessing (called local mapping), and even allows these to be different according to locale or application program. But we certainly don't want different programs (browsers, email clients, etc) to map these characters differently. That would cause a huge interoperability problem. Look, for example, at what could happen to different strings in the following table under locale-specific mappings.

An IDNA2008-conformant implementation could remap any of the items #4 to #9 in the Link Text column. using a local mapping -- or not, in which case they would fail. It could remove the illegal characters in #8 to #9, or not remove them and have the lookup fail. It could map the ligature ff to ff, or not. It could even decide, for example, based on locale linguistic mappings (using the UI language of the client, or the language of the email, or the default system language), that it could map #5 and #6 to different valid domain names, different than what IDNA2003 does. That means on the same page, a browser might go to different places depending on what the user's language was.

With IDNA2003, in contrast, the mappings for all of these are completely determinant (with all but the first being allowed, and the last being disallowed). Instead of a free-for-all of local mappings, what we need is a common mapping that maintains compatibility with IDNA2003, accomodates the newer Unicode characters, provides stability over time, and thus allows for interoperability among all programs that use it. In discussion with IETF people at the Ireland meeting, there appears to be general consensus that the specification of such a mapping does not belong in the IETF.

Note that lower-level protocols, such as the SMTP envelope, should require the strict use of already-transformed IDNs, and thus not use the preprocessing specified here. Language-specific modifications to the preprocessing specified in this document are outside the scope of this document; they are, however, very strongly discouraged because of the problems they pose for interoperability.

1.1 Requirements

• The mapping should be as compatible as possible with IDNA2003. To try out IDNA2003, see http://demo.icu-project.org/icu-bin/idnbrowser . There will be some minor differences, but non that should have any practical import. See Section 5 Differences from IDNA2003
• Where the user has a reasonable expectation that giving the character "X" as input will be treated as equivalent to "X'", and it's possible to determine this unambiguously, this is what should happen.
• Where the preprocessing results in an "abort with error", the input is not interpreted as valid. In a user-interface, this should be indicated with a warning; in other processing (such as search-engine parsing of a web page), lookup would fail.

2. Preprocessing

The input to the preprocessing is a domain_name string, which is a sequence of labels with dot separators, such as "Bücher.de". (For more about the parts of a URL, including the domain name, see http://tools.ietf.org/html/rfc3987). The preprocessing consists of the following steps, performed in order.

1. Convert the input domain_name string to Unicode.
   • Abort with error if there is any conversion problem.
   • This may involve converting escapes in the domain_name string to Unicode code points as necessary, depending on the environment in which it is being used. For example, this can include:
     • HTML NCRs like 十 for  U+5341 ( 十 ) CJK UNIFIED IDEOGRAPH-5341
     • Javascript escapes like \u5341 for U+5341 ( 十 ) CJK UNIFIED IDEOGRAPH-5341
     • URI/IRI %-escapes like %2e for U+002E ( . ) FULL STOP.
2. Map the domain_name string according to the IDNA Preprocessing Table (see below).
3. Normalize the domain_name string to Unicode Normalization Form C:
   • domain_name = toNFC(domain_name)
   • Note: because of the construction of the table, characters are limited to those already allowed by NFKC, so this is equivalent to toNFKC().
4. Parse the domain_name string into labels, using U+002E ( . ) FULL STOP as the label delimiter.
   • Note that the dot may have resulted from a mapping from other characters, such as U+2488 ( ⒈ ) DIGIT ONE FULL STOP or U+FF0E ( ． ) FULLWIDTH FULL STOP. See below.
5. Verify that each label in the domain_name complies with IDNAbis.
   • Abort with error if it does not comply
   • Each label that contains only characters [\-a-zA-Z0-9] is an ASCII label. Each other label must conform to the IDNAbis specification.
6. Return the string resulting from the successive application of the above steps.

Note that the processing matches what is commonly done with label delimiters (eg by browsers), whereby characters containing periods are NFKC'd before labels are separated. These characters can be seen with the Unicode utilities using a regular expression: