Re: Counting Codepoints

Replace U+FFFE by U+FFFD in my message (but there are applications that
also prefer using non-characters for those replacements, this is also an
additional alternative, as U+FFFE has a valid representation as well in
UTF-16). U+FFFD is not the only possible replacement even if it is
recommended (by a "best practrice", which is not a "requirement" for
conformance purpose).

2015-10-12 17:29 GMT+02:00 Philippe Verdy <verdy_p_at_wanadoo.fr>:

> 2015-10-12 14:42 GMT+02:00 Mark Davis ☕️ <mark_at_macchiato.com>:
>
>> If these are not all aligned, then all heck breaks loose: you are letting
>> yourself in for code breakage and/or security problems.
>>
>> So the corresponding code point count would just return a count of 1 for
>> an isolated surrogate.
>>
>
> But the behavior in this case is absolutely not defined, and applications
> are free to do what they want when they encounter them. There's not even
> any warranty that any further (correctly encoded) code point will be
> returned, even if a replacement character like U+FFFE is returned, it could
> replace all the rest.
>
> So the count of 1 is possible for the first isolated surrogate but all the
> rest count count as 0 as well, or all the further characters could be
> replaced by U+FFFE independantly of what they initially represented. This
> would also be a "sanitized" result.
>
> TUS gives freedom of choice in application. There's absolutely no warranty
> that all possible "sanitized" results will be the same for all
> applications, and TUS does not even mandate which replacement character to
> use (not necessarily U+FFFE, it could as well be an ASCII '?' character or
> a C0 <SUB> or <DEL> control, when further processed to an application
> converting the result to some legacy 7-bit or 8-bit charset).
>
> My opinion is that the only really safe result is to not return any count
> of code points but instead throw an error (counting code points and with a
> function returning an integer is only valid if the UTF-16 input is actually
> a valid representation of code points, you cannot return a single integer
> as the application using that integer could expect to allocate some
> processing buffer, and then get this exact number of code points when
> reading the data into some processing buffer, and could leave initialized
> some positions in that buffer, or the application could assume that the
> input was left untouched and could then get an unexpected mismatch of
> digital signature).
>
> If your function counting codepoints and returning an integer counts those
> lone surrogates as 1, it assumes that exactly one codepoint will be
> returned for each lone surrogate, and it should document that clearly,
> meaning that the result is only valid if this matches the results of the
> actual input scanner. In that case that function will never fail and throw
> an exception. But between two implementations the result of the scanner
> could still be different because the replacement character is not
> specified. If that result "sanitized" string is then used to generate an
> URI, the URI is also unpredictable and will vary between implementations,
> as well as its effective length. If it is used to generate an identifier
> granting some new access, such as a user name, several new user names
> could be generated from the same input.
>
> So in all cases using replacements will also create security problems.
> This will not happen if you don't return any result but throw an exception
> (that counting function should document this exception so that it is not
> unexpectedly thrown and left unhandled, causing the program to abort
> prematurely in an unsafe state including loosing other data or transaction
> elsewhere in an incoherent state).
>
> For all programs taking some standard UTF input, the input scanner or
> processing functions MUST be prepared to handle the encoding error
> exception, which is an result expected equally to the return of a value or
> the execution of some code ! Sanitization is possible, but not described in
> the standard, and there are several conflict ways of doing it, it should be
> a separate subprocess documented separately.
>
Received on Mon Oct 12 2015 - 10:34:23 CDT