Re: Unicode String Models

Mark

On Tue, Oct 2, 2018 at 8:31 PM Daniel Bünzli <daniel.buenzli_at_erratique.ch>
wrote:

> On 2 October 2018 at 14:03:48, Mark Davis ☕️ via Unicode (
> unicode_at_unicode.org) wrote:
>
> > Because of performance and storage consideration, you need to consider
> the
> > possible internal data structures when you are looking at something as
> > low-level as strings. But most of the 'model's in the document are only
> > really distinguished by API, only the "Code Point model" discussions are
> > segmented by internal storage, as with "Code Point Model: UTF-32"
>
> I guess my gripe with the presentation of that document is that it
> perpetuates the problem of confusing "unicode characters" (or integers, or
> scalar values) and their *encoding* (how to represent these integers as
> byte sequences) which a source of endless confusion among programmers.
>
> This confusion is easy lifted once you explain that there exists certain
> integers, the scalar values, which are your actual characters and then you
> have different ways of encoding your characters; one can then explain that
> a surrogate is not a character per se, it's a hack and there's no point in
> indexing them except if you want trouble.
>
> This may also suggest another taxonomy of classification for the APIs,
> those in which you work directly with the character data (the scalar
> values) and those in which you work with an encoding of the actual
> character data (e.g. a JavaScript string).
>

Thanks for the feedback. It is worth adding a discussion of the issues,
perhaps something like:

A code-point-based API takes and returns int32's, although only a small
subset of the values are valid code points, namely 0x0..0x10FFFF. (In
practice some APIs may support returning -1 to signal an error or
termination, such as before or after the end of a string.) A surrogate code
point is one in U+D800..U+DFFF; these reflect a range of special code units
used in pairs in UTF-16 for representing code points above U+FFFF. A scalar
value is a code point that is not a surrogate.

A scalar-value API for immutable strings requires that no surrogate code
points are ever returned. In practice, the main advantage of that API is
that round-tripping to UTF-8/16 is guaranteed. Otherwise, a leaked
surrogate code point is relatively harmless: Unicode properties are devised
so that clients can essentially treat them as (permanently) unassigned
characters. Warning: an iterator should *never* avoid returning surrogate
code points by skipping them; that can cause security problems; see
https://www.unicode.org/reports/tr36/tr36-7.html#Substituting_for_Ill_Formed_Subsequences
and
https://www.unicode.org/reports/tr36/tr36-7.html#Deletion_of_Noncharacters.

There are two main choices for a scalar-value API:

   1. Guarantee that the storage never contains surrogates. This is the
   simplest model.
   2. Substitute U+FFFD for surrogates when the API returns code
   points. This can be done where #1 is not feasible, such as where the API is
   a shim on top of a (perhaps large) IO buffer of buffer of 16-bit code units
   that are not guaranteed to be UTF-16. The cost is extra tests on every code
   point access.

> > In reality, most APIs are not even going to be in terms of code points:
> > they will return int32's.
>
> That reality depends on your programming language. If the latter supports
> type abstraction you can define an abstract type for scalar values (whose
> implementation may simply be an integer). If you always go through the
> constructor to create these "integers" you can maintain the invariant that
> a value of this type is an integer in the ranges [0x0000;0xD7FF] and
> [0xE000;0x10FFFF]. Knowing this invariant holds is quite useful when you
> feed your "character" data to other processes like UTF-X encoders: it
> guarantees the correctness of their outputs regardless of what the
> programmer does.
>

If the programming language provides for such a primitive datatype, that is
possible. That would mean at a minimum that casting/converting to that
datatype from other numerical datatypes would require bounds-checking and
throwing an exception for values outside of [0x0000..0xD7FF
0xE000..0x10FFFF]. Most common-use programming languages that I know of
don't support that for primitives; the API would have to use a class, which
would be so very painful for performance/storage. If you (or others) know
of languages that do have such a cheap primitive datatype, that would be
worth mentioning!

> Best,
>
> Daniel
>
>
>
Received on Wed Oct 03 2018 - 02:17:51 CDT