Kenneth Whistler wrote:
> Jianping responded:
>
> > Kenneth Whistler wrote:
> >
> > > Jianping wrote:
> > >
> > > > One thing needs to clarify here is that there is no four byte encoding in
> > > > UTF-8S proposal and four byte encoding is illegal but not irregular. As
> > > > everything in UTF-8S is perfect match to UTF-16, any blame to this proposal
> > > > also applies to UTF-16 encoding form.
> > >
> > > Well after a couple months arguing about this, it is nice to have
> > > this little detail drop into place. Perhaps in another couple
> > > of months we could have a complete specification, and then
> > > restart the argument.
> >
> > This is not truth. In its very beginning, we stated that the supplementary character
> > will be encoded as a pair of three bytes. That's why we have a new proposal,
> > otherwise the proposal will be groundless as it will be same as UTF-8.
>
> You're missing the point. To date the specification for UTF-8s
> has not been complete. People have openly speculated on the
> list (before my message under this new topic) about what the
> status of a four-byte supplementary character representation in
> UTF-8s would be.
>
> You did state that supplementary characters are encoded in UTF-8s
> as pairs of three bytes. Everybody understood that. That was the
> well-formedness condition.
>
> What you didn't state were the ill-formedness conditions. Was
> valid UTF-8 considered allowable or not under UTF-8s? Was it
> ill-formed, and if so, how interpreted, or not?
>
> What you finally stated today is that <F0 90 80 80> is flat-out
> *illegal* in UTF-8s. That was a missing piece of the puzzle for anyone
> trying to interpret what you are proposing.
>
In the UTF-8S, there should be no irregular forms, should we repeat the history again?
Nobody except you though that 4-byte is allowed in UTF-8S.
>
> > > =============================================================
> > >
> > > What Jianping is saying now is that F0..F4 are illegal as
> > > initiators in UTF-8s. (They are legal initiators in UTF-8.)
> > >
> > > Also, judging from his statement that "everything in UTF-8S is
> > > perfect match to UTF-16", it is quite clear that UTF-8s does
> > > *not* meet the Unicode Standard's definition of a UTF. To be
> > > a UTF, it has to be a reversible transform of code points (or
> > > Unicode scalar values -- there is some argument about which).
> > >
> >
> > Does UTF-16 meet? If UTF-16 does, UTF-8S should.
>
> UTF-16 is a reversible transform of Unicode scalar values
> to 16-bit code units.
>
> By contrast, you have conceived and defined UTF-8s as a
> reversible transform of UTF-16 16-bit code units to 8-bit
> code units.
>
> Not the same thing.
>
> Presumably UTF-8s could also be defined as a UTF, but
> that isn't how you have been presenting or (apparently)
> conceiving it.
>
> > > But UTF-8s is designed and conceived as a CODE UNIT TRANSFORM
> > > of UTF-16. (A "CUT", not a "UTF".)
> > >
> > > Basically, instead of starting with the code points, and deriving
> > > the three UTF's, for UTF-8s you start with UTF-16 and derive
> > > UTF-8s directly from it. (This is why I have been pounding on
> > > the point that in order to understand the Oracle proposal, you
> > > have to think in terms of the UTF-16 <==> UTF-8 convertors,
> > > rather than in terms of the definitional UTF's.)
> > >
> >
> > This is your perception.
>
> Yep, and I stand by it.
>
> > > In other words, while others are seeing:
> > >
> > > U-00010000 ==> ED A0 80 ED B0 80 in UTF-8s
> > > ==> D800 DC00 in UTF-16
> > >
> > > Oracle is seeing:
> > >
> > > (D800)(DC00) <==> (ED A0 80)(ED B0 80)
> > >
> >
> > That's also your perception but not Oracle as we already support standard UTF-8
> > encoding in 9i.
>
> How is Oracle's support for standard UTF-8 relevant to the conceptual
> definition of UTF-8s?
That means we do recognize U-00010000 in our implementation for UTF formats.
>
>
> > > and pointing out the tremendous simplicity of the fact that
> > > a code point, err... code unit in UTF-16 always corresponds
> > > *exactly* to a code point, errr... well a 1-, 2-, or 3- code
> > > unit sequence in UTF-8s that always corresponds to a, umm..
> > > character, well, sort of.
> > >
> >
> > It is meaningless to examine each bytes of UTF-8S encoding, and this also applies to
> > UTF-8. What is code unit in UTF-8S should be 1-, 2- or 3-bytes unit, and one or two
> > code-unit will be one codepoint. If we still look at each byte of UTF-8S/UTF-S and
> > make random truncation, you will get meaningless bytes there. The best practice here
> > is that you have to treat this 1-, 2-, or 3-bytes encoding as one unit.
>
> Beautifully put. I think you have just confirmed my argument.
>
> > > Now, perhaps Jianping will care to step in an clarify how UTF-32
> > > fits in this picture. How, for example, are the irregular UTF-32
> > > sequences in k and l above to be treated? As I have indicated?
> > > (in which case, as Peter points out, there is an ambiguity in
> > > the interpretation of any 6-byte UTF-8s representation) Or in
> > > some other manner? And if so, how so?
> > >
> >
> > Before answering these questions, just replace UTF-8S by UTF-16, can you give me
> > good answers here? If this is any ambiguity for UTF-8S, so as UTF-16.
>
> Certain I can give you a good answer. Return to Case I of my original document.
>
> In that formulation, the code point U-0000D800 and the code point
> U-0000DC00 are not mapped by the UTF's at all. <U-0000D800, U-0000DC00>
> is just an illegal representation.
>
> The UTF-16 code unit sequence <D800 DC00> *always* corresponds to U+10000.
> It also always corresponds to the UTF-32 code unit sequence <00010000>
> and the UTF-8 code unit sequence <F0 90 80 80>.
>
> No ambiguities, no mapping issues.
>
> Now please answer the question for UTF-32 under your formulation of
> UTF-8s.
My answer here is quite simple:
The UTF-8S code unit sequence <ED A0 80 ED B0 80> *always* corresponds to U+10000.
It also always corresponds to the UTF-32 code unit sequence <00010000>
and the UTF-8 code unit sequence <F0 90 80 80>.
No ambiguities, no mapping issues.
Regards,
Jianping.
>
> > I don't think there is ground here to argue this syntax or semantics issue as UTF-8S
> > should meet the standard requirement exactly the same way as UTF-16. I think the key
> > issue here is its benefit and its implication to the implementor, and I think we
> > should get a best balance between these two.
>
> No comment.
>
> --Ken
>
> >
> > Regards,
> > Jianping.
This archive was generated by hypermail 2.1.2 : Fri Jul 06 2001 - 00:17:18 EDT