Re: Corrigendum #9

On 06/02/2014 09:48 AM, Markus Scherer wrote:
> On Mon, Jun 2, 2014 at 8:27 AM, Doug Ewell <doug_at_ewellic.org
> <mailto:doug_at_ewellic.org>> wrote:
>
> I suspect everyone can agree on the edge cases, that noncharacters are
> harmless in internal processing, but probably should not appear in
> random text shipped around on the web.
>
>
> Right, in principle. However, it should be ok to include noncharacters
> in CLDR data files for processing by CLDR implementations, and it should
> be possible to edit and diff and version-control and web-view those
> files etc.
>
> It seems that trying to define "interchange" and "public" in ways that
> satisfy everyone will not be successful.
>
> The FAQ already gives some examples of where noncharacters might be
> used, should be preserved, or could be stripped, starting with "Q: Are
> noncharacters intended for interchange?
> <http://www.unicode.org/faq/private_use.html#nonchar6>"
>
> In my view, those Q/A pairs explain noncharacters quite well. If there
> are further examples of where noncharacters might be used, should be
> preserved, or could be stripped, and that would be particularly useful
> to add to the examples already there, then we could add them.
>
> markus
>
>

I was unaware of this FAQ. Having read it and re-read this entire
thread, I am still troubled.

I have a something like a library that was written a long time ago (not
by me) assuming that noncharacters were illegal in open interchange.
Programs that use the library were guaranteed that they would not
receive noncharacters in their input. They thus are free to use any
noncharacter internally as they wish. Now that Corrigendum #9 has come
out, I'm getting requests to update the library to not reject
noncharacters. The library itself does not use noncharacters. If I (or
someone else) makes the requested change, it may silently cause security
holes in those programs that were depending on it doing the rejection,
and who upgrade to use the new version. Some of these programs may have
been written many years ago. The original authors are now dead in some
instances, or have turned the code over to someone else, or haven't
thought about it in years. The current maintainers of those programs
may be unaware of this dependence, and hence may upgrade without
realizing the consequences. Further, the old versions of the library
will soon be unsupported, so there is pressure to upgrade to get bug
fixes and the promise of future support. This means there could be
security holes that a hacker who gets a hold of the source can exploit.

I don't see anything in the FAQ that really addresses this situation. I
think there should be an answer that addresses code written before the
Corrigendum, and that goes into detail about the security issues. My
guess is that the UTC did not really consider the potential for security
holes when making this Corrigendum.

I agree that CLDR should be able to use noncharacters for internal
processing, and that they should be able to be stored in files and
edited. But I believe that version control systems and editors have
just as much right to use noncharacters for their internal purposes. I
disagree with the FAQ that seems to say if you write a utility you
should avoid using noncharacters in its implementation. It might be
that competitive pressure, or just that the particular implementations
don't need non-characters, would cause some such utilities to accept
some or all non-characters as inputs. But If I were writing such code,
I can see now how using noncharacters for my purposes would be quite
convenient. CLDR could be considered to be a utility, and its users
might want to use noncharacters for their purposes. Is CLDR constructed
so there is no potential for conflicts here? That is, does it reserve
certain noncharacters for its own use?

The FAQ talks about how various now-noncharacter code points were touted
as sentinel candidates in earlier Unicode versions, and that they are no
longer so. But it really should emphasize that old code may very well
want to continue to use them as sentinels. The answer "Well, the short
answer is no, that is not true—at least, not entirely true." is
misleading in this regard.

The FAQ mentions using 0x7FFFFFFF as a possible sentinel. I did not
realize that that was considered representable in any UTF. Likewise -1.

_______________________________________________
Unicode mailing list
Unicode_at_unicode.org
http://unicode.org/mailman/listinfo/unicode
Received on Wed Jun 11 2014 - 23:31:26 CDT