L2/02-392

The following is collected (unedited) feedback on TR29, sometimes with a few comments from me interspersed.

----- Original Message -----
From: "Murray Sargent" <murrays@Exchange.Microsoft.com>
To: "Mark Davis" <mark.davis@jtcsv.com>
Sent: Friday, October 11, 2002 18:18
Subject: RE: TR29 Text Boundaries


> I think Mynmar should be Myanmar. Thanks for putting in the comment
> about
" in Hebrew.
>
> Murray

> I have several comments on
>  http://www.unicode.org/unicode/reports/tr29/
> Draft Unicode Technical Report #29 Text Boundaries
>
> 1.  I strongly disagree with this treatment of colon (:)
> as a MidLetNum, as stated:
> <quote>
> MidNumLet Any of the following:
> U+002E (.) FULL STOP (period)
> U+003A (:) COLON (used in Swedish)
> </quote>
> ...
> <quote>
> Certain cases like colon in words (c:a) are included even though they may
> be
> specific to relatively small user communities (Swedish) because they don’t
> occur otherwise in normal text, and so don’t cause a problem for other
> languages.
> </quote>
>
> I am not sure what if anything is considerted "normal text".
> Much of what I am reading and writing about these days either is XML,
> or discusses XML.  Thus I see a great many strings of the form
> foo:name where foo is a namespace prefix and name is an element name
> (or function name, etc.)  The colon is also used in non-XML
> files, e.g. to separate key:value pairs, e.g. color:red.
> I would certainly want words broekn at the colon.
>
> As the spec is currently written the colon is always a MidLetNum.
> I suggest changing the wording to state that this is
> a tailoring, only forSwedish.
> (This would be analagous to the wording in the very
> next sentence of the spec:
> <quote>
> For Hebrew, a tailoring may include a double quotation mark between
> letters,
> since legacy data may contain that in place of U+05F4 (״) gershayim. This
> can be done by adding double quotation mark to MidLetter.
> </quote>
>
> Also, I suspect that even in Sweden many people, e.g. those
> reading [about] XML, will also prefer words to be split at the colon.
> So they might want a way to turn that tailoring off.
>
> *** I will raise this point to the UTC.
>
>
> 2. We have had some discussions about whether these rules in TR 29
> are normative in any way, and I think we concluded that they are not.
> However you stated that it might be useful for an implementation
> to be able to claim some kinf of conformance anyway.
> I agree, but would like some clarification.
> I think many implementations will be close,
> but not exact in their conformance.  I think an implementation
> should be able to make one of these statements:
> 1. complies exactly with the example approach in TR 29
> 2. makes all these word splits, and possibly more
> 3. makes no other word splits, but may not make all of these
> 4. anything else
>
> *** The rules are not normative in the sense that conformance to the
> Unicode
> Standard does not require them. I agree with the options that you cite,
> and
> will bring those to the UTC.
>
>
> 3.  The recent widespread adoption of the naming convention
> usually called CamelCase has added the need for another
> kind of word split pattern.  For example in the string
> getWordPiecesFromName I want to allow the word splits
> marked by the vertical bars: get|Word|Pieces|From|Name
>
> An algorithm for processing these word split rulls
> in presented in Appendix C of
> The Java™ Architecture for XML Binding (JAXB)
> Public Draft 2, V0.75  October 4, 2002
> which I downloaded from http://java.sun.com/webapps/download/DisplayLinks
> after first going to http://java.sun.com/xml/downloads/jaxb.html
> I only quote a small excerpt, but it should be sufficient,
> and I do include all the patterns.
> <quote>
> An XML name is split into a word list by removing any leading and trailing
> punctuation characters and then searching for word breaks. A word break is
> defined by three regular expressions: A prefix, a separator, and a suffix.
> The
> prefix matches part of the word that precedes the break, the separator is
> not part
> of any word, and the suffix matches part of the word that follows the
> break.
> The
> word breaks are defined as:
> Table 3-1 XML Word Breaks
> Prefix   Separator Suffix      Example
> [^punct] punct+    [^punct]    foo|--|bar
> digit              [^digit]    foo22|bar
> [^digit]           digit       foo|22
> lower              [^lower]    foo|Bar
> upper              upper lower FOO|Bar
> letter             [^letter]   Foo|\u2160
> [^letter]          letter      \u2160|Foo
> </quote>
>
> I think it is reasonable for an implementation to
> introduce word split points where the case changes.
> Note especially that FOOBar is split FOO|Bar.
> I generally like this approach.
>
> However I suspect that this JAXB approach, which is based upon
> regular expressions, requires more power that
> the approach used in TR29, which seems to only
> look at two adjacent characters.  Actually I think one
> only needs an additional one character lookahead to
> handle the abbove patterns.
>
> *** The TR is not limited to a single character context before and after;
> the context can be more complex expressions. However, I do not believe
> that
> this is the correct approach to take in the default case. This would cause
> problems for real, natural language cases like McDowell, or vederLa
> (italian), or other cases of interior case changes in real words.
>
> What could be done is to point out that this is a possible tailoring for
> programming language contexts.
> ***
>
>
> 4.  I also provided detailed feedback to you on an earlier
> version of TR 29.  It might be nice to add my name to the list
> of people in the Acknowledgments section.
>
> *** This was a distinct oversight on my part. My apologies -- your name
> will
> definitely be on the next version.
>
>
> Hopefully helpfully yours,
> Steve

> -----Original Message-----
> From: Tolkin, Steve [SMTP:Steve.Tolkin@FMR.COM]
> Sent: Monday, October 28, 2002 1:04 PM
> To: 'Mark Davis'; Michael Rys; Paul Cotton
> Cc: w3c-query-operators@w3.org; w3c-i18n-ig; Andy Heninger; Cathy Wissink; Murray Sargent; Michel Suignard
> Subject: RE: FTTF Agenda Item 4, Review of Unicode boundaries
>
I had a private exchange with Steve and he suggested I post this to the list...

> 1.  I strongly disagree with this treatment of colon (:)
> as a MidLetNum, as stated:
>
<deleted/>

> Much of what I am reading and writing about these days either is XML,
> or discusses XML.  Thus I see a great many strings of the form
> foo:name where foo is a namespace prefix and name is an element name
> (or function name, etc.)  The colon is also used in non-XML
> files, e.g. to separate key:value pairs, e.g. color:red. 
> I would certainly want words broekn at the colon.
>
It is because of that that I'd like to see ":" not be a word boundary (in all cases).  In other words, when I tokenize some text that talks about XML, I'd like to get "xsl:apply-templates" as a single token because that "word" is more likely to be thought of in the writer's/querier's mind as a single "concept" rather than as "xsl" and "apply-templates". 

I see the case here as very similar to German compound words...computertastatur should tokenize as a single "word" on 1st pass...if you want to break it down into "computer" and "tashtatur" on a 2nd-level tokenization then that's fine.  Given a 1st-level tokenization that says "xsl:apply-templates" is a single "word" it is nice to have an algorithm that tells me this one word is probably composed of the two words "xsl" and "apply-templates" (the 2nd of which might also be decomposed into "apply" and "templates").  But I consider that fine grained tokenization to be a 2nd-level process.  The same applies to your "camelCase" examples.

Have you thought about it in these terms?

pvb